etcd defragmentation doesnt work as expected

[sridhav]

What happened:
One of our etcd cluster (running a 5 node quorum) hit ‘mvcc: database space exceeded’ errors. This is when etcd was running with default quota backend size of 2 GB. We increased this limit to 4 GB to unblock the write operations to etcd cluster and analyze the increase in DB size. But unfortunately this size also filled up pretty fast. When we tried compaction and defragmentation, it reduced the DB size on only 2 of the 5 hosts and the remaining 3 nodes continue to show very high db size of 4 GB and the alarm of database space exceeded continue to remain on the cluster.

What you expected to happen:
defragmentation should reduce the db size and remove previously compacted versions

How to reproduce it (as minimally and precisely as possible):
This is an issue we observed only on one particular cluster

Anything else we need to know?:
Validations that we performed:

1. All the keys when listed only had 1 revision (due to compaction).
2. We tried compaction and defragmentation commands multiple times but the DB size goes down only on 2 hosts (as mentioned earlier in the post) and the other 3 nodes continue to show 4GB db size and the alarm persisted.

Environment:

• Kubernetes version (use kubectl version): v1.10.8
• etcd version: v3.4.2
• Cloud provider or hardware configuration: Bare Metal
• OS (e.g: cat /etc/os-release): centos 7
• Kernel (e.g. uname -a): 3.10.0
• Install tools:
• Network plugin and version (if this is a network-related bug): i
• Others:

[agargi]

@sridhav Please share commands fired to perform defragmentation. In addition, hope all etcd cluster clients are using API v3.

[sridhav]

@agargi these are the commands that we have used to defragment the cluster

etcdctl compact <latest_revision>
etcdctl defrag --cluster=true
etcdctl alarm disarm

etcdctl in here is a wrapper script that we have on top of etcdctl docker instance that automatically adds the common params like endpoints, certs etc.,

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed after 21 days if no further activity occurs. Thank you for your contributions.