You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
What happened:
One of our etcd cluster (running a 5 node quorum) hit ‘mvcc: database space exceeded’ errors. This is when etcd was running with default quota backend size of 2 GB. We increased this limit to 4 GB to unblock the write operations to etcd cluster and analyze the increase in DB size. But unfortunately this size also filled up pretty fast. When we tried compaction and defragmentation, it reduced the DB size on only 2 of the 5 hosts and the remaining 3 nodes continue to show very high db size of 4 GB and the alarm of database space exceeded continue to remain on the cluster.
What you expected to happen:
defragmentation should reduce the db size and remove previously compacted versions
How to reproduce it (as minimally and precisely as possible):
This is an issue we observed only on one particular cluster
Anything else we need to know?:
Validations that we performed:
All the keys when listed only had 1 revision (due to compaction).
We tried compaction and defragmentation commands multiple times but the DB size goes down only on 2 hosts (as mentioned earlier in the post) and the other 3 nodes continue to show 4GB db size and the alarm persisted.
Environment:
Kubernetes version (use kubectl version): v1.10.8
etcd version: v3.4.2
Cloud provider or hardware configuration: Bare Metal
OS (e.g: cat /etc/os-release): centos 7
Kernel (e.g. uname -a): 3.10.0
Install tools:
Network plugin and version (if this is a network-related bug): i
Others:
The text was updated successfully, but these errors were encountered:
etcdctl in here is a wrapper script that we have on top of etcdctl docker instance that automatically adds the common params like endpoints, certs etc.,
This issue has been automatically marked as stale because it has not had recent activity. It will be closed after 21 days if no further activity occurs. Thank you for your contributions.
What happened:
One of our etcd cluster (running a 5 node quorum) hit ‘mvcc: database space exceeded’ errors. This is when etcd was running with default quota backend size of 2 GB. We increased this limit to 4 GB to unblock the write operations to etcd cluster and analyze the increase in DB size. But unfortunately this size also filled up pretty fast. When we tried compaction and defragmentation, it reduced the DB size on only 2 of the 5 hosts and the remaining 3 nodes continue to show very high db size of 4 GB and the alarm of database space exceeded continue to remain on the cluster.
What you expected to happen:
defragmentation should reduce the db size and remove previously compacted versions
How to reproduce it (as minimally and precisely as possible):
This is an issue we observed only on one particular cluster
Anything else we need to know?:
Validations that we performed:
Environment:
kubectl version
): v1.10.8cat /etc/os-release
): centos 7uname -a
): 3.10.0The text was updated successfully, but these errors were encountered: