-
Notifications
You must be signed in to change notification settings - Fork 9.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Etcd reboot causes RSS to double #7822
Comments
How was etcd being stressed? Any details on the workloads? |
We are using the Etcd benchmark tool.
|
@davissp14 Thanks for that command. Let me try to profile that workloads. |
Yep! |
Confirmed that most memory usage comes from /cc @heyitsanthony @xiang90 |
yea. it is just go GC. |
There's a TODO in kvstore.go's restore code about loading all the keys into memory, which is probably why it's eating so much memory on boot. |
@heyitsanthony yea... i am not sure if this worth to fix. this is the first time someone bring it up. |
For additional context I am working on trying to figure out a scaling scheme for Etcd on Compose. Right now in my testing, nodes are being prematurely scaled due to the RSS spikes during reboots, which isn't ideal. If it didn't take 5 minutes to settle it wouldn't be such an issue, but right now I don't have a very good scaling story using RSS. Any other suggestions on how to deal with this? |
Loading all keys at once would cause etcd to use twice as much memory than it would need to serve the keys, causing RSS to spike on boot. Instead, load the keys into the mvcc by chunk. Uses pipelining for some concurrency. Fixes etcd-io#7822
Loading all keys at once would cause etcd to use twice as much memory than it would need to serve the keys, causing RSS to spike on boot. Instead, load the keys into the mvcc by chunk. Uses pipelining for some concurrency. Fixes etcd-io#7822
Loading all keys at once would cause etcd to use twice as much memory than it would need to serve the keys, causing RSS to spike on boot. Instead, load the keys into the mvcc by chunk. Uses pipelining for some concurrency. Fixes #7822
Loading all keys at once would cause etcd to use twice as much memory than it would need to serve the keys, causing RSS to spike on boot. Instead, load the keys into the mvcc by chunk. Uses pipelining for some concurrency. Fixes etcd-io#7822
Version: 3.1.5
I am doing some fail testing with Etcd and noticed that RSS appears to double in the event of a reboot and seems settle after 5 minutes or so. My initial assumption was a process was being forked and the RSS doubling was just due to copy-on-write. Is my assumption correct? Or is there something else at play here?
This isn't a great image i know, but the red line is RSS and orange is Cache.
![screen shot 2017-04-18 at 10 55 42 am](https://cloud.githubusercontent.com/assets/423038/25439389/f4973d9e-2a61-11e7-90b8-b085233fc9e5.png)
My second question is what metrics should we be listening on scaling purposes? I initially thought that RSS would be good enough, but after a lot of testing it seems RSS is too spiky to be a reliable metric to listen on. I see quite a few situations where RSS quickly spikes to memory capacity then drops quickly as it sends things to SWAP.
Thanks
The text was updated successfully, but these errors were encountered: