-
Notifications
You must be signed in to change notification settings - Fork 9.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
"fatal error: runtime: out of memory" #3300
Comments
@spacejam When doing the snapshot, etcd might use 2x memory or even more. etcd does not store anything on-disk. We can try to detect a potential memory usage issue. But it is not our priority (partially because we are working on new backend ) |
@spacejam For the goroutine issue, most of them are net routines controlled by go lib. We have been running etcd under very heavy load for months and never observed a routine leak. |
/cc @yichengq |
@spacejam Some related issue about memory usage: #2657 Based on the keys that you write into etcd this time, I think the data in store should be a lot. The goroutine list is good. The server hosts 102 connections, based on goroutine that contains |
The load generator is basically throwing 0-1mb values that average 500k at ~100 req/s into 52 keys, so this is totally an uncommon workload that was designed to stress things out. I don't think it's likely to be hit by many people, but there may be room somewhere to add backpressure that prevents accepting more work than can be safely handled. |
I agree that the new back-end will very likely effect this behavior, so it's just something to keep in mind until that comes to fruition. |
@spacejam I will open up anther issue to track this for the new backend. |
Sounds good! |
This may be expected behavior, but I figured I would paste it here because it surprised me that it crashed instead of degrading gracefully.
While doing some benchmarks for the work I've been doing on the runtime performance analyzer, I was using this load generator:
which led to this OOM death, possibly highlighting a goroutine leak where a goroutine pool maybe should have been used instead of a spawn.
The text was updated successfully, but these errors were encountered: