-
Notifications
You must be signed in to change notification settings - Fork 1.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Deployment objects are not garbage collected #3244
Comments
I have 100K deployments, and pulling on the API causes about 100MB of data...(0.6.0 here) |
Can you all share a sample deployment that won’t gc |
@dadgar are you sure gc for deployments is implemented? Because AFAICT all out ours are just piling up, and there's nothing special about them, super simple update block 99% with single max_parallel statement. |
Yes it is implemented. The deployments won't be garbage collected when there is a running allocation referencing it. |
That's interesting :). I'll try to see if any of our job specs trigger this on a dev cluster. Is the GC event driver and triggers immediately after the last allocation is removed, or do I have to wait for periodic clean-up? |
You would have to wait for a periodic clean up or you can run |
in hashi-ui thats is also a button under |
Thanks. I'll try to investigate deeper first thing tomorrow morning. |
Soooo. I sent The only jobs that have more than one deployment share a job version, so it's most likely that the newer ones were just moving a subset of allocations due to node failure. BUT. Any reason why GC is not running on its own? EDIT/UPDATE: after one forced GC no other cleanup ran fot the past 7h+
|
Great news :) thank you @dadgar |
@dadgar it does, clean out when forcing GC, thx! |
@hsmade Sweet! Thanks both of you! Will be fixed in 0.7! |
BTW.. A note on the huge data transfer I saw. This was mainly caused by us configuring the reserved ports (20000-32000) because of a misunderstanding of the docs (fixed the docs since then). If you have 64 clients, and each of the produces about 1MB of data for just the port reservations, that adds up :) |
@hsmade Yeah that is definitely something we are aware of and would like to make a simple two integers for a range rather than materializing each port! |
hehe, yes :) |
I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues. |
Nomad version
0.6.3
Issue
As mentioned in #3157, during normal use the number of objects returned by
nomad deployment list
increases without any apparent bound.Reproduction steps
$ nomad deployment list | wc -l 1224
While I do not see any performance or stability hit as of now, I'm starting to get interested in knowing where we will start to observe some kind of cluster degradation? never? in 5 minutes? 2K/10K/100K/1M/10M deployments?
The text was updated successfully, but these errors were encountered: