-
Notifications
You must be signed in to change notification settings - Fork 1.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Memory leak on nomad servers #14842
Comments
Hi @D4GGe and thanks for raising this issue. I don't believe you are experiencing a memory leak on the servers because, as you mention, running the Nomad internal GC process reduces the memory overhead down. It seems like this is therefore normal behaviour on Nomad's part and I will try to explain in a little more detail below what is happening along with some configuration options that might help.
Each job registration results in a number of objects (jobspec, evaluation, allocation, deployment, etc...) being created and subsequently stored within the Nomad state store. This state store is held stored in memory and replicated to each server via Raft from the leader. I would therefore expect Nomads memory usage to grow as more jobs and allocations are added to the cluster.
The Nomad leader runs a number of internal routines to clean up its state store at regular intervals. This interval usually defaults to 5 minutes. At each interval the leader will identify state objects such as jobs, evaluations, allocations, and deployments which have reached a terminal state, are no longer needed as reference by other objects, and have been in this terminal state for a configurable threshold time. The The GC intervals and thresholds can be modified via the server configuration block to be more aggressive to account for high job turnover. There are a number of options; the job_gc_interval and job_gc_threshold would be good places to start.
The Nomad API would provide some information about what objects are being stored within state such as jobs, allocations, and allocations. Another option would be to use the operator debug CLI tool to dump out information regarding your system for further investigation. This is a common practice we use internally and will provide a wide range of data. The data directory on the server most importantly holds the boltdb data which contains Raft log entries. This allows the Nomad process to restart and reload data without any data loss. It is possible to view this data with boltdb CLI tools, however, I do not believe it would be useful in this case.
This could be an indication of disk contention and performance issues or general Raft contention but I can't identify that from any information presented. It is not related to the memory consumption you are seeing. Hopefully this explanation makes sense and is useful. Please let me know if you have any follow up questions, otherwise I will close this issue after a few days. |
Thank you for really good explanation! Would it also help to increase numbers of servers and or increase the performance of them. |
I don't believe increasing the number of servers will have any effect as each server stores a replica of the state store in memory. It has the potential to make situations worse, as the data needs to be replicate more times from the leader. Depending on how tuning the GC parameters goes, it could be beneficial to increase the available RAM for the servers, but I am confident the GC modifications are the correct solution.
We don't have exact figure or data and each environment is different. That being said, I have certainly seen this pattern before in clusters which have high turnover rates of batch workloads. As with above, I would expect the GC tuning to help make this a situation where the current specs are enough. |
Hello, Each time the job is applied, it will "re-register" the job (we can see it in the evaluations history) but do nothing as there is no change or no version change of the job. Forcing a "nomad system gc" doesn't change anything because it seems that it won't GC the evaluations entries as long as the job exist. However, removing the job and applying a "nomad system gc" before the operator recreate the job with the same name do reset the evaluations and reduce the average used memory from 5Go to 780Mo instantly. So, in our case, the fix could be:
I hope this little observation will be useful to understand this behavior and find a common denominator. EDIT: Seems a ticket about this eval cleaning behavior do exist: |
Hi @D4GGe and @arsiesys, thanks for all the additional information. It seems like there are potentially two related issues being observed here (thanks for linking those), so I will mark that this requires further investigation and add it to our backlog. If you find any additional information, please feel free to add it to this issue. |
Hey there! I was writing another bug report when I was pointed to this one. I think I might have rootcaused the problem. Please see the linked issue above. |
Fixed in #15097 which will ship in Nomad 1.5.0 (with backports) |
Nomad version
Nomad v1.3.1 (bc49012743edc0e2adb565e0fbcbc36c184f8ce4+CHANGES)
Operating system and Environment details
3 nomad servers
~10 large nomad clients
No nvidia plugin installed
~20 nomad clients of various sizes
Issue
Memory leak on nomad servers
We have a memory leak on our nomad servers, the cluster have around 30 clients and the memory leak shows when we run a lot of of short lived jobs (parameterized job ~100 new allocs per hour), but seams to build up even if we lower the load just slower.
The only way for us to fix this is to run an
nomad system gc
on one of the servers. Otherwise it uses up all memory on the server until the server needs a reboot to be usable again.When this happens the only way to start nomad without nomad using 100% memory is to remove the
data_dir
and start over.Right now in our production we run automatically
nomad system gc
each time memory goes over 30% and this is how it looks.Are there any way of analyzing what actually take up this amount of memory?
Can I analyze the
data_dir
in any meaningful way?This log show up ever now and then can it have anything to do with it?
Configuration
Server
Clients
Reproduction steps
Expected Result
Actual Result
Job file (if appropriate)
We have many jobs running but this is the main one used with Levant
job.hcl
Nomad Server logs (if appropriate)
This is some sample logs from the server
server-logs.txt
Nomad Client logs (if appropriate)
The text was updated successfully, but these errors were encountered: