-
Notifications
You must be signed in to change notification settings - Fork 1.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Nomad server nodes using up all available host memory #12445
Comments
Hi @robsonj there is some documentation on guidance for resources here. However this certainly seems concerning
If there were a simple memory leak a simple restart of the agent should suffice; instead it sounds like there's an explosion of data happening in the persistent state store, which would then be cached in memory, of which there is less available. Any chance you've got metrics on both memory and disk usage, and see if there is a correlation? Next time an agent starts chewing up memory, see if you can grab a heap dump and stack trace using nomad operator debug which will help track down a leak if there is one. |
Hi @shoenig I have a copy of the serverdata directory before I removed it, would its size help you in any way? One thing I was wondering is maybe we just have too many pending jobs for the 16gb of memory. Is there a way to set a limit in nomad where it will reject job submissions if there are too many in a pending state perhaps? |
@shoenig serverdata directory is 9.8gb in size, drilling down into that directory shows 2 snapshot directories, each about 4.3gb in size, another .tmp directory inside the snapshots directory is about 349mb in size. So wondering if its going into some spin taking or restoring from a snapshot maybe? |
This occurred again yesterday, seemingly randomly, had to shut everything down, remove the server data directory, and rebuild the cluster |
Something similar occurred on my setup on two separate occasions; the first time was valid, when I was running servers with very less memory (8GB). On both occasions, bumping up the memory to 64GB (!!) allowed the Nomad server ASG rotation to ultimately work. ref: https://discuss.hashicorp.com/t/sudden-increase-in-nomad-server-memory/39743/2 During the outage, when I manually add-one-delete-one node, memory consumption peaks upto 34 GB and over the course of about an hour goes down to the usual 10GB (or less) value. |
Is there any specific investigation going on with the high server memory usage for specific profiles of workloads? We have been able to contain the server memory (sort of) by running a |
We gave up on Nomad for queueing. I believe it's storing everything internally as json and memory growth seems exponential to the number of items you have queued, we wanted to queue up about 1,000,000 items. We've moved to an external JIT queue to feed Nomad, it maintains a queue of about 1,000,000 jobs in 300mb of memory |
We've got #15097 open which should fix a bug around garbage collection of evals that impacts batch workloads in particular. |
Fixed in #15097 which will ship in Nomad 1.5.0 (with backports) |
So the problem here could be that we just need more recources for each of the server nodes, we currently have 3 nodes, each with 2 cores, 16gb memory.
For a while they run just fine, then about once a week, they start to chew up all the available memory on each of their respective hosts. After which time, the only way I've found to recover the cluster, is to completely delete the serverdata directories on each of the server nodes, and re-bootstrap the ACL's.
Is this just a symptom of not enough hardware resources, or is there something else in play here?
Thank you,
Jonathan
The text was updated successfully, but these errors were encountered: