-
Notifications
You must be signed in to change notification settings - Fork 1.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Nomad task got OOM killed when it was using only ~70% of it's MemoryMB limit #4495
Comments
@fho thanks for the details. We plan to fix executor memory utilization in the upcoming release, 13MB is rather high. |
@preetapan Let's assume the memory consumption of |
@fho see comments me and my co-worker made already about why the executor and script checks have to be in the same cgroup. #4491 (comment) There's always going to be some amount of overhead from using the executor, and we will address that with a TBD mechanism - will likely either account for that when creating the container, or use soft limits. |
I don't understand yet why they have to be in the same cgroup.
thanks a lot |
I'm having a hard time understanding this. I have a container with 600MB limit. A java process with use heap+non-heap used of ~360. And its getting oom killed every 10 minutes or so. I can't be that nomad services are using 240MB? And if not, how can i tell why the processing is getting killed? |
I would just like to add that with nomad 0.9 the resource footprint of nomad processes within the cgroup have increased even more. |
Wanted to clarify the behavior of Nomad 0.9:
|
I'm closing this ticket as exec driver has been significantly changed since 0.8 and I believe the notes here are either addressed or no longer relevant. I'd encourage users experiencing memory issues to create a new issue against 0.10. Since my last May 31, comment we made the following changes:
Please let us know of any issues you see and we will follow up. Thanks! |
I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues. |
This is the same issue then described in #4491 but as bug report.
Nomad version
Nomad v0.8.3
Issue
We have a nomad job that runs an application called
claimsearch-service
with the exec executor.The memory limit is set to 50MiB in the nomad job file.
The application got OOM killed when it was only using 35,35MB RSS.
In the memory cgroup were the following processes with the following RSS usage:
Expected behaviour
MemoryMB
parameter in theResources
Stanza of the nomad job file.That the memory consumption of other processes are accounted into the memory limit is non-intuitive, it's not documented and it makes it difficult to calculate the correct Memory limit value for a task.
See also: #4491
OOM kill Kernel log
Job file
The full job file can be found at: http://dpaste.com/05YWFVW
The text was updated successfully, but these errors were encountered: