Exec/Java: Default OOMScoreAdjust causes memory limit to be ignored

[MikeN123]

Nomad version

Nomad v1.1.0 (2678c3604bc9530014208bc167415e167fd440fc)

Operating system and Environment details

Debian Linux 10

Issue

The default setting of OOMScoreAdjust=-1000 in the systemd unit file makes sure Nomad will never be killed by the OOM killer. But, this setting in inherited by child executors and processes, causing the cgroup memory limit not to be enforced. Instead, the OOM killer will log error messages to syslog like crazy, filling up the disk.

Reproduction steps

1. Use the default systemd unit file.
2. Use the exec or java driver and start a task that runs over its memory limit.

Expected Result

OOM killer should kick in and kill the process.

I believe the drivers should reset the OOMScoreAdjust for child processes to some sensible value. The only other workaround (at least for the java driver) is to not set an OOMScoreAdjust on the nomad process, but that has the disadvantage that nomad may be killed if the host experiences memory pressure.

Actual Result

the task is not killed, as the OOM killer is unable to kill any process due to the OOMScoreAdjust.

This was also noted in the issue where the default was suggested, but apparently never picked up: #6672 (comment)
The message logged to syslog looks like this:

May 24 21:31:47 d-store-01 kernel: [1775404.602255] Memory limit reached of cgroup /nomad/11bda2ef-ac35-d99a-aaac-d7fb38c21966
May 24 21:31:47 d-store-01 kernel: [1775404.602256] memory: usage 524356kB, limit 524288kB, failcnt 202
May 24 21:31:47 d-store-01 kernel: [1775404.602256] memory+swap: usage 0kB, limit 9007199254740988kB, failcnt 0
May 24 21:31:47 d-store-01 kernel: [1775404.602256] kmem: usage 2980kB, limit 9007199254740988kB, failcnt 0
May 24 21:31:47 d-store-01 kernel: [1775404.602257] Memory cgroup stats for /nomad/11bda2ef-ac35-d99a-aaac-d7fb38c21966: cache:0KB rss:521340KB rss_huge:65536KB shmem:0KB mapped_file:0KB dirty:0KB writeback:0KB inactive_anon:260816KB active_anon:260544KB inactive_file:0KB active_file:0KB unevictable:0KB
May 24 21:31:47 d-store-01 kernel: [1775404.602259] Tasks state (memory values in pages):
May 24 21:31:47 d-store-01 kernel: [1775404.602259] [  pid  ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
May 24 21:31:47 d-store-01 kernel: [1775404.602264] [  20937] 65534 20937   915250   137172  1466368        0         -1000 java
May 24 21:31:47 d-store-01 kernel: [1775404.602264] Out of memory and no killable processes...
May 24 21:31:47 d-store-01 kernel: [1775404.602342] http-nio-8082-e invoked oom-killer: gfp_mask=0x6000c0(GFP_KERNEL), nodemask=(null), order=0, oom_score_adj=-1000
May 24 21:31:47 d-store-01 kernel: [1775404.602343] http-nio-8082-e cpuset=shared mems_allowed=0
May 24 21:31:47 d-store-01 kernel: [1775404.602344] CPU: 0 PID: 20986 Comm: http-nio-8082-e Not tainted 4.19.0-16-amd64 #1 Debian 4.19.181-1
May 24 21:31:47 d-store-01 kernel: [1775404.602344] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 05/28/2020
May 24 21:31:47 d-store-01 kernel: [1775404.602344] Call Trace:
May 24 21:31:47 d-store-01 kernel: [1775404.602345]  dump_stack+0x66/0x81
May 24 21:31:47 d-store-01 kernel: [1775404.602346]  dump_header+0x6b/0x283
May 24 21:31:47 d-store-01 kernel: [1775404.602347]  out_of_memory.cold.33+0xa/0x7e

[notnoop]

Hi @MikeN123 ! Thanks for raising it. This is indeed a bad behavior and should be fixed. We'll aim to ship a fix soon.

[MikeN123]

Thanks!

I also noted that the PR for memory oversubscription said something on oom_score_adj, not sure if that was ever completed. You may also want to consider varying the oom_score_adj based on the task being oversubscribed or not, K8s does something like that, but maybe that's complicated also keeping in mind Docker and other drivers...

[github-actions]

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.