-
Notifications
You must be signed in to change notification settings - Fork 1.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fix: Docker container are using swap #6522
Conversation
How about disabling SWAP as an option, and let the user choose if they want to let the task swap. I am all for letting docker containers to use swap. At the most, make one the default behavior but still let the user choose what to do! |
I totally agree and also favor the approach to have an option to enable/disable swapping per job. With this PR I want to fix the discrepancy that according to the nomad documentation containers do not swap (https://www.nomadproject.io/docs/drivers/docker.html#memory) but they do. |
Could somebody rerun the CI check? |
@fho could you run |
@nickethier yes, done |
The go-dockerclient library that was used did not support to disable swapping for a docker-container. When MemorySwappiness was set to 0, the value was not passed to the docker-client. Therefore the swappiness setting of the parent cgroup was used. Version 1.4.1 contains the change fsouza/go-dockerclient#776. MemorySwappiness is changed to a pointer type. If MemorySwappiness is null, the value will not be passed. If it's 0, swappiness is set to 0. This will allow to disable swapping for docker containers.
The API of go-dockerclient changed in the new version. PidsLimit and MemorySwappiness are now pointers. Because MemorySwappiness is a pointer and the default is nil, we do not have to assign it anymore to deactivate setting swappiness.
According to the nomad documentation docker containers cannot use swap. This was not the case. Container were swapping. The swappiness value for docker containers was unset. This had the effect that the swappiness value is inherited from the parent cgroup, where swappiness is enabled by default. Setting memory.memsw.limit_in_bytes (MemorySwap) to the same value then memory.limit_in_bytes (task.Resources.LinuxResources.MemoryLimitBytes) is not sufficient to disable swapping of a container. If the host system runs out of memory, the container will start swapping to make more RAM available for other processes if swappiness is not set to 0.
I force pushed to improve the commit messages |
@nickethier could you provide feedback for the PR? thanks! :-) |
This is really hurting us with |
Superceeded by #7550 . Thanks @sirkjohannsen - the PR was very descriptive and helped me understand the situation more! |
I'm going to lock this pull request because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active contributions. |
This fixes the issue: #6085
To reproduce that a docker container swap when
memory.memsw.limit_in_byte
=memory.limit_in_bytes
, do the following:--memory-swap
and--memory
, set swappiness to the max. value to make swapping happen faster:docker run -i -t --memory-swap=1G -m 1G --memory-swappiness=100 debian:buster bash
apt-get update && apt-get install stress -y
stress -m 1 --vm-bytes 1048576000
--vm-bytes
set to all or little bit then the available RAMmemory.stat
file of the docker container cgroup, theswap
value will start to show a value != 0