CGroup Memory Resource Controller for kubelet node in a container

[louiznk]

Trouble

When you use Kubernetes in docker (ie: kind or k3d or ...) and you add some memory limit on your container, the kubelet component doesn't use this limit.
So for example when you start a test Kubernetes cluster with 3 nodes on 3 containers on a computer with 32 Gi for your cluster you have 3 nodes of 32Gi, even if you set some limit on every container.

This is an issue at least for :

• kind : Configure capacity of the worker nodes kubernetes-sigs/kind#877
• k3d : [Feature] limit cpu/memory resource on node k3d-io/k3d#13

How to reproduce

For this example, I will use K3S container in version v1.19 (but you can use every version) and a variante of ubuntu 20.4 computer with 32 GiB and 12 processors and docker community v19.03.12.
The container is launch with a memory limit of 2 GiB.

Start k3s in docker

$ docker run --privileged --rm -d -p 6443:6443 -p 80:80 -p 443:443 --memory=2g --memory-swap=-1 rancher/k3s:v1.19.2-k3s1 server           
5ecde3a8c64defdb9452f62bc940947c24b26391e8c879a08752a9810886c964

✅ the container is started

Check the memory limit of the container

$ docker stats --no-stream 
CONTAINER ID        NAME                CPU %               MEM USAGE / LIMIT   MEM %               NET I/O             BLOCK I/O           PIDS
5ecde3a8c64d        naughty_lumiere     15.34%              667.1MiB / 2GiB     32.57%              115MB / 546kB       124MB / 82.6MB      234

✅ the container use 2 GiB

Check the memory available for kubelet

Open a shell in the container

$ docker exec -it 5ecde3a8c64defdb9452f62bc940947c24b26391e8c879a08752a9810886c964 sh

/ # grep MemTotal /proc/meminfo 
MemTotal:       32486464 kB

❗ the system memory is 32 GiB

/ # cat /sys/fs/cgroup/memory/memory.limit_in_bytes
2147483648

✅ good, the memory limit is 2 GiB

/ # kubectl get node -o=jsonpath="{.items[*]['status.capacity.memory']}"
32486464Ki

/ # kubectl top node
NAME           CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%   
5ecde3a8c64d   177m         1%     686Mi           2%        

❌ but for kubelet the memory available is 32 GiB, kubelet doesn't use the memory limit of the docker container.

The cause

The function GetMachineMemoryCapacity (from machine.go) is use by kubelet and this function doesn't take care of the cgroup limit

[iwankgb]

See: #2699 (comment)