[Feature] limit cpu/memory resource on node

[ravirdv]

Currently when runningkubectl describe nodes, node capacity is shown same as host machine. It would be great to have a way to specify cpu/memory limit for node.

[iwilltry42]

I'll have a look into this once #12 is finished (and merged), since it's changing a lot.

[iwilltry42]

A brief thought on this. We can certainly set a memory limit on the node container that will have effect on the memory that can be used. But since the node containers have to run in privileged mode, they will still show you the whole memory available from the host system.

[ravirdv]

yes, that's what I observed. I tried setting mem_limits & cpus on node via docker-compose. I'd love to work on this, please let me know if you have any ideas on how to achieve this.

[iwilltry42]

A possibility would be to find out which requirements k3s has exactly that it needs to run in privileged mode inside docker. Maybe it's possible to only add it to specific groups (--group-add) or give it some specific capabilities (--cap-add) instead of going all-in with --privileged?
Docker docs: https://docs.docker.com/engine/reference/run/#runtime-privilege-and-linux-capabilities

[iwilltry42]

I'm pretty sure that we won't find a proper solution for this very soon 😞
Since the Kubernetes components kinda have to run privileged..

[iwilltry42]

Closing this, since there's currently no possibility to achieve this...

[louiznk]

Closing this, since there's currently no possibility to achieve this...

@iwilltry42 good news, there is solution, but it's in cAdvisor. I just make a PR and a related issue.
If it's validate I will make the other PR for integrate this to K3S (I don't know if it will be necessary to back port but it's a very small change) and after I can add this option on K3D for add the memory limit on docker.
I make some tests and it's working 🎉 (with a patched k3s and a custom k3d) (with a 32 GiB machine)

$ ./bin/k3d cluster create memlimit -p "80:80@server[0]" --image louiznk/k3s:v1.19.2-poc-mem  --agents 2 --no-lb --memory 3G          
INFO[0000] Created network 'k3d-memlimit'               
INFO[0000] Created volume 'k3d-memlimit-images'         
INFO[0001] Creating node 'k3d-memlimit-server-0'        
...
$ kubectl top node                                                                                                                        
NAME                    CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%   
k3d-memlimit-agent-0    23m          0%     208Mi           6%        
k3d-memlimit-agent-1    22m          0%     165Mi           5%        
k3d-memlimit-server-0   57m          0%     536Mi           17%   
$ kubectl get node -o=jsonpath="{.items[*]['status.capacity.memory']}"                                                                    
3Gi 3Gi 3Gi

[iwilltry42]

@louiznk , woah, you really went down a rabbit hole there!
Pretty cool, that you invested so much time into fixing this issue! :)
I directly subscribed to your PR and issue on cAdvisor and am looking forward to have this merged upstream.
Good job!