[FEATURE] Resources requests and limits for each node

[konradmalik]

This is rather a suggestion/question that I can start to work on and open a PR some time in the future if there is a need or it would be useful to have. Reactions are welcome 😉

Basically the feature in question is to be able to specify requests and limits in terms of cpu and memory for each docker container that runs a k3d node. Similar stuff that is possible with docker-compose for example.

First iteration would be to specify server and agent separately, regardless of their number. So no custom setups like 2 CPU for server 1 and 3 CPU for server 2, only things like: 1 CPU for servers, 3 CPU for agents. Granular control could be implemented later, or via adding node by node with different specs.
Specification of those limits would naturally be implemented in the yaml config and cli config defaulting to no-limits.

As another step (after the first implementation because I'm not sure how to do this or even if this is possible), this info could be propagated somehow to the k3s inside the dockers. This is also related to the multi-cluster setup. Not sure if I'm right but the last time I checked, one of the limitations of the k3d vs k3s was that kubectl describe node gives no info on the resources available on the node, so if I'm right nodes can be easily overprovisioned. Maybe the provided limits could be used somehow to force the k3s inside the container to acknowledge the resources? Not sure, this one is just an idea, something to investigate.

What do you think about the limits implementation?

[iwilltry42]

Hi @konradmalik , I'd love to see this and even more I'd love to accept a PR that makes this possible.
Unfortunately though we failed the last time we tried it, because we cannot actually impose those limits on the containers themselves, as they run in privileged mode, so they will always see the full host resources. We would need to investigate in full detail, exactly which privileges k3s in a container requires to run properly and hope that we can only set those (cap-add) and that they do not include the cgroup stuff required for setting resource limitations.
Second try was to just make cadvisor think it has less resources, as it's the source of truth for Kubernetes. There was an issue opened on cadvisor, but nothing was done so far to achieve this.
Refs:

• k3d issue: [Feature] limit cpu/memory resource on node #13
• k3d PR: 🗜️ Add docker limit memory option #316
• cAdvisor issue: CGroup Memory Resource Controller for kubelet node in a container google/cadvisor#2698
• cAdvisor PR: Feat: use memory limit from cgroup when set and lower google/cadvisor#2699

I'm sure, @louiznk would love to help to achieve this :)

[konradmalik]

Ha, did not expect such problems honestly. But I'll definitely investigate and who knows, maybe succeed at least on a part of it.
But I guess I need something explained - I don't really understand why #316 was not a (partial) success?
It is true that k3s in docker does not see those limits, but I thought it is always the case?
When you investigate available memory etc. inside a pod, you'll see the node's full memory, not the limits imposed by kubernetes. Nevertheless, even though pod sees for example 64GB of memory, it won't use more than 1GB if we specified that, right? Same thing is with docker containers when deployed using docker-compose. They allways "see" all resources, just won't use it all if limits were specified.

So I think there are 2 parts of this feature:

• implement basic docker limits on cpu and memory. Those will be respected, meaning that nodes won't use more memory and cpu that we specify, a hard limit. This is the easy part.
• this will cause confusion to k3s, it will see all memory so the services will be overprovisioned, but docker won't let it allocate more than we specified so there may be OOM's. This is the second (hard) part of the assignment that you mentioned and that needs to be investigated in detail on how to "tell" k3s that it has less resources.

Am I wrong somewhere?

[louiznk]

• this will cause confusion to k3s, it will see all memory so the services will be overprovisioned, but docker won't let it allocate more than we specified so there may be OOM's. This is the second (hard) part of the assignment that you mentioned and that needs to be investigated in detail on how to "tell" k3s that it has less resources.

Hi @konradmalik
The difficulty is as you write how to "tell" k3s that it has less resources. . The memory and cpu available for a node is give by kubelet who uses cAdvisor for collecting system informations. Unfortunately for this special case where kubelet runs in docker cAdvisor collects information from the system host and not docker's cgroup constraints. So if you start kubelet (k3s) in a container with memory limit (or cpu limit) your cluster won't know this.

I tried a hack on cAdvisor (for memory limit only). It's works but change the behavior of cAdvisor can have too many impacts.
There is a lot of discussion on this PR and an alternative is to add some flag to kubelet to specify to ask to cAdvisor to use capacity as specified in cgroup but I don't test it.

[konradmalik]

Thank you for clarification @louiznk 👍 , I have a need for this functionality in k3d so I'll def. investigate further and post back here as soon as I'll have some ideas/working demo.

[iwilltry42]

Leaving this here: k3s-io/k3s#3005 (https://twitter.com/_AkihiroSuda_/status/1366689973672402945)