[EKS] [request]: Use systemd cgroupdriver by default

[Callisto13]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Tell us about your request
Please start the docker daemon and the kubelet with the cgroupdriver set to systemd when systemd is the init system.

Which service(s) is this request for?
EKS

Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?
This is standard in the community kubernetes/kubeadm#1394 (comment).

Are you currently working around this issue?
We are using cloudconfig/userdata to edit the kubelet.yaml and /etc/docker/daemon.json (in the case of EKS AMIs), or /etc/sysconfig/docker (in the case of accelerated AL2-GPU AMIs, as the daemon.json is removed there).

Additional context
I believe this is soon to happen in standard EKS AMIs awslabs/amazon-eks-ami#490, but the builder for the accelerated AMIs does not appear to be public. (If there is a public accelerated AMI builder somewhere and it is just particularly well hidden, let me know and I will go PR something 😄 .)

[reegnz]

I've reopened the change in awslabs/amazon-eks-ami#593, as the original was actually reverted by someone because the change initially broke eksctl. Now that eksctl can work with this change, there should be no problem with making systemd the default cgroup driver in amazon-eks-ami.

[reegnz]

@Callisto13 maybe another request issue would be in order to request the GPU AMI-s to also become open-source.

[josephprem]

This is what I have put in my userdata

Docker options set with sysconfig environment file

cat << EOF >> /etc/sysconfig/docker
OPTIONS='--icc=false --exec-opt=native.cgroupdriver=systemd'
EOF
systemctl restart docker

Systemd does not support appending environment variables. Add through systemd drop-ins

sed -i 's/KUBELET_EXTRA_ARGS/KUBELET_EXTRA_ARGS $EXTENDED_KUBELET_ARGS/' /etc/systemd/system/kubelet.service
cat << EOF > /etc/systemd/system/kubelet.service.d/9999-extended-kubelet-args.conf
[Service]
Environment='EXTENDED_KUBELET_ARGS=--register-with-taints="is_edge=true:NoExecute" --cgroup-driver=systemd'
EOF
systemctl daemon-reload

[josephprem]

Hello folks,
Now that containerd can be enabled as cri, looks like we will need to handle ourselves to enable cgroup in /etc/containerd/config.toml.
It is pretty messy when EKS does not support out of the box to enable these production settings

[stevehipwell]

Is there a valid PR for this open? I can see that awslabs/amazon-eks-ami#593 is blocked over concerns over users of older eksctl versions. Maybe a PR to only enable this for containerd by adding a kubelet-config.containerd.json file and using bootstrap.sh to swap this in if the containerd runtime is enabled? I assume that older versions of eksctl don't support containerd?

[josephprem]

@stevehipwell agree, no clear view if this issue is in backlog.
Not sure why folks talk about eksctl when these requirements should be first addressed natively in EKS AMI's

[josephprem]

checkout how I enable cgroup-driver to systemd when containerd is used as CRI
awslabs/amazon-eks-ami#593 (comment)

[reegnz]

@stevehipwell agree, no clear view if this issue is in backlog.
Not sure why folks talk about eksctl when these requirements should be first addressed natively in EKS AMI's

That's exactly what I've been saying... 😒 eksctl is downstream, shouldn't block upstream work.

[reegnz]

Can someone open a new PR for this though? The containerd stuff is different enough from my PR to be worked on separately, and I also don't have any capacity to deal with this.

[josephprem]

There are three tickets already on this subject. Not sure if a new one should be opened

1. [EKS] [kubelet/docker]: Change cgroup driver to systemd #1413
2. [EKS] [request]: Use systemd cgroupdriver by default #1210
3. systemd cgroup driver is unsupported #1118

[stevehipwell]

@reegnz I can open a PR to just make the changes for containerd.

[reegnz]

@josephprem I don't want a new ticket, just a separate PR for containerd. Ticket !== PR.

@stevehipwell that would be great!

[stevehipwell]

@mikestef9 there is an open PR (awslabs/amazon-eks-ami#717) to add this for containerd, which hasn't had a single maintainer response, and the Docker version (awslabs/amazon-eks-ami#593) appears to be stuck due to concerns with eksctl. Could we at least get some sort of feedback on the status of this? It's quite clear in the Kubernetes documentation that the systemd driver should be used when systemd is the init system so unless there is a good reason not to do this it should be a priority, especially now that resources not IPs will be the limiting factor for nodes. Furthermore when containerd was introduced as a brand new runtime why was it was setup to use cgroupfs instead of the best practice systemd?

[stevehipwell]

@mikestef9 or any of the other maintainers could we get a comment on this?

[stevehipwell]

@mikestef9 do you know where this has got to? The AMI PRs had responses a couple of months ago but there has been nothing since.

I'm also interested in the status of cgroup v2 in the EKS optimised AMI? Do we need a new issue opened for this (#1535 is for ECS and I don't think it's AL2 based)?

[dims]

• Use systemd as the containerd cgroup driver awslabs/amazon-eks-ami#717 has landed.
• Change cgroup driver to systemd awslabs/amazon-eks-ami#593 (comment) is a WONT-FIX.

yes, please open another issue for cgroup v2

[stevehipwell]

See also.

• Added runtime.slice for containerd runtime awslabs/amazon-eks-ami#1051