Does k3s support cgroup v2 or will it support it?

[davidshen84]

Hi,

I enabled cgroup v2 with kernelCommandLine = cgroup_no_v1=all systemd.unified_cgroup_hierarchy=1 on my WSL system, then I found my k3s server won't start. The error in the log is:

"Failed to start ContainerManager" err="system validation failed - wrong number of fields (expected 6, got 7)"

The server was restored after I reverted the kernel command line configuration.

The upstream k8s already supported cgroup v2 at v1.25. How come k3s still doesn't support it at v1.3x? What is the roadmap to align with upstream?

Of course, it might be cgroup v2 doesn't work correctly in WSL and failed k3s.

[brandond]

Yes, k3s supports cgroup v2, same as upstream Kubernetes. No special conditions necessary, it should work out of the box. I suspect there is something wrong with your WSL kernel configuration.

[davidshen84]

I did not compile my kernel. I use the official kernel with Ubuntu 22.04. If I install k3s when the system uses *cgroupv2*, I get the "Failed to start ContainerManager" error message. I tried multiple times. However, if I install k3s when the system uses "cgruopv1", the k3s service starts successfully. Then I shutdown WSL and configured it to use *cgroupv2*. After that, k3s runs successfully. I did not use a `/etc/rancher/k3s/config.yaml` file. This is how I configure WSL to use cgroupv2: ``` [wsl2] swap=32GB kernelCommandLine = cgroup_no_v1=all systemd.unified_cgroup_hierarchy=1 ```

…

On Wed, 23 Oct 2024 at 17:50, Brad Davidson ***@***.***> wrote: Yes, k3s supports cgroup v2, same as upstream Kubernetes. I suspect there is something wrong with your WSL kernel configuration. — Reply to this email directly, view it on GitHub <#11149 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAQBTMLES6XV6NOOGM45A3Z45BKVAVCNFSM6AAAAABQOBQ3BKVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTCMBSGU2DGNY> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[brandond]

Ubuntu 22.04 should use cgroupv2 by default. What steps are you using to validate your cgroup configuration? What's the output of grep cgroup /proc/mounts?

[davidshen84]

Sorry, I meant to say I am using Ubuntu 24.04 LTS. Ubuntu official kernel might be using cgroupv2 by default, Ubuntu in WSL 2 may not.

On my Windows 11 24H2, with everything up to date. My kernel version is 5.15.153.1-microsoft-standard-WSL2, and without kernelCommandLine configuration, grep cgroup /proc/mounts returns as the follows:

grep cgroup /proc/mounts
tmpfs /sys/fs/cgroup tmpfs ro,nosuid,nodev,noexec,size=4096k,nr_inodes=1024,mode=755 0 0
cgroup2 /sys/fs/cgroup/unified cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate 0 0
cgroup /sys/fs/cgroup/cpuset cgroup rw,nosuid,nodev,noexec,relatime,cpuset 0 0
cgroup /sys/fs/cgroup/cpu cgroup rw,nosuid,nodev,noexec,relatime,cpu 0 0
cgroup /sys/fs/cgroup/cpuacct cgroup rw,nosuid,nodev,noexec,relatime,cpuacct 0 0
cgroup /sys/fs/cgroup/blkio cgroup rw,nosuid,nodev,noexec,relatime,blkio 0 0
cgroup /sys/fs/cgroup/memory cgroup rw,nosuid,nodev,noexec,relatime,memory 0 0
cgroup /sys/fs/cgroup/devices cgroup rw,nosuid,nodev,noexec,relatime,devices 0 0
cgroup /sys/fs/cgroup/freezer cgroup rw,nosuid,nodev,noexec,relatime,freezer 0 0
cgroup /sys/fs/cgroup/net_cls cgroup rw,nosuid,nodev,noexec,relatime,net_cls 0 0
cgroup /sys/fs/cgroup/perf_event cgroup rw,nosuid,nodev,noexec,relatime,perf_event 0 0
cgroup /sys/fs/cgroup/net_prio cgroup rw,nosuid,nodev,noexec,relatime,net_prio 0 0
cgroup /sys/fs/cgroup/hugetlb cgroup rw,nosuid,nodev,noexec,relatime,hugetlb 0 0
cgroup /sys/fs/cgroup/pids cgroup rw,nosuid,nodev,noexec,relatime,pids 0 0
cgroup /sys/fs/cgroup/rdma cgroup rw,nosuid,nodev,noexec,relatime,rdma 0 0
cgroup /sys/fs/cgroup/misc cgroup rw,nosuid,nodev,noexec,relatime,misc 0 0
cgroup /sys/fs/cgroup/systemd cgroup rw,nosuid,nodev,noexec,relatime,xattr,name=systemd 0 0

Also, stat -fc %T /sys/fs/cgroup returns tmpfs, which indicates it uses cgroupv1.

With kernelCommandLine = cgroup_no_v1=all systemd.unified_cgroup_hierarchy=1, grep cgroup /proc/mounts returns

cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate 0 0

And stat -fc %T /sys/fs/cgroup returns cgroup2fs, which indicates it is using cgroupv2.

[brandond]

The error is from core Kubernetes, not K3s itself, ref: https://github.com/kubernetes/kubernetes/blob/v1.31.2/pkg/kubelet/cm/container_manager_linux.go#L155-L158
https://github.com/kubernetes/mount-utils/blob/master/mount_linux.go#L752-L756

This indicates that there is an entry in /proc/mounts with 7 fields instead of the expected 6. This isn't really something that we can fix here in this project, you'll need to find and address the unexpected entry on your host.

[davidshen84]

With *cgroupv2* enabled, the k3s service fails to start every time after installation. However, if I shutdown WSL and restart it, the k3s service will start, but not always.

…

On Wed, 23 Oct 2024 at 20:30, Xi Shen ***@***.***> wrote: I did not compile my kernel. I use the official kernel with Ubuntu 22.04. If I install k3s when the system uses *cgroupv2*, I get the "Failed to start ContainerManager" error message. I tried multiple times. However, if I install k3s when the system uses "cgruopv1", the k3s service starts successfully. Then I shutdown WSL and configured it to use *cgroupv2*. After that, k3s runs successfully. I did not use a `/etc/rancher/k3s/config.yaml` file. This is how I configure WSL to use cgroupv2: ``` [wsl2] swap=32GB kernelCommandLine = cgroup_no_v1=all systemd.unified_cgroup_hierarchy=1 ``` On Wed, 23 Oct 2024 at 17:50, Brad Davidson ***@***.***> wrote: > Yes, k3s supports cgroup v2, same as upstream Kubernetes. I suspect there > is something wrong with your WSL kernel configuration. > > — > Reply to this email directly, view it on GitHub > <#11149 (comment)>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/AAAQBTMLES6XV6NOOGM45A3Z45BKVAVCNFSM6AAAAABQOBQ3BKVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTCMBSGU2DGNY> > . > You are receiving this because you authored the thread.Message ID: > ***@***.***> >

[davidshen84]

Strange... ``` cat /proc/mounts | while IFS= read -r l; do echo $l | wc -w done | uniq ``` It shows all lines have six fields. The fields are parsed at https://github.com/kubernetes/mount-utils/blob/05c86c7f6d53cf740ce87969fa862d479bdfef38/mount_linux.go#L752, and it looks good to me.

…

On Thu, 24 Oct 2024 at 10:03, Brad Davidson ***@***.***> wrote: The error is from core Kubernetes, not K3s itself, ref: https://github.com/kubernetes/kubernetes/blob/v1.31.2/pkg/kubelet/cm/container_manager_linux.go#L155-L158 https://github.com/kubernetes/mount-utils/blob/master/mount_linux.go#L752-L756 This indicates that there is an entry in /proc/mounts with 7 fields instead of the expected 6. This isn't really something that we can fix in K3s. — Reply to this email directly, view it on GitHub <#11149 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAQBTI3F56ZLA3L5GRHXPLZ5ATLBAVCNFSM6AAAAABQOBQ3BKVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTCMBTGQ2TSOI> . You are receiving this because you authored the thread.Message ID: ***@***.***>