cpuset disappears from /sys/fs/cgroup/cgroup.subtree_control after adding it

[harche]

After the OS boots and the kubelet starts, it adds cpuset to /sys/fs/cgroup/cgroup.subtree_control. However, it seems that kubelet calls the runc method setUnitProperties, which in turn calls SetUnitPropertiesContext from the go-systemd library. At this point, we notice that the cpuset, which was already present, gets removed (for reasons unknown). This causes kubelet to restart, but this time, the cpuset gets added back and remains in /sys/fs/cgroup/cgroup.subtree_control.

More details are in the description of kubernetes/kubernetes#122955

On my local OCP node, I was able to observe this issue as well,

[root@ip-10-0-2-54 ~]# systemctl restart kubelet
[root@ip-10-0-2-54 ~]# cat /sys/fs/cgroup/cgroup.subtree_control
cpu io memory hugetlb pids rdma misc
[root@ip-10-0-2-54 ~]# cat /sys/fs/cgroup/cgroup.subtree_control
cpu io memory hugetlb pids rdma misc
[root@ip-10-0-2-54 ~]# cat /sys/fs/cgroup/cgroup.subtree_control
cpu io memory hugetlb pids rdma misc
[root@ip-10-0-2-54 ~]# cat /sys/fs/cgroup/cgroup.subtree_control
cpu io memory hugetlb pids rdma misc
[root@ip-10-0-2-54 ~]# cat /sys/fs/cgroup/cgroup.subtree_control
cpu io memory hugetlb pids rdma misc
[root@ip-10-0-2-54 ~]# cat /sys/fs/cgroup/cgroup.subtree_control
cpu io memory hugetlb pids rdma misc
[root@ip-10-0-2-54 ~]# cat /sys/fs/cgroup/cgroup.subtree_control
cpu io memory hugetlb pids rdma misc
[root@ip-10-0-2-54 ~]# cat /sys/fs/cgroup/cgroup.subtree_control
cpu io memory hugetlb pids rdma misc
[root@ip-10-0-2-54 ~]# cat /sys/fs/cgroup/cgroup.subtree_control
cpu io memory hugetlb pids rdma misc
[root@ip-10-0-2-54 ~]# cat /sys/fs/cgroup/cgroup.subtree_control
cpu io memory hugetlb pids rdma misc
[root@ip-10-0-2-54 ~]# cat /sys/fs/cgroup/cgroup.subtree_control
cpuset cpu io memory hugetlb pids rdma misc                      <<<<<< cpuset is added
[root@ip-10-0-2-54 ~]# cat /sys/fs/cgroup/cgroup.subtree_control    
cpu io memory hugetlb pids rdma misc                             <<<<<< gone here. 
[root@ip-10-0-2-54 ~]# cat /sys/fs/cgroup/cgroup.subtree_control
cpuset cpu io memory hugetlb pids rdma misc                      <<<<<< kubelet restart, and it is back again. 
[root@ip-10-0-2-54 ~]# cat /sys/fs/cgroup/cgroup.subtree_control
cpuset cpu io memory hugetlb pids rdma misc
[root@ip-10-0-2-54 ~]# cat /sys/fs/cgroup/cgroup.subtree_control
cpuset cpu io memory hugetlb pids rdma misc
[root@ip-10-0-2-54 ~]# cat /sys/fs/cgroup/cgroup.subtree_control
cpuset cpu io memory hugetlb pids rdma misc
[root@ip-10-0-2-54 ~]# cat /sys/fs/cgroup/cgroup.subtree_control
cpuset cpu io memory hugetlb pids rdma misc

[harche]

I have observed this happening only on cgroup v2 systems.

[haircommander]

I think kubernetes/kubernetes#125923 fixes this

[harche]

/close as it seems the bug was in the kubelet code.