cpuset support with cgroupsv2

[notnoop]

Nomad 1.1 Reserved CPUs feature relies on cpuset cgroup capabilities, and uses a technique no longer supported in cgroup-v2.

The reserved CPUs are dedicated to their tasks - other tasks may not access them. Nomad achieve this by using the cpuset cgroups controller. Nomad creates a /nomad/shared cgroup for all non-reserving tasks, which starts unconstrained. If a reserved CPU task is placed on the host, Nomad removes the reserved core from this shared cgroup - constraining already running tasks from running on the core.

Thus, non-reserving tasks belong to at least two cgroups: the nomad/shared cpuset cgroup as well as cgroup that's created by the task driver. For example, a docker task process has the following cgroup assignment (note the cpuset value):

cat /proc/11050/cgroup
12:devices:/docker/10234c078fce6da0cb3e58c9da71a01852f0241bd5ce6b0e9fd19ec296e1aea0
11:pids:/docker/10234c078fce6da0cb3e58c9da71a01852f0241bd5ce6b0e9fd19ec296e1aea0
10:net_cls,net_prio:/docker/10234c078fce6da0cb3e58c9da71a01852f0241bd5ce6b0e9fd19ec296e1aea0
9:freezer:/docker/10234c078fce6da0cb3e58c9da71a01852f0241bd5ce6b0e9fd19ec296e1aea0
8:rdma:/
7:perf_event:/docker/10234c078fce6da0cb3e58c9da71a01852f0241bd5ce6b0e9fd19ec296e1aea0
6:cpuset:/nomad/shared
5:cpu,cpuacct:/docker/10234c078fce6da0cb3e58c9da71a01852f0241bd5ce6b0e9fd19ec296e1aea0
4:memory:/docker/10234c078fce6da0cb3e58c9da71a01852f0241bd5ce6b0e9fd19ec296e1aea0
3:blkio:/docker/10234c078fce6da0cb3e58c9da71a01852f0241bd5ce6b0e9fd19ec296e1aea0
2:hugetlb:/docker/10234c078fce6da0cb3e58c9da71a01852f0241bd5ce6b0e9fd19ec296e1aea0
1:name=systemd:/docker/10234c078fce6da0cb3e58c9da71a01852f0241bd5ce6b0e9fd19ec296e1aea0
0::/system.slice/containerd.service

This technique relies on cgroup-v1 hierarchy flexibility. cgroup-v2 no longer permits a process to belong to multiple cgroups! We will need to rethink this mechanism.

Cgroups v2 unified hierarchy
In cgroups v1, the ability to mount different controllers against
different hierarchies was intended to allow great flexibility for
application design. In practice, though, the flexibility turned
out to be less useful than expected, and in many cases added
complexity. Therefore, in cgroups v2, all available controllers
are mounted against a single hierarchy.

From https://man7.org/linux/man-pages/man7/cgroups.7.html

Oddly enough, Nomad uses the same pattern and creates a cpuset cgroup for reserving tasks, instead of setting cpuset cgroup in the driver-managed cgroup. I'm unsure of what the motivation; if deemed unnecessary, we can have the drivers set the cpuset settings appropriately.

Resources

• Control Group v2 administrator guide
• Control Group v2 man page
• FODSEM talk slides

[tgross]

I think this can be solved in a way that still works with Docker by leaning on the --cgroup-parent flag that we can set in the HostConfig. This field is available in the Docker API since API v1.18 (at least... that's as far back as their docs go), and that's been available in Docker since Docker 1.10 shipped in early 2016.

This way we don't need to move the process into the new cgroup and can instead create it under the correct hierarchy and only change the cgroup controllers.

• We set up the /nomad cgroup (rooted under /sys/fs/cgroup/) with no processes in it (this might be named differently if the cgroup_parent name is set on the client).
• We set up the /nomad/shared cgroup with cpuset set to all CPUs.
• We set up the /nomad/reserved cgroup with no cpuset set.
• When a task without cores is created it's created in the /nomad/shared/<alloc ID>/<task name> cgroup without any cpuset (cgroup_parent = /nomad/shared).
• When a task with cores is created, it's created in the /nomad/reserved/<alloc ID>/<task name> cgroup with the appropriate cpuset (cgroup_parent = /nomad/reserved). And we update the /nomad/shared cgroup to remove that cpu.

So for example, assume we have 2 tasks with cpu = 1024 and memory = 1024 on an 8-core machine. The cgroup hierarchy looks like this:

/nomad
└── /nomad/shared: cpuset:0-7
    ├──  /nomad/shared/alloc_id_1/taskname: cpu:1024,mem:1024 (inherits cpuset:0-7)
    └──  /nomad/shared/alloc_id_2/taskname: cpu:1024,mem:1024 (inherits cpuset:0-7)
└── /nomad/reserved:
    (no children)

Then we add a new task with cores = 1 and memory = 1024. The new cgroup hierarchy looks like this:

/nomad
└── /nomad/shared: cpuset:1-7
    ├──  /nomad/shared/alloc_id_1/taskname: cpu:1024,mem:1024 (inherits cpuset:1-7)
    └──  /nomad/shared/alloc_id_2/taskname: cpu:1024,mem:1024 (inherits cpuset:1-7)
└── /nomad/reserved:
    └──  /nomad/reserved/alloc_id_3: cpuset:0,mem:1024

As far as I can tell this will work just fine for both cgroups v1 and v2, but I do want to do some hands-on testing to be sure. It looks like this approach was mentioned as a comment to the original design doc NMD-098 (internal doc) but didn't get followed-up on. 🤷

There's some backwards compatibility concerns here for upgrading existing clients that have running cgroup v1 containers. In the design doc we said:

During inplace client upgrades to 1.1 tasks will be running outside the desired cgroup. As part of the upgrade path, the Nomad client will create the necessary cpuset cgroups and move the task pids into them so as to not disrupt any reserved core workloads that are placed on the client in the future.

It looks like this works by unconditionally writing the PID to the new cgroup in setCPUSetCgroup. For this new hierarchy we'd need to:

• write a new cgroup at the expected path if it doesn't exist
• set the values to match all the current cgroups
• move the PID into that cgroup

[tgross]

Well it turns out we can't have an intermediate hierarchy on the cgroup like /nomad/shared/alloc_id_1/taskname if we want to pick up the cpuset from /nomad/shared. ref cgroup v2 docs:

Enabling a controller in a cgroup indicates that the distribution of the target resource across its immediate children will be controlled.

So at the very least we'll end up with a flat per-task cgroup hierarchy, which is what we have already for everything other than the cpuset controller. Not a big deal but we'll have to adjust the logic in client/lib/cgutil/cpuset_manager_linux.go to use a flat namespace. (It's not clear to me yet if this applies to v1, but in any case we'd want to use the most-restricted option.)

[github-actions]

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.