Fix DevicesSets being removed when cpusets are reloaded with cgroup v2

[TrueBrain]

This meant that if any allocation was created or removed, all active DevicesSets were removed from all cgroups of all tasks.

This was most noticeable with "exec" and "raw_exec", as it meant they no longer had access to /dev files.

Fixes #12877, and possibly a bunch of other related but different reports in the issue tracker. See #12877 (comment) for steps how to reproduce this problem, and validate it is actually fixed with this change. Most important piece of context: https://github.com/opencontainers/runc/blob/5cf9bb229feed19a767cbfdf9702f6487341e29e/libcontainer/cgroups/devices/v2.go#L55-L57

Few warnings:

• 4 hours ago I had no idea how cgroups worked, nor how BPF and DevicesSets interacted with cgroups
• 1 hour ago I never touched a Go project in my life (but a long history of other languages, so "its fine")

PS: I checked if there were any e2e tests this could be part of; but it seems there is nothing there yet, neither for cgroup or cpuset. If I missed it, please let me know.

[hashicorp-cla]

All committers have signed the CLA.

[shoenig]

This is great @TrueBrain!

On your branch, can you run make cl to generate a changelog entry?

I turned your repro steps into an e2e test in #17546, can confirm it seems to work. Most cgroups tests are in the drivers/ packages but just following the repro steps in the e2e environment is pretty effective.

before (main)

➜ go test -v -run TestCgroup
=== RUN   TestCgroupDevices
=== RUN   TestCgroupDevices/testDevicesUsable
    assert.go:11:
        devices_test.go:53: expected nil error
        ↪ error: command nomad [alloc exec 9f0e235a-3cb2-db10-a5ac-3450cad63cda dd if=/dev/zero of=/dev/null count=1] failed: exit status 1
        Output: dd: failed to open '/dev/zero': Operation not permitted
--- FAIL: TestCgroupDevices (1.58s)
    --- FAIL: TestCgroupDevices/testDevicesUsable (1.46s)
FAIL
exit status 1
FAIL    github.com/hashicorp/nomad/e2e/isolation        1.587s

after with (7bdde48)

➜ go test -v -run TestCgroup
=== RUN   TestCgroupDevices
=== RUN   TestCgroupDevices/testDevicesUsable
--- PASS: TestCgroupDevices (1.64s)
    --- PASS: TestCgroupDevices/testDevicesUsable (1.52s)
PASS
ok      github.com/hashicorp/nomad/e2e/isolation        1.644s

[TrueBrain]

Shall I also cherry-pick your e2e in this PR?

And awesome, happy there is a place for some e2e for this :D

[shoenig]

Shall I also cherry-pick your e2e in this PR?

Go for it!

[shoenig]

LGTM!