Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

cgroups: unable to initialize cpuset manager on CentOS9 #14494

Closed
shoenig opened this issue Sep 8, 2022 · 2 comments · Fixed by #15027
Closed

cgroups: unable to initialize cpuset manager on CentOS9 #14494

shoenig opened this issue Sep 8, 2022 · 2 comments · Fixed by #15027
Assignees
@shoenig
Labels
theme/cgroups cgroups issues type/bug

Comments

@shoenig
Copy link
Member

shoenig commented Sep 8, 2022

Out of the box, CentOS 9 is disabling cpuset (among other controllers) from being created by subgroups. We can probably turn it back on ourselves, but this is probably an artifact of them RH trying to push apps into using a systemd cgroups driver.

➜ cat /sys/fs/cgroup/cgroup.subtree_control 
cpu memory pids

➜ nomad version 
Nomad v1.3.5 (1359c2580fed080295840fb888e28f0855e42d50)

2022-09-08T07:46:30.088-0500 [WARN]  client.cpuset.v2: failed to lookup cpus from parent cgroup; disable cpuset management: error="openat2 /sys/fs/cgroup/nomad.slice/cpuset.cpus.effective: no such file or directory"
@shoenig shoenig added type/bug theme/cgroups cgroups issues labels Sep 8, 2022
@Procsiab
Copy link

Procsiab commented Oct 1, 2022

Hello there, are you running the Nomad binary with a rootless account or with root privileges?

@soupdiver soupdiver mentioned this issue Oct 5, 2022
Closed
@shoenig shoenig self-assigned this Oct 24, 2022
shoenig added a commit that referenced this issue Oct 24, 2022
@shoenig
client: ensure minimal cgroup controllers enabled
709460e 
This PR fixes a bug where Nomad could not operate properly on operating
systems that set the root cgroup.subtree_control to a set of controllers that
do not include the minimal set of controllers needed by Nomad.

Nomad needs these controllers enabled to operate:
- cpuset
- cpu
- io
- memory
- pids

Now, Nomad will ensure these controllers are enabled during Client initialization,
adding them to cgroup.subtree_control as necessary. This should be particularly
helpful on the RHEL/CentOS/Fedora family of system. Ubuntu systems should be
unaffected as they enable all controllers by default.

Fixes: #14494
@shoenig shoenig mentioned this issue Oct 24, 2022
Merged
shoenig added a commit that referenced this issue Oct 24, 2022
@shoenig
client: ensure minimal cgroup controllers enabled
fb67296 
This PR fixes a bug where Nomad could not operate properly on operating
systems that set the root cgroup.subtree_control to a set of controllers that
do not include the minimal set of controllers needed by Nomad.

Nomad needs these controllers enabled to operate:
- cpuset
- cpu
- io
- memory
- pids

Now, Nomad will ensure these controllers are enabled during Client initialization,
adding them to cgroup.subtree_control as necessary. This should be particularly
helpful on the RHEL/CentOS/Fedora family of system. Ubuntu systems should be
unaffected as they enable all controllers by default.

Fixes: #14494
shoenig added a commit that referenced this issue Oct 24, 2022
@shoenig
client: ensure minimal cgroup controllers enabled (#15027)
d978e77 
* client: ensure minimal cgroup controllers enabled

This PR fixes a bug where Nomad could not operate properly on operating
systems that set the root cgroup.subtree_control to a set of controllers that
do not include the minimal set of controllers needed by Nomad.

Nomad needs these controllers enabled to operate:
- cpuset
- cpu
- io
- memory
- pids

Now, Nomad will ensure these controllers are enabled during Client initialization,
adding them to cgroup.subtree_control as necessary. This should be particularly
helpful on the RHEL/CentOS/Fedora family of system. Ubuntu systems should be
unaffected as they enable all controllers by default.

Fixes: #14494

* docs: cleanup doc string

* client: cleanup controller writes, enhance log messages
This was referenced Oct 24, 2022
Merged
Merged
@github-actions
Copy link

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

@github-actions github-actions bot locked as resolved and limited conversation to collaborators Feb 22, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@shoenig shoenig

Labels
theme/cgroups cgroups issues type/bug
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

client: ensure minimal cgroup controllers enabled hashicorp/nomad
2 participants
@shoenig @Procsiab