Nomad client fails to start a job #17448
Assignees
Labels
stage/accepted
Confirmed, and intend to work on. No timeline committment though.
theme/cgroups
cgroups issues
type/bug
Projects
Milestone
Comments
|
@akamensky can you ensure cgroups are enabled on the node? https://developer.hashicorp.com/nomad/docs/drivers/exec#resource-isolation |
|
@shoenig cgroups are enabled. As you can see from the logs it fails to set swappiness to 0 in cgroup. If I recall correctly Centos 7 kernel version just doesn't have separate swappiness for cgroup it is inherited from the main namespace. (Hence the Docker versions that can run on C7.x don't have that either as well as any other sysctl overrides). |
|
@shoenig sorry to ping you on this, but do you have any feedback? This is a make or break issue for us as we have to run older systems in our infra. If we have no workaround or fix, then we would have to abandon Nomad. |
|
It might be possible to provide a fix for this. Looking at the libcontainer implementation it should bypass writing to the swappiness interface file if we pass in a nil @akamensky are you able to test out such a fix yourself? We don't have any machines this old in our CI systems. |
|
@shoenig I would be able to test if you can provide compiled binary or at least a patch and build instructions. |
shoenig
added a commit
that referenced
this issue
Jun 20, 2023
This PR adds a workaround for very old Linux kernels which do not support the memory swappiness interface file. Normally we write a "0" to the file to explicitly disable swap. In the case the kernel does not support it, give libcontainer a nil value so it does not write anything. Fixes #17448
shoenig
added
the
stage/accepted
shoenig
added a commit
that referenced
this issue
Jun 20, 2023
This PR adds a workaround for very old Linux kernels which do not support the memory swappiness interface file. Normally we write a "0" to the file to explicitly disable swap. In the case the kernel does not support it, give libcontainer a nil value so it does not write anything. Fixes #17448
|
@akamensky The PR #17625 contains a fix I think should work, but like I mentioned we simply don't have kernels that old handy to test with. Probably the easiest way to check yourself would be to run the test case.
You can also build Nomad yourself |
|
Thanks @shoenig I got the code and follow the steps you outlined: The Not sure if this is expected. Next I will build the binary, but I need to hack your Update: Built Nomad binary and swap the one installed from RPM. Version shows as: However the task still fails: When I check the cgroups sys dir contents I see: So it does show Looking at the diff (I left similar comment there). The file it |
|
@shoenig I throw together a simple check whether the sys fs path is actually writeable (not specific to control groups): package main
import (
"fmt"
"io"
"os"
)
func main() {
cgOptions := []string{
"/sys/fs/cgroup/memory/memory.swappiness",
}
for _, cgOption := range cgOptions {
if !isCgOptionExists(cgOption) {
// if cgroup option does not exist,
// then nothing to do
continue
}
if _, err := isCgOptionWritable(cgOption); err != nil {
fmt.Println(cgOption, "is not writable due to:", err.Error())
} else {
fmt.Println(cgOption, "ok")
}
}
}
func isCgOptionExists(sysFsPath string) bool {
_, err := os.Stat(sysFsPath)
if os.IsNotExist(err) {
return false
}
return true
}
func isCgOptionWritable(sysFsPath string) (bool, error) {
currValue, err := readCgOption(sysFsPath)
if err != nil {
return false, err
}
err = writeCgOption(sysFsPath, currValue)
if err != nil {
return false, err
}
return true, nil
}
func readCgOption(sysFsPath string) ([]byte, error) {
fd, err := os.Open(sysFsPath)
if err != nil {
return nil, err
}
defer fd.Close()
return io.ReadAll(fd)
}
func writeCgOption(sysFsPath string, value []byte) error {
fd, err := os.OpenFile(sysFsPath, os.O_TRUNC|os.O_WRONLY|os.O_SYNC, 0644)
if err != nil {
return err
}
defer fd.Close()
_, err = fd.Write(value)
if err != nil {
return err
}
return nil
}On the kernel in question it produces: On somewhat newer kernel it produces: On very recent kernel it produces: Update: I then applied below patch on your branch (note: this would need to be checked in same way for any sysctl params set for control group as older kernels don't support some/all sysctl params override -- those are always inherited from system): compiled and ran, and it can run tasks/jobs on this kernel just fine now:
|
|
Thanks @akamensky that is super helpful. I added a commit to the branch which should be a succinct version of what you have. (We can get away with just trying to write to the |
|
Thanks @shoenig I've tested updated patch and it works as expected now. I have not tried any other job configuration options with this, but if I find any issues with those old kernels will report them separately. |
shoenig
added a commit
that referenced
this issue
Jun 22, 2023
#17625) * client: do not disable memory swappiness if kernel does not support it This PR adds a workaround for very old Linux kernels which do not support the memory swappiness interface file. Normally we write a "0" to the file to explicitly disable swap. In the case the kernel does not support it, give libcontainer a nil value so it does not write anything. Fixes #17448 * client: detect swappiness by writing to the file * fixup changelog Co-authored-by: James Rasell <jrasell@users.noreply.github.com> --------- Co-authored-by: James Rasell <jrasell@users.noreply.github.com>
This was referenced Jun 22, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Nomad version
Operating system and Environment details
Issue
Submitting job fails with:
Reproduction steps
Expected Result
Actual Result
Job file (if appropriate)
Nomad Server logs (if appropriate)
N/A
Nomad Client logs (if appropriate)
See above
The text was updated successfully, but these errors were encountered: