-
Notifications
You must be signed in to change notification settings - Fork 1.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Nomad client fails to start a job #17448
Comments
@akamensky can you ensure cgroups are enabled on the node? https://developer.hashicorp.com/nomad/docs/drivers/exec#resource-isolation |
@shoenig cgroups are enabled. As you can see from the logs it fails to set swappiness to 0 in cgroup. If I recall correctly Centos 7 kernel version just doesn't have separate swappiness for cgroup it is inherited from the main namespace. (Hence the Docker versions that can run on C7.x don't have that either as well as any other sysctl overrides).
|
@shoenig sorry to ping you on this, but do you have any feedback? This is a make or break issue for us as we have to run older systems in our infra. If we have no workaround or fix, then we would have to abandon Nomad. |
It might be possible to provide a fix for this. Looking at the libcontainer implementation it should bypass writing to the swappiness interface file if we pass in a nil @akamensky are you able to test out such a fix yourself? We don't have any machines this old in our CI systems. |
@shoenig I would be able to test if you can provide compiled binary or at least a patch and build instructions. |
This PR adds a workaround for very old Linux kernels which do not support the memory swappiness interface file. Normally we write a "0" to the file to explicitly disable swap. In the case the kernel does not support it, give libcontainer a nil value so it does not write anything. Fixes #17448
This PR adds a workaround for very old Linux kernels which do not support the memory swappiness interface file. Normally we write a "0" to the file to explicitly disable swap. In the case the kernel does not support it, give libcontainer a nil value so it does not write anything. Fixes #17448
@akamensky The PR #17625 contains a fix I think should work, but like I mentioned we simply don't have kernels that old handy to test with. Probably the easiest way to check yourself would be to run the test case.
You can also build Nomad yourself |
Thanks @shoenig I got the code and follow the steps you outlined: The
Not sure if this is expected. Next I will build the binary, but I need to hack your Update: Built Nomad binary and swap the one installed from RPM. Version shows as:
However the task still fails:
When I check the cgroups sys dir contents I see:
So it does show
Looking at the diff (I left similar comment there). The file it
|
@shoenig I throw together a simple check whether the sys fs path is actually writeable (not specific to control groups): package main
import (
"fmt"
"io"
"os"
)
func main() {
cgOptions := []string{
"/sys/fs/cgroup/memory/memory.swappiness",
}
for _, cgOption := range cgOptions {
if !isCgOptionExists(cgOption) {
// if cgroup option does not exist,
// then nothing to do
continue
}
if _, err := isCgOptionWritable(cgOption); err != nil {
fmt.Println(cgOption, "is not writable due to:", err.Error())
} else {
fmt.Println(cgOption, "ok")
}
}
}
func isCgOptionExists(sysFsPath string) bool {
_, err := os.Stat(sysFsPath)
if os.IsNotExist(err) {
return false
}
return true
}
func isCgOptionWritable(sysFsPath string) (bool, error) {
currValue, err := readCgOption(sysFsPath)
if err != nil {
return false, err
}
err = writeCgOption(sysFsPath, currValue)
if err != nil {
return false, err
}
return true, nil
}
func readCgOption(sysFsPath string) ([]byte, error) {
fd, err := os.Open(sysFsPath)
if err != nil {
return nil, err
}
defer fd.Close()
return io.ReadAll(fd)
}
func writeCgOption(sysFsPath string, value []byte) error {
fd, err := os.OpenFile(sysFsPath, os.O_TRUNC|os.O_WRONLY|os.O_SYNC, 0644)
if err != nil {
return err
}
defer fd.Close()
_, err = fd.Write(value)
if err != nil {
return err
}
return nil
} On the kernel in question it produces:
On somewhat newer kernel it produces:
On very recent kernel it produces:
Update: I then applied below patch on your branch (note: this would need to be checked in same way for any sysctl params set for control group as older kernels don't support some/all sysctl params override -- those are always inherited from system):
compiled and ran, and it can run tasks/jobs on this kernel just fine now: |
Thanks @akamensky that is super helpful. I added a commit to the branch which should be a succinct version of what you have. (We can get away with just trying to write to the |
Thanks @shoenig I've tested updated patch and it works as expected now. I have not tried any other job configuration options with this, but if I find any issues with those old kernels will report them separately. |
#17625) * client: do not disable memory swappiness if kernel does not support it This PR adds a workaround for very old Linux kernels which do not support the memory swappiness interface file. Normally we write a "0" to the file to explicitly disable swap. In the case the kernel does not support it, give libcontainer a nil value so it does not write anything. Fixes #17448 * client: detect swappiness by writing to the file * fixup changelog Co-authored-by: James Rasell <jrasell@users.noreply.github.com> --------- Co-authored-by: James Rasell <jrasell@users.noreply.github.com>
Nomad version
Operating system and Environment details
Issue
Submitting job fails with:
Reproduction steps
Expected Result
Actual Result
Job file (if appropriate)
Nomad Server logs (if appropriate)
N/A
Nomad Client logs (if appropriate)
See above
The text was updated successfully, but these errors were encountered: