-
Notifications
You must be signed in to change notification settings - Fork 1.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Failed to open netns when using Consul Connect: Unknown FS magic 1021994 #8371
Comments
Hi @liuzhen! I was looking to see if I could find that error message and saw you'd opened containernetworking/plugins#507 in the CNI plugins project. That led me to this closed issue in the same project: containernetworking/plugins#69 which suggests this could be a problem with how the file system of the environment has been mounted (or possibly permissions to that filesystem). Is your Nomad client running as root on the host? Or is there anything unusual about how the file system mounts are built in your environment? |
Hi, I saw that issue. We run nomad as root with Systemd.
and for the filesystems:
|
We can look into this @liuzhen but I recommend re-opening containernetworking/plugins#507 as well to see if that project might have a better idea of what's going on. |
I encountered the same error by following the official Consul Connect example at https://www.nomadproject.io/docs/integrations/consul-connect After some research, I believe the issue is related to how Nomad handles network namespaces in my specific environment. However, there are not enough logs for me to investigate further.
The error messages:
|
I found the issue despite the lack of logs/error messages. |
Hi,
With consul connect enabled I run into the same problem with
I have the same setup running on an This only happens when network mode is set to
The problem is that consul connect needs |
I ran into both the "Unknown FS magic" and "cannot join network of a non running container" errors when using Nomad 0.12.9 on a Raspberry Pi 4 (Arm64). I determined this is happening because Nomad 0.12.x defaults to using I was successfully able to start a container with a Connect sidecar after configuring HCL plugin "docker" {
config {
infra_image = "gcr.io/google_containers/pause-arm64:3.0"
}
} JSON {
"plugin": {
"docker": {
"config": {
"infra_image": "gcr.io/google_containers/pause-arm64:3.0"
}
}
}
} This config should not be needed in Nomad 1.0 (see PR: #8957), although I have not yet tested that version in my environment. |
Can folks test this with Nomad 1.0.1 to see if you're still seeing this error message? I think @blake's comment is spot on and likely the problem with most comments I see. |
@nickethier I upgraded to Nomad 1.0.1 last week, removed the This seems to have been the issue all along in my environment. |
Glad to hear that! |
I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues. |
Nomad version
Nomad v0.11.3 (8918fc8)
Operating system and Environment details
Red Hat Enterprise Linux Server release 7.5 (Maipo)
LANG=en_US.UTF-8PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/opt/consul
Issue
Reproduction steps
setup cni-plugins-v0.8.6
create job
Job file (if appropriate)
Nomad Client logs (if appropriate)
If possible please post relevant logs in the issue.
Logs and other artifacts may also be sent to: nomad-oss-debug@hashicorp.com
Please link to your Github issue in the email and reference it in the subject
line:
Emails sent to that address are readable by all HashiCorp employees but are not publicly visible.
Nomad Server logs (if appropriate)
The text was updated successfully, but these errors were encountered: