-
Notifications
You must be signed in to change notification settings - Fork 1.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Panic on nil pointer dereference using CNI with Calico #9647
Comments
Hi @radriaanse! Looks from the stacktrace that you're right that the NPE is getting hit in |
@radriaanse I've got a patch #9648 which will protect against the NPE. We'll land that in 1.0.1 which should ship very soon. Something we'll need to follow up on: what's the set of circumstances that causes the plugin to give us no interfaces with a sandbox? The CNI spec suggests this is a "MUST" field for this use case.
|
@tgross Nice, thanks! The CNI spec is clear about that field indeed, something I noticed while checking that just now is that:
From the calico-cni.log only includes the host's endpoint interface. I'll try to figure out why it's doing that, but that might not have anything to do with Nomad after all. |
I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues. |
Nomad version
Compiled for debugging with:
Operating system and Environment details
Issue
Hi!
I'm trying to use Calico via calico-cni using Nomad for a Docker container. On my test setup when I run the bellow job file Nomad panics with the nil pointer error as seen in the logs.
Because I'm not quite sure if this was caused by either my setup and/or Nomad/Calico config, I did some debugging yesterday but can't quite make out where this goes wrong.
I've recompiled Nomad on tag v1.0.0 without optimization and went through the program a couple of times, I noticed that the CNI interfaces configuration is indeed empty, causing this exception when Nomad tries to parse it's IP from it:
But i'm having a bit of a hard time figuring out why that's the case, the
res
that's created earlier seems to contain my interfaces but theSandbox
value is unset; this seems to be somehow important since the logic on line 113+. I sadly can't figur out a way to debug this properly since setting breakpoints there will result in multiple RPC timeout errors so that doesn't seem reliable.Something else that's interesting is that after Nomad crashes, systemd restarts it and Nomad continues with executing the Job. It eventually crashes again but leaves behind the init container (it also doesn't clean this up after running gc later on):
This also means that the network seems to have been setup regarding of the crash:
And I'm able to ping the host from within the netns, so Calico seems to be functioning fine (IP is a placeholder):
Any ideas on what might cause this?
I tried multiple settings but nothing really seems to make a difference, should there be anything else I can test let me know!
And if it's something with my setup; sorry for the noise but might be good either way to make sure Nomad will error more gracefully should this happen :).
Job file (if appropriate)
Nomad Client logs
Gist for full logs: https://gist.github.com/radriaanse/d2f46f6bd92126906430aab078735474
Coredump on: nomad-oss-debug@hashicorp.com
The text was updated successfully, but these errors were encountered: