-
Notifications
You must be signed in to change notification settings - Fork 1.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
CSI: client restarts cause later plugin restarts to fail #12744
Comments
Hi @kvkang! I was able to reproduce what you're seeing. What makes this a little interesting is that there's two variants of the problem that I could create: one where the plugin task was killed while the client was offline, and one where the plugin task was killed after the client came back from being offline. If any task (not just CSI plugins) stops while the Nomad client is stopped, the Nomad client will be unable to restore the task handle and will have to restart the task. But in the case of a CSI plugin we can't come back from that restart. If I kill a plugin task without having restarting the client, everything works as expected:
But if I restart the client and wait for it to come back, then kill the plugin task again, we get the error you reported:
If we don't wait for the client to come back and kill the plugin task while the client is offline, we get a crash loop but with slightly different error messages:
The plugin error I get is similar:
In both cases, eventually after we've gone thru the During normal plugin restarts, we'd expect skipping the This definitely a bug and it's not fixed by the work we've done for CSI in Nomad 1.3.0-beta.1. I'll dig into this further and report back here shortly. |
Will be fixed in #12752 which will ship in Nomad 1.3.0 GA, and backported to 1.2.x and 1.1.x. |
That's been merged and will ship in the upcoming release. Thanks for opening this issue @kvkang! |
@tgross May Poststart of csiPluginSupervisorHook being leaking goroutine for ensureSupervisorLoop each time? |
I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues. |
Nomad version
Nomad v1.2.6 (a6c6b47)
Operating system and Environment details
Ubuntu 20.04.3 LTS
Issue
Reproduction steps
Expected Result
csi container start with mount "/csi" and "/dev"
Actual Result
contrainer start fail and alloc log show show
F0422 05:28:57.588532 1 server.go:81] Failed to listen: listen unix /csi/csi.sock: bind: no such file or directory
Job file (if appropriate)
Nomad Server logs (if appropriate)
Nomad Client logs (if appropriate)
The text was updated successfully, but these errors were encountered: