-
Notifications
You must be signed in to change notification settings - Fork 1.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
nomad 1.3.0: node drain stuck at csi controller #12835
Comments
Hi @tgross, yeah I read that just 10 minutes ago and also saw your update to the docs in the PR. However I am quite suprised by this because previously with 1.2 I could drain system jobs immediately without issue. Looking at the code I think the system jobs are supposed to be drained when all "normal" allocations are gone, not only when the deadline expires. I think the fix would be for the isDone check to also need to ignore Plugin Jobs aswell nomad/nomad/drainer/draining_node.go Lines 69 to 73 in 4d404b3
otherwise nomad/nomad/drainer/drainer.go Line 292 in 4d404b3
I guess isDone could generally just be replaced by len(DrainingJobs()) == 0 ? |
Oh shoot, you're right. Good catch on the |
I've opened #12846 with that fix. |
I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues. |
Nomad version
Nomad v1.3.0-beta.1 (2eba643)
Operating system and Environment details
Ubuntu 20.04.4 LTS
Linux 5.4.0-109-generic #123-Ubuntu SMP Fri Apr 8 09:10:54 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Issue
I don't know whether this is related to #12324 but it seems that nomad is not trying to stop the controller allocation at all with 1.3.0.
Reproduction steps
Run csi controller as service job and nodes as system jobs. Issue single node drain for the node running the controller.
Expected Result
Should continue draining.
Actual Result
Drain gets stuck with the controller and csi node left running.
Job file (if appropriate)
csi controller jobfile
csi node jobfile
Nomad Server logs (if appropriate)
Nomad Client logs (if appropriate)
The text was updated successfully, but these errors were encountered: