-
Notifications
You must be signed in to change notification settings - Fork 1.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
CSI plugin expected controllers exceeds actual controller count #12771
Comments
Hi @iSchluff! I want to make sure I can reproduce this scenario accurately:
So you had a job that was registered as a controller, and then changed it to node and re-ran? Did the old controller allocations get replaced entirely by node allocations or did they get in-place updates? |
Yes exactly, I accidentally ran node tasks as controller tasks. The old allocations got replaced by an in-place job update. |
Ok, thank you. So there's likely two bugs here: the counts aren't being reset properly, but also we should have replaced the tasks entirely in that case and not done an in-place update. I'll see if I can put together a quick repro and report back. |
I was able to reproduce the non-destructive update pretty easily just by switching the |
I've merged that fix and it'll ship in the GA release of Nomad 1.3.0. Thanks for opening this issue @iSchluff! |
thanks for the quick response |
I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues. |
Nomad version
Nomad v1.3.0-beta.1 (2eba643)
Operating system and Environment details
Ubuntu 20.04.4 on x64
Issue
Due to operator error I registered csi nodes as controllers, therefore my expected controller count on the plugin is now too high.
Nomad still shows the plugin as healthy and the volumes as schedulable, however waiting for plugin healthy via terraform fails:
Is it possible to reduce the expected controller count without recreating the cluster?
Reproduction steps
Start more csi controllers than you need, stop some of them.
Expected Result
expected controller count should probably never exceed 1?
Nomad Server logs (if appropriate)
Nomad Client logs (if appropriate)
The text was updated successfully, but these errors were encountered: