-
Notifications
You must be signed in to change notification settings - Fork 1.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
nomad volume deregister doesn't work without -force #8949
Comments
It seems similar to #8100 , however that shows as being fixed in v12.2, but my issue persists on v12.5, so I don't think this is a duplicate. |
@hongkongkiwi does the volume have allocation claims? |
Yes, it has allocation claims. These claims do not disappear when the only job using the volume job is stopped. In-fact there is no way to remove the claims, either with detach, stopping the job or any other method. They stay there and block until a -force is used. |
Sorry to hear that, @hongkongkiwi. At this point I thought we'd fixed all the ways in which we could get these un-reclaimable volumes... obviously I've missed something though. It might help me diagnose if you were to run |
Hi @tgross , We think that this happens whenever an allocation (that uses a CSI volume) fails or it cannot stops smoothly, or the underlying node dies. job status placement error
nomad volume status zookeeper-data-1
nomad alloc status 8cafab90
nomad server(s) monitor
|
Thanks for providing those logs @chebelom, because that helped me realize we were detecting the "plugin in use" error message incorrectly during plugin GC, which is preventing us from running volume GC when you run Still working on the fix for the nil-alloc scenario. |
WIP PR is #9239 and I'm looking to ship this in the 1.0 final. |
I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues. |
Nomad version
Nomad v0.12.5 (514b0d6)
Operating system and Environment details
Linux builder0 4.15.0-112-generic 113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Issue
Using the
nomad volume deregister
command always fails with the errorI have stopped the job using the volume and I have also successfully detached the volume (nomad volume detach) command without error.
But I cannot deregister the volume even though the job using it has stopped and the detach command was successful. The only thing that works is
nomad volume deregister -force
Volume Details
Job Spec (for digitalocean plugin)
Job Spec (for job using volume)
The text was updated successfully, but these errors were encountered: