CSI can't deregister volume

[anthonymq]

I'm having troubles with an amazon efs volume that i deleted before stopping my nomad job who was using it. I can't stop allocations and can't deregister the volume.

Now I have two allocations that i can't deregister because the clients have been deleted and a volume that has been deleted on aws.

[anthonymq]

On another cluster, allocations can't be interrupted. This time the volume is still alive on aws.


I should only have the first and the last allocations.
The two allocation on version 15 have been stopped on their clients. But the nomad has not refreshed is state.

[tgross]

Hi @anthonymq sorry about this! The first problem is the sort of thing I'm going to give you some tools to solve with #8251 and #8252.

On another cluster, allocations can't be interrupted
The two allocation on version 15 have been stopped on their clients. But the nomad has not refreshed is state.

I can't think of a way that could be related to volumes, but stranger things have happened. I'll see if I can come up with a reproduction. Can you confirm that the command line shows this as well, and that it's not a web UI issue?

[anthonymq]

hi @tgross ,
Yes the cli responds the same thing. The volume is attached to multiple expired allocations.

[tgross]

Hey @anthonymq we shipped a nomad volume deregister -force in 0.12.0 which might help you out here. In the meantime, we're getting ramped up to wrap up these remaining CSI issues over the next couple weeks so hopefully we should have some progress for you.

[tgross]

Wanted to give a quick status update. I've landed a handful of PRs that will be released as part of the upcoming 0.12.2 release:

• csi: retry controller client RPCs on next controller #8561 retries controller RPCs so that we take advantage of controllers deployed in a HA configuration.
• csi: add unpublish RPC #8572 csi: update volumewatcher to use unpublish RPC #8579 are some improved plumbing that makes volume claim reaping synchronous in common cases, which reduces the number of places where things can go wrong (and makes it easier to reason about).
• csi: release claims via csi_hook postrun unpublish RPC #8580 uses that plumbing to drive the volume claim unpublish step from the client, so that in most cases (except when we lose touch with the client) we're running the volume unpublish synchronously as part of allocation shutdown.
• CSI unpublish error handling improvements #8605 improves our error handling so that the checkpointing we do will work correctly by ignoring "you already did that" errors.
• CSI: fix missing ACL tokens for leader-driven RPCs #8607 fixes some missing ACLs and Region flags in the Nomad leader

I believe these fixes combined should get us into pretty good shape, and #8584 will give you an escape hatch to manually detach the volume via nomad volume detach once that's merged.

[tgross]

For sake of our planning, I'm going to close this issue. We'll continue to track progress of this set of problems in #8100.

[github-actions]

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.