-
Notifications
You must be signed in to change notification settings - Fork 1.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
CSI: move node unmount to server-driven RPCs #7596
Conversation
I just ran an end-to-end test of this where I deployed plugins, deployed jobs that need those plugins, removed the plugins, and then stopped the volume-claiming jobs. The deployed jobs stop normally, but the GC job fails and gets requeued as expected; this prevents a |
Interesting failure mode that stops a system GC. I had a volume claim that wasn't released for some reason (TBD), but it looks like the node plugin work had been completed. So when the system GC fired, it's hitting an error. We need to make sure the node client endpoint handles that gracefully.
|
8cd461f
to
a9e0d16
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ok! I think it looks good
I'm going to lock this pull request because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active contributions. |
Fixes #7180
If a volume-claiming alloc stops and the CSI Node plugin that serves that alloc's volumes is missing, there's no way for the allocrunner hook to send the
NodeUnpublish
andNodeUnstage
RPCs. Added complications:ControllerUnpublish
RPC happens after those two RPCs.This changeset addresses this issue with a redesign of the client-side for CSI. Rather than unmounting in the alloc runner hook, the alloc runner hook will simply exit. When the server gets the
Node.UpdateAlloc
for the terminal allocation that had a volume claim, it creates a volume claim GC job. This job will made client RPCs to a new node plugin RPC endpoint, and only once that succeeds, move on to making the client RPCs to the controller plugin. If the node plugin is unavailable, the GC job will fail and be requeued.We don't have to track any state about what the client has done, just that the allocs are terminal. When we do the "heartyeet" work in #2185, most of the plumbing is already in place. And this gives us the opportunity to add a Nomad command for operators to forcibly drop claims via the CLI in the future.
There's a bit of work done here (probably easiest to review commit-by-commit):