Smarter node draining handling when using CSI plugins

[RickyGrassmuck]

Proposal

Currently, when performing a node drain on a client that has a CSI plugin running as a system task, you have to go through the workflow below in order to drain the node without leaving volumes attached to it:

1. Perform a drain on the node with the "Drain System Jobs" options turned off
2. Wait for all jobs to migrate off of the node and wait for them to be reallocated (to ensure the volumes have been completely detached from the node)
3. Perform a second node drain with the "Drain System Jobs" option enabled to shutdown the CSI Plugin on that node.

Failure to perform the drain procedure in steps like this can result in volumes being left attached to a node causing jobs that were drained to not be able to reschedule.

There are probably multiple ways to address this scenario but I think ideally, the node draining logic would be able to tap into the CSI Plugin state to determine if there are tasks with volumes mounted on the node being drained and if so, wait for each of the volumes to be detached before spinning down the CSI plugin jobs registered to handle those volumes.

Use-cases

Being able to perform a single Node Drain and have jobs using CSI volumes be properly handled to ensure they are able to be rescheduled appropriately.

Attempted Solutions

As mentioned above, this can be worked around by performing the drain in multiple steps, first without draining system jobs, then draining system jobs after everything has migrated.

[jrasell]

Hi @rigrassm, thanks a lot for rising this issue. I've marked it so we can further discuss this when it is roadmapped.

[tgross]

I've opened #12324 with a change that defers plugins until last, just as we currently do with system jobs. When combined with #11892, which will also ship in Nomad 1.3.0, we should be covered for this feature request. Thanks for opening the issue @RickyGrassmuck!

[tgross]

#12324 has been merged and will ship in Nomad 1.3.0

[github-actions]

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.