Backport of CSI: improve controller RPC reliability into release/1.4.x

[hc-github-team-nomad-core]

Backport

This PR is auto-generated from #17996 to be assessed for backporting due to the inclusion of the label backport/1.4.x.

The below text is copied from the body of the original PR.

---

The CSI specification says that we "SHOULD" send no more than one in-flight request per volume at a time, with an allowance for losing state (ex. leadership transitions) which the plugins "SHOULD" handle gracefully. We mostly successfully serialize node and controller RPCs for the same volume, except when Nomad clients are lost. (See also container-storage-interface/spec#512)

These concurrency requirements in the spec fall short because Storage Provider APIs aren't necessarily safe to call concurrently on the same host even for different volumes. For example, concurrently attaching AWS EBS volumes to an EC2 instance results in a race for device names, which results in failure to attach (because the device name is taken already and the API call fails) and confused results when releasing claims. So in practice many CSI plugins rely on k8s-specific sidecars for serializing storage provider API calls globally. As a result, we have to be much more conservative about concurrency in Nomad than the spec allows.

This changeset includes four major changes to fix this:

• Add a serializer method to the CSI volume RPC handler. When the RPC handler makes a destructive CSI Controller RPC, we send the RPC thru this serializer and only one RPC is sent at a time. Any other RPCs in flight will block.
• Ensure that requests go to the same controller plugin instance whenever possible by sorting by lowest client ID out of the plugin instances.
• Ensure that requests go to healthy plugin instances only.
• Ensure that requests for controllers can go to a controller on any live node, not just ones eligible for scheduling (which CSI controllers don't care about)

Fixes: #15415