Backport of CSI: retry claims from client into release/1.2.x

[hc-github-team-nomad-core]

Backport

This PR is auto-generated from #12113 to be assessed for backporting due to the inclusion of the label backport/1.2.x.

The below text is copied from the body of the original PR.

---

Fixes #8609 when combined with #12112
Fixes #11477

When the alloc runner claims a volume, an allocation for a previous
version of the job may still have the volume claimed because it's
still shutting down. In this case we'll receive an error from the
server. Retry this error until we succeed or until a very long timeout
expires, to give operators a chance to recover broken plugins.

Make the claim hook tolerate temporary RPC failures in general to
prevent errors during client restart or during leadership elections.

---

(Tests don't pass until #12112 is merged; I'll rebase this on that PR)

[github-actions]

I'm going to lock this pull request because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active contributions.
If you have found a problem that seems related to this change, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.