CSI: skip node unpublish on GC'd or down nodes

[tgross]

Fixes #13264

If the node has been GC'd or is down, we can't send it a node
unpublish. The CSI spec requires that we don't send the controller
unpublish before the node unpublish, but in the case where a node is
gone we can't know the final fate of the node unpublish step.

The csi_hook on the client will unpublish if the allocation has
stopped and if the host is terminated there's no mount for the volume
anyways. So we'll now assume that the node has unpublished at its
end. If it hasn't, any controller unpublish will potentially hang or
error and need to be retried.

(Note that while this behavior isn't ideal, it appears to match user
expectations and the behavior reported by k8s users.)

[tgross]

Note for reviewers: I'm torn on whether to call this a bug or improvement but calling it a bug makes it something we can backport so I'm leaning that way.

[lgfa29]

I was thinking about other possible node status (like initializing or draining) but those would be fine since the node is still around to handle the request right?

[tgross]

I was thinking about other possible node status (like initializing or draining) but those would be fine since the node is still around to handle the request right?

Exactly! The disconnected state is a little weird as well, but in that case we're keeping the claims if the node can be marked disconnected, because the allocations will still be up and running, just temporarily unavailable.

[github-actions]

I'm going to lock this pull request because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active contributions.
If you have found a problem that seems related to this change, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.