Backport of gracefully recover tasks that use csi node plugins into release/1.5.x #16848
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Backport
This PR is auto-generated from #16809 to be assessed for backporting due to the inclusion of the label backport/1.5.x.
The below text is copied from the body of the original PR.
With happy running tasks that use a CSI volume, on client restart they may fail to be recovered and instead be killed and rescheduled, even though there is a healthy happy CSI plugin running right next to them.
The simplest version of the error condition I could replicate was using this repo's
demo/csi/hostpath
example. With the tasks all set up, I stop the client and start it again:The client doesn't know yet that the CSI plugin task is healthy, because it happens to try to recover the task that needs it first.
After this change, which retries for some time (one minute (is that ok?)) to find the plugin that it expects to exist:
We may wish to test with a more complex CSI setup to make sure that this fully Fixes #13028