fix(executor): Retry kubectl on transient error
#6472
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Closes argoproj#6467 Signed-off-by: William Van Hevelingen <william.vanhevelingen@acquia.com>
Signed-off-by: William Van Hevelingen <william.vanhevelingen@acquia.com>
alexec
reviewed
Aug 6, 2021
| @@ -34,7 +35,11 @@ func (we *WorkflowExecutor) ExecResource(action string, manifestPath string, fla | |||
| cmd := exec.Command("kubectl", args...) | |||
| log.Info(strings.Join(cmd.Args, " ")) | |||
|
|
|||
| out, err := cmd.Output() | |||
| var out []byte | |||
| err = retry.OnError(retry.DefaultBackoff, argoerr.IsTransientErr, func() error { | |||
There was a problem hiding this comment.
hmmm... what does this do, run the command X many times?
@sarabala1979 @whynowy I did not know about retry.OnError - better than argowait I think
There was a problem hiding this comment.
agree with @alexec . argowait should be used. you can enhance argowait with onError() if you want it.
There was a problem hiding this comment.
sorry - I actually meant the opposite - we should stop using argowait, and use retry.OnError instead
There was a problem hiding this comment.
Example:
err := waitutil.Backoff(defaultRetry,
func() (bool, error) {
log.Infof("GCS List bucekt: %s, key: %s", artifact.GCS.Bucket, artifact.GCS.Key)
client, err := g.newGCSClient()
if err != nil {
log.Warnf("Failed to create new GCS client: %v", err)
return isTransientGCSErr(err), err
}
defer client.Close()
files, err = listByPrefix(client, artifact.GCS.Bucket, artifact.GCS.Key, "")
if err != nil {
return isTransientGCSErr(err), err
}
return true, nil
})Becomes
err := retry.OnError(defaultRetry, isTransientGCSErr,
func() error {
log.Infof("GCS List bucekt: %s, key: %s", artifact.GCS.Bucket, artifact.GCS.Key)
client, err := g.newGCSClient()
if err != nil {
log.Warnf("Failed to create new GCS client: %v", err)
return err
}
defer client.Close()
files, err = listByPrefix(client, artifact.GCS.Bucket, artifact.GCS.Key, "")
if err != nil {
return err
}
return nil
})
Codecov Report
@@ Coverage Diff @@
## master #6472 +/- ##
==========================================
+ Coverage 48.61% 48.67% +0.06%
==========================================
Files 260 262 +2
Lines 18855 18983 +128
==========================================
+ Hits 9166 9240 +74
- Misses 8683 8707 +24
- Partials 1006 1036 +30
Continue to review full report at Codecov.
|
Signed-off-by: William Van Hevelingen <william.vanhevelingen@acquia.com>
|
It's not working yet. https://gist.github.com/blkperl/524ba2b7b0765c8d289ed3bcb892a292 It didn't detect the error as transient but it did execute the |
|
If it is a work in progress, do you want to put it into draft? |
This comment has been minimized.
This comment has been minimized.
Signed-off-by: William Van Hevelingen <william.vanhevelingen@acquia.com> Co-Authored-By: Glenn Pratt <glenn.pratt@acquia.com>
🍏 Manual Test 🍏Test 1: Cluster remains unavailableTest 2: Cluster recovers half way through the retriesTest 3: K8s api is stable |
alexec
approved these changes
Sep 7, 2021
alexec
changed the title
kubectl on transient error
This was referenced Mar 7, 2022
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Closes #6467
Signed-off-by: William Van Hevelingen william.vanhevelingen@acquia.com
Co-Authored-By: Glenn Pratt glenn.pratt@acquia.com
Checklist:
Tips:
git commit --signoff.make pre-commit -Bto fix codegen or lint problems.I'm not sure how to test this change.