Allow retry of 500 API errors to be handled by restart policies #3073
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
dadgar
reviewed
Aug 22, 2017
client/driver/docker.go
Outdated
| time.Sleep(1 * time.Second) | ||
| goto START | ||
| } | ||
| return structs.NewRecoverableError(startErr, true) |
There was a problem hiding this comment.
Can you change it such that we still retry quickly for 5 times and then fall back to the recoverable error if it is a 500? We want the fast retry behavior in this case since often it recovers quickly and the users restart policy may cause the task to fail.
|
Thanks @clinta |
|
I'm going to lock this pull request because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active contributions. |
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
API error 500 from docker is often recoverable, in our case we typically see this error from a volume driver after a container crashes. The volume driver must wait for a timeout from the now dead container before allowing a new container to mount it. This can take longer than 5 seconds.
Allowing these retires to be handled by the jobs configured restart policies seems to be a more intuitive solution than having a special hard-coded retry just for these errors.