-
Notifications
You must be signed in to change notification settings - Fork 4.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Retry pending deployments longer before failing them #13550
Retry pending deployments longer before failing them #13550
Conversation
@mfojtik didn't we see this flake somewhere? Failed deployment because of a missing deployer but the deployer actually existed? |
[test] |
@Kargakis yes, there was one. good catch! |
@@ -20,6 +20,8 @@ import ( | |||
"github.com/openshift/origin/pkg/util" | |||
) | |||
|
|||
const maxRetryCount = 10 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
please drop a comment here explaining what is this retry count
this LGTM |
Evaluated for origin test up to df79f89 |
[merge] |
continuous-integration/openshift-jenkins/test FAILURE (https://ci.openshift.redhat.com/jenkins/job/test_pull_request_origin/476/) (Base Commit: 959be87) |
#11114 [merge] |
#12072 [merge] |
Evaluated for origin merge up to df79f89 |
continuous-integration/openshift-jenkins/merge SUCCESS (https://ci.openshift.redhat.com/jenkins/job/merge_pull_request_origin/212/) (Base Commit: 959be87) (Image: devenv-rhel7_6104) |
// The first requeue is after 5ms and subsequent requeues grow exponentially. | ||
// This effectively can extend up to 5^10ms which caps to 1000s: | ||
// | ||
// 5ms, 25ms, 125ms, 625ms, 3s, 16s, 78s, 390s, 1000s, 1000s |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
thx @Kargakis What determines the rate of backoff for each requeue? Is it configurable?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Not configurable currently. See here.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I guess though you can construct your own ratelimiter
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
thx
@mfojtik ptal