-
Notifications
You must be signed in to change notification settings - Fork 65
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add a retry for the image pull during Preflight execution #785
Comments
It looks like See Crane code here |
Hi, @acornett21 thanks for checking. I checked a Crane code in more detail, and it seems like they have already a reasonable policy by retrying 3 times: at 0s, at 1s, and 4s.
Out of my experience with quay.io, more than 3 short-term retries would activate throttling, so probably the only thing we could do here is to increase the waiting time in-between attempts by setting Normally, this PR allows configuring the retry attempts. |
@tkrishtop Thanks for looking further, I'm not sure if these defaults get picked up since we are passing in our own options. Would need to look at the stacktrace to see if they get picked-up. Go Playground only allows for go core library to be used. |
@acornett21 thanks, I got help and managed to fix the invocation. main.go
execution:
Don't know how to test though, I hesitate to trigger throttling and get blocked :) |
@tkrishtop My initial thought is that you should rerun preflight in this case, but I would imagine the trick is detecting this case has occurred within your automation? With your concern around throttling in mind, I'm hesitate to do a full exponential backoff. Maybe a single retry after 5-10 seconds would help? Seems like we should be able to do that using the remote.Backoff implementation. WDYT? |
Is your feature request related to a problem? Please describe.
Currently, Preflight does not retry image pull
openshift-preflight/certification/internal/engine/engine.go
Lines 80 to 85 in ed82fca
and that produces the errors like this
Describe the solution you'd like.
Usually, these registry errors are temporary, I'd like to wait for 5 seconds and retry again.
Not sure if it's better to do it in the Preflight code or implement a retry feature at Crane.
Additional context.
This error is pretty often to bother.
The text was updated successfully, but these errors were encountered: