docker push retry loop

[scones]

Actual behavior
first of: this is feature request rather than a bug report
i am running a private cluster and sometimes run into the occasion, that the push to the (on-cloud) registry receives a 504.
I am aware, that this is an issue with the cluster itself, but this doesn't make the feature request any less valid:

please wrap the actual docker push command into i retry loop.
additional thoughts on that:

• exponential backoff
• activated by flag (like --retries=0 as default to --retries=10 or more)

Expected behavior
avoid starting kaniko all over again and computing everything all over again.
have it retry the push only

To Reproduce
Steps to reproduce the behavior:
i would assume it should be possible to shutdown the target repository for the first few retries to test/implement that

Additional Information
irrelevant, i suppose

best regards and thanks for this marvelous project!

[balopat]

I just ran into this on our integration tests for skaffold - it would be very good if we could retry temporary errors in Kaniko, Docker also retries pushes:

error pushing image: failed to push to destination gcr.io/k8s-skaffold/skaffold-example:v0.37.1-223-gd2701f2: Patch https://gcr.io/v2/k8s-skaffold/skaffold-example/blobs/uploads/ABTmro6w_NXP3bfCN4xRTOpqbI8zPHZLwF0fTLX-NZaZUHaPliXwgjlx9nB31-z1prYDx0Rss-9fXdxowbL1vLs: io: read/write on closed pipe

[aivarasko]

Retry is already in google/go-containerregistry#459 but kaniko needs to upgrade this dependency.

[aivarasko]

Did extra checking and I think the issue is that just tryUpload part but not checkExistingBlob part in

kaniko/vendor/github.com/google/go-containerregistry/pkg/v1/remote/write.go

Line 291 in e0e59e6

func (w *writer) uploadOne(l v1.Layer) error {

is wrapped in retry

[ptemmer]

+1 Very much desired functionality

[rrichardson]

I have Kaniko jobs that run in parallel on my CI system. If the system is busy, I see these errors:

PUT https://registry.mydomain.com/myimage/manifests/latest: MANIFEST_BLOB_UNKNOWN: blob unknown to registry; sha256:7bbd61231447a971b972bf6e62b7e5aecc52c8edbdf1cf372d63aa1a3b1ed821

This appears to be some problem on my registry side, but it only happens in high traffic. It would be great to have a retry, since the build succeed, but the push temporarily failed.

[EmilyEmily]

+1 Continue waiting for this functionality.

[romainourgorry]

+1 Very much desired functionality

[FranciscoMSM]

👍

[scones]

in my experience p3 is the "edge feature" / "nice to have" category. i would kindly request bumping it to p2.

Since the continued annoyance of devs and ops using this project seems not to be enough: please think of the excess co2 produced by the unnecessary recompilations of already successfully compiled steps. (yes, the 'think of the trees!' argument)

[crumohr]

There seems to be a commit google/go-containerregistry@3b7741e from Jan 7. Could this solve our issue for good?

Addendum: this commit is part of github.com/google/go-containerregistry v0.4.0

[FritzOscar]

+1 Very much desired functionality

[ae-v]

Users of registries with a HA loadbalancing in front need this fix urgently @priyawadhwa @tejal29 @dlorenc Thanks in advance.

[scones]

yay, 2 years later :)