-
Notifications
You must be signed in to change notification settings - Fork 527
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
crane: crane copy from GCP GAR to dockerhub fails with invalid repository name #1679
Comments
@jonjohnsonjr there was a recent similar report in the I suspect this is due to a recent change in Dockerhub, where it rejects bad repo names in mount requests with this 400, instead of the 403 we (IIRC) expect and use as a signal to fallback to the full upload. We optimistically try to do a cross-registry mount, with the origin registry's repo name, which Dockerhub is now responding to with 400. I don't think this is due to any recent code change in ggcr. |
We have noticed the same issue and this change seems to make it work: @@ -271,6 +271,16 @@ func (w *writer) initiateUpload(ctx context.Context, from, mount, origin string)
}
u.RawQuery = uv.Encode()
+ // In case of unexpected errors when the layer is mounted, attempt to retry without mounting
+ fallbackWithoutMount := func(err error) (string, bool, error) {
+ if origin != "" && origin != w.repo.RegistryStr() {
+ // https://github.com/google/go-containerregistry/issues/1404
+ logs.Warn.Printf("retrying without mount: %v", err)
+ return w.initiateUpload(ctx, "", "", "")
+ }
+ return "", false, err
+ }
+
// Make the request to initiate the blob upload.
req, err := http.NewRequest(http.MethodPost, u.String(), nil)
if err != nil {
@@ -279,15 +289,16 @@ func (w *writer) initiateUpload(ctx context.Context, from, mount, origin string)
req.Header.Set("Content-Type", "application/json")
resp, err := w.client.Do(req.WithContext(ctx))
if err != nil {
+ if mount != "" {
+ return fallbackWithoutMount(err)
+ }
return "", false, err
}
defer resp.Body.Close()
if err := transport.CheckError(resp, http.StatusCreated, http.StatusAccepted); err != nil {
- if origin != "" && origin != w.repo.RegistryStr() {
- // https://github.com/google/go-containerregistry/issues/1404
- logs.Warn.Printf("retrying without mount: %v", err)
- return w.initiateUpload(ctx, "", "", "")
+ if mount != "" {
+ return fallbackWithoutMount(err)
}
return "", false, err
} It seems it is a change in how the DockerHub API handles the upload request. However, you may have a deeper understanding of the system. If this bug fix looks appropriate to you, I will submit a PR. |
Same errors from GHCR to Dockerhub btw. docker pull/tag/push works |
@jonjohnsonjr thank you for fixing this! |
Thanks all for reporting it <3 |
Describe the bug
crane copy
from GCP GAR (Google Artifact Registry) to Docker Hub stopped working since yesterday and fails with:To Reproduce
Create a private GCP GAR docker registry and copy an image from it to dockerhub:
E.g.
Expected behavior
It should succeed
Additional context
We use crane in our CI workflow (https://github.com/aptos-labs/aptos-core/blob/main/docker/release-images.mjs#L166-L167) incl. for nighly image publishes and this stopped working only about 36h ago. Possibly some changes on the dockerhub API?
Note I can not reproduce this issue when copying an image from GAR to GAR or from dockerhub to dockerhub, but only when copying from GAR to dockerhub.
crane version
: 0.14.0The text was updated successfully, but these errors were encountered: