crane: crane copy from GCP GAR to dockerhub fails with invalid repository name

[geekflyer]

Describe the bug

crane copy from GCP GAR (Google Artifact Registry) to Docker Hub stopped working since yesterday and fails with:

unexpected status code 400 Bad Request: {"details":"invalid repository name"}

To Reproduce

Create a private GCP GAR docker registry and copy an image from it to dockerhub:

E.g.

crane copy us-docker.pkg.dev/aptos-registry/docker/indexer-grpc:ad7aac200e5f5af409568ca6f645e867be8b0e41 aptoslabs/indexer-grpc:ad7aac200e5f5af409568ca6f645e867be8b0e41

Expected behavior

It should succeed

Additional context

We use crane in our CI workflow (https://github.com/aptos-labs/aptos-core/blob/main/docker/release-images.mjs#L166-L167) incl. for nighly image publishes and this stopped working only about 36h ago. Possibly some changes on the dockerhub API?

Note I can not reproduce this issue when copying an image from GAR to GAR or from dockerhub to dockerhub, but only when copying from GAR to dockerhub.

• Output of crane version: 0.14.0

[imjasonh]

@jonjohnsonjr there was a recent similar report in the #ko-build Slack channel, which was reportedly resolved by upgrading to ko@0.13.0.

I suspect this is due to a recent change in Dockerhub, where it rejects bad repo names in mount requests with this 400, instead of the 403 we (IIRC) expect and use as a signal to fallback to the full upload.

We optimistically try to do a cross-registry mount, with the origin registry's repo name, which Dockerhub is now responding to with 400. I don't think this is due to any recent code change in ggcr.

[jotadrilo]

We have noticed the same issue and this change seems to make it work:

@@ -271,6 +271,16 @@ func (w *writer) initiateUpload(ctx context.Context, from, mount, origin string)
    }
    u.RawQuery = uv.Encode()

+   // In case of unexpected errors when the layer is mounted, attempt to retry without mounting
+   fallbackWithoutMount := func(err error) (string, bool, error) {
+       if origin != "" && origin != w.repo.RegistryStr() {
+           // https://github.com/google/go-containerregistry/issues/1404
+           logs.Warn.Printf("retrying without mount: %v", err)
+           return w.initiateUpload(ctx, "", "", "")
+       }
+       return "", false, err
+   }
+
    // Make the request to initiate the blob upload.
    req, err := http.NewRequest(http.MethodPost, u.String(), nil)
    if err != nil {
@@ -279,15 +289,16 @@ func (w *writer) initiateUpload(ctx context.Context, from, mount, origin string)
    req.Header.Set("Content-Type", "application/json")
    resp, err := w.client.Do(req.WithContext(ctx))
    if err != nil {
+       if mount != "" {
+           return fallbackWithoutMount(err)
+       }
        return "", false, err
    }
    defer resp.Body.Close()

    if err := transport.CheckError(resp, http.StatusCreated, http.StatusAccepted); err != nil {
-       if origin != "" && origin != w.repo.RegistryStr() {
-           // https://github.com/google/go-containerregistry/issues/1404
-           logs.Warn.Printf("retrying without mount: %v", err)
-           return w.initiateUpload(ctx, "", "", "")
+       if mount != "" {
+           return fallbackWithoutMount(err)
        }
        return "", false, err
    }

It seems it is a change in how the DockerHub API handles the upload request. However, you may have a deeper understanding of the system.

If this bug fix looks appropriate to you, I will submit a PR.

[roderik]

Same errors from GHCR to Dockerhub btw. docker pull/tag/push works

[jotadrilo]

@jonjohnsonjr thank you for fixing this!

[jonjohnsonjr]

Thanks all for reporting it <3