crane: Latest version of debug image broke crane cp

[fkuhnert]

Describe the bug

A gitlab CI/CD pipeline was incorrectly using gcr.io/go-containerregistry/crane:debug as the base image. Since it was always using the latest image, I caught a bug that is not present on the gcr.io/go-containerregistry/crane/debug:v0.14.0 version or even running v0.14.0 locally.

The bug itself: all the pipeline did was run crane cp [our gitlab] [our AWS ECR registry] after authenticating to such registries using crane auth login. However, after a certain commit (we don't know which), the latest image stopped working, splurting out the following error:

Error: failed to copy image: HEAD https://[our ECR]/[our image]/blobs/sha256:fb3e216f5c10201b4e52856afc7739b07b2276d26087325315129efcc47cefdf: unexpected status code 401 Unauthorized (HEAD responses have no body, use GET for details)

After inspecting with --verbose, this error pretty much happens on every blob that tries to be pushed.

All other commands work fine on both registries, so this isn't an authentication issue.

To Reproduce

As explained earlier, all we did was authenticate to both registries with crane auth login and run crane cp [our gitlab] [our AWS ECR registry] on the latest debug version. I suspect this might happen on regular versions as well, but I haven't tested that.

Expected behavior

Crane correctly copies the images from one registry to another.

Additional context

After updating the image to use debug:v0.14.0, the pipeline runs correctly.

• Output of crane version: 217318c3b8e3aea04c135097619d83b22ed25fe2
• Registry used (e.g., GCR, ECR, Quay): gitlab container registry and AWS ECR

[cuthbeorht]

Same here.

A quick workaround is to use a previous image:

gcr.io/go-containerregistry/crane@sha256:b0850af7d1d2aee3b16c5ca79cc7cf516fa20330ec0f4a5badd5fda44924fae9

[maxenced]

Can confirm the issue after a few hours of debugging.
crane push and crane pull to / from destination registry works, but crane cp src dst fails on dst.

Using debug (-v) I can see the requests for token are exactly the same :

crane cp

2023/04/20 08:08:26 --> GET https://eu.gcr.io/v2/token?scope=repository%3Amyproject%2Fxxxi%3Apush%2Cpull&service=eu.gcr.io [body redacted: basic token response contains credentials]
2023/04/20 08:08:26 GET /v2/token?scope=repository%3Amyproject%2Fxxx%3Apush%2Cpull&service=eu.gcr.io HTTP/1.1
Host: eu.gcr.io
User-Agent: crane/1cb7e133961581d6e0f5564e187b8f4dbf33f9d5 go-containerregistry/(devel)
Authorization: <redacted>
Accept-Encoding: gzip


2023/04/20 08:08:26 <-- 401 https://eu.gcr.io/v2/token?scope=repository%3Amyproject%xxx%3Apush%2Cpull&service=eu.gcr.io (37.115555ms) [body redacted: basic token response contains credentials]

crane push to same dest

2023/04/20 08:08:56 --> GET https://eu.gcr.io/v2/token?scope=repository%3Amyproject%2Fxxx%3Apush%2Cpull&service=eu.gcr.io [body redacted: basic token response contains credentials]
2023/04/20 08:08:56 GET /v2/token?scope=repository%3Amyproject%2Fxxx%3Apush%2Cpull&service=eu.gcr.io HTTP/1.1
Host: eu.gcr.io
User-Agent: crane/1cb7e133961581d6e0f5564e187b8f4dbf33f9d5 go-containerregistry/(devel)
Authorization: <redacted>
Accept-Encoding: gzip


2023/04/20 08:08:56 <-- 200 https://eu.gcr.io/v2/token?scope=repository%3Amyproject%2xxx%3Apush%2Cpull&service=eu.gcr.io (131.458531ms) [body redacted: basic token response contains credentials]

So i suspect bad authorization is used

[maxenced]

So, for amd64:

• sha256:e6464163a27f3fef9616cf38b0011251c7c26e07d465e90f6a8f079407aeac6d (version e055961ad3aaa32f3562d1944a00a9a02beafd36) is the last working image
• sha256:e6464163a27f3fef9616cf38b0011251c7c26e07d465e90f6a8f079407aeac6d (version 1cb7e133961581d6e0f5564e187b8f4dbf33f9d5 ) is the first broken one

Commit 1cb7e13 seems a good candidate

[imjasonh]

Sorry about that! Thanks for your thorough analysis, #1603 seems like a likely candidate, and #1664 reverts it.

After that's released let me know if this issue goes away.

[imjasonh]

You should soon be able to use gcr.io/go-containerregistry/crane/debug:27a6ad623ea9926c58e2bddb7158e9e5278a7dc4 which will be built from the commit that reverted the bad change.