error pulling and saving library/busybox: 'signatures' when using digest

[ixdy]

I'm attempting to update our busybox dependency.

$ docker pull busybox:latest
latest: Pulling from library/busybox
Digest: sha256:1669a6aa7350e1cdd28f972ddad5aceba2912f589f19a090ac75b7083da748db
Status: Image is up to date for busybox:latest

so I update our WORKSPACE to the following:

docker_pull(
    name = "official_busybox",
    digest = "sha256:1669a6aa7350e1cdd28f972ddad5aceba2912f589f19a090ac75b7083da748db",
    registry = "index.docker.io",
    repository = "library/busybox",
)

This then fails:

ERROR: [elided]/src/k8s.io/kubernetes/build/BUILD:49:2: no such package '@official_busybox//image': Pull command failed: F0212 14:45:56.746001   51438 __main__.py:107] Error pulling and saving image index.docker.io/library/busybox@sha256:1669a6aa7350e1cdd28f972ddad5aceba2912f589f19a090ac75b7083da748db: 'signatures'
 ([elided]/.cache/bazel/_bazel_jgrafton/75e5737b0f848bf1b7c3ab3303c8bf72/external/puller/file/puller.par --directory [elided]/bazel/_bazel_jgrafton/75e5737b0f848bf1b7c3ab3303c8bf72/external/official_busybox/image --name index.docker.io/library/busybox@sha256:1669a6aa7350e1cdd28f972ddad5aceba2912f589f19a090ac75b7083da748db) and referenced by '//build:kube-controller-manager-internal'                                                                                                                       

Interestingly, if I use tag = "latest" instead of digest = "sha256:..." it seems to work.

[ixdy]

an older version of rules_docker gave a more elaborate stacktrace:

W0212 22:06:42.529] ERROR: /workspace/k8s.io/kubernetes/build/BUILD:49:2: no such package '@official_busybox//image': Pull command failed: Traceback (most recent call last):
W0212 22:06:42.530]   File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main
W0212 22:06:42.531]     "__main__", fname, loader, pkg_name)
W0212 22:06:42.532]   File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
W0212 22:06:42.532]     exec code in run_globals
W0212 22:06:42.533]   File "/root/.cache/bazel/_bazel_root/e9f728bbd90b3fba632eb31b20e1dacd/external/puller/file/puller.par/__main__.py", line 93, in <module>
W0212 22:06:42.534]   File "/root/.cache/bazel/_bazel_root/e9f728bbd90b3fba632eb31b20e1dacd/external/puller/file/puller.par/__main__.py", line 86, in main
W0212 22:06:42.535]   File "/root/.cache/bazel/_bazel_root/e9f728bbd90b3fba632eb31b20e1dacd/external/puller/file/puller.par/containerregistry/client/v2_2/v2_compat_.py", line 106, in __init__
W0212 22:06:42.536]   File "/root/.cache/bazel/_bazel_root/e9f728bbd90b3fba632eb31b20e1dacd/external/puller/file/puller.par/containerregistry/client/v2_2/v2_compat_.py", line 110, in _ProcessImage
W0212 22:06:42.537]   File "/root/.cache/bazel/_bazel_root/e9f728bbd90b3fba632eb31b20e1dacd/external/puller/file/puller.par/containerregistry/client/v2/docker_image_.py", line 177, in manifest
W0212 22:06:42.537]   File "/root/.cache/bazel/_bazel_root/e9f728bbd90b3fba632eb31b20e1dacd/external/puller/file/puller.par/containerregistry/client/v2/docker_digest_.py", line 32, in SignedManifestToSHA256
W0212 22:06:42.538]   File "/root/.cache/bazel/_bazel_root/e9f728bbd90b3fba632eb31b20e1dacd/external/puller/file/puller.par/containerregistry/client/v2/util_.py", line 96, in DetachSignatures
W0212 22:06:42.538] KeyError: 'signatures'
W0212 22:06:42.539]  (/usr/bin/python /root/.cache/bazel/_bazel_root/e9f728bbd90b3fba632eb31b20e1dacd/external/puller/file/puller.par --directory /root/.cache/bazel/_bazel_root/e9f728bbd90b3fba632eb31b20e1dacd/external/official_busybox/image --name index.docker.io/library/busybox@sha256:1669a6aa7350e1cdd28f972ddad5aceba2912f589f19a090ac75b7083da748db) and referenced by '//build:kube-scheduler-internal'

not sure if that is relevant or helpful at all.

[ixdy]

I'm guessing this bug should really apply to google/containerregistry.

[ixdy]

@mattmoor guessed this was relating to manifest lists. I think he's right.

I checked out github.com/google/containerregistry and added a debugging line:

diff --git a/client/v2/util_.py b/client/v2/util_.py
index e8ada13..794517f 100755
--- a/client/v2/util_.py
+++ b/client/v2/util_.py
@@ -92,6 +92,8 @@ def DetachSignatures(
   # First, decode the manifest to extract the list of signatures.
   json_manifest = json.loads(manifest)
 
+  print json.dumps(json_manifest, sort_keys=True, indent=4)
+
   # Next, extract the signatures that have signed a portion of the manifest.
   signatures = json_manifest['signatures']
 

I then ran bazel-bin/puller --directory /tmp/image --name index.docker.io/library/busybox@sha256:1669a6aa7350e1cdd28f972ddad5aceba2912f5 89f19a090ac75b7083da748db and got this output (before it errored out with the KeyError):

{
    "manifests": [
        {
            "digest": "sha256:4cee1979ba0bf7db9fc5d28fb7b798ca69ae95a47c5fecf46327720df4ff352d", 
            "mediaType": "application/vnd.docker.distribution.manifest.v2+json", 
            "platform": {
                "architecture": "amd64", 
                "os": "linux"
            }, 
            "size": 527
        }, 
        {
            "digest": "sha256:69d47339ca07d1153927214ad124597cab7f2a41b0ae13fd39acbfa2773df86b", 
            "mediaType": "application/vnd.docker.distribution.manifest.v2+json", 
            "platform": {
                "architecture": "arm", 
                "os": "linux", 
                "variant": "v5"
            }, 
            "size": 527
        }, 
        {
            "digest": "sha256:1583672a4c321cc8492d32247ef014a1030c5da49e3645d787bc791968fd6ce2", 
            "mediaType": "application/vnd.docker.distribution.manifest.v2+json", 
            "platform": {
                "architecture": "arm", 
                "os": "linux", 
                "variant": "v6"
            }, 
            "size": 527
        }, 
        {
            "digest": "sha256:f7ef7c0fffd7af2954ccdeefb85ddff7fe31b4bd4b38a8424ccf484b173595aa", 
            "mediaType": "application/vnd.docker.distribution.manifest.v2+json", 
            "platform": {
                "architecture": "arm", 
                "os": "linux", 
                "variant": "v7"
            }, 
            "size": 527
        }, 
        {
            "digest": "sha256:1a167b143c86c57fc2892eefe5d6f3fbc143a0a65ebc940f584b21c631b88927", 
            "mediaType": "application/vnd.docker.distribution.manifest.v2+json", 
            "platform": {
                "architecture": "arm64", 
                "os": "linux", 
                "variant": "v8"
            }, 
            "size": 527
        }, 
        {
            "digest": "sha256:e781fae1693341a7948267b6c242f81512d27943074d987a49ca85eca75ccbdc", 
            "mediaType": "application/vnd.docker.distribution.manifest.v2+json", 
            "platform": {
                "architecture": "386", 
                "os": "linux"
            }, 
            "size": 527
        }, 
        {
            "digest": "sha256:3c5b08ce23cf8610f44b841e34254ee0922295222180c79637230be5717efb80", 
            "mediaType": "application/vnd.docker.distribution.manifest.v2+json", 
            "platform": {
                "architecture": "ppc64le", 
                "os": "linux"
            }, 
            "size": 528
        }, 
        {
            "digest": "sha256:6ba746ef4d4e309daaf44780a368f82121d63b7d5640829916d9417c6c97b37b", 
            "mediaType": "application/vnd.docker.distribution.manifest.v2+json", 
            "platform": {
                "architecture": "s390x", 
                "os": "linux"
            }, 
            "size": 528
        }
    ], 
    "mediaType": "application/vnd.docker.distribution.manifest.list.v2+json", 
    "schemaVersion": 2
}

[dekkagaijin]

Almost certainly related to manifests lists, we have an internal change in the works to deal with it

Edit: beat me to it...

[ixdy]

and if I use sha256:4cee1979ba0bf7db9fc5d28fb7b798ca69ae95a47c5fecf46327720df4ff352d (the linux/amd64 digest listed above) the docker_pull works.

I'll use that for now... I need to figure out how to get docker to tell me that digest in the future.

[jonjohnsonjr]

Ack - working on a change that will accept the manifest lists's digest and just resolve it to amd64/linux, which should let you keep using the digest that docker spits out.

[ixdy]

@jonjohnsonjr perfect! will the platform be overridable? it's not something we need right now, but we will likely need it in the future.

[ixdy]

I now see this issue is basically a duplicate of #308. Closing in favor of that one.

[jonjohnsonjr]

@ixdy it's hardcoded for now (to fix this), but will be trivial to expose a flag to change it.