HTTP GET a plugin manifest by version fails with 401

[rgl]

Problem description

While trying to understand why I could not use regular tools to download a docker plugin manifest, I discovered something that I think is a bug in the current docker hub registry api.

When we try to HTTP GET a plugin manifest by version (e.g. 2.2.1) that request wrongly fails with 401. It only works by digest (e.g. sha256:c7cf4544f8eee4e9d55e701e01f6ec5a290a807f0e75dfaf14ec463e3d973963).

Here is how I found the differences between what docker daemon does versus what a library like Docker.Registry.DotNet does.

To install a docker plugin with docker daemon, one uses something like:

docker_plugin_reference="grafana/loki-docker-driver:2.2.1"
docker plugin install \
  --disable \
  --grant-all-permissions \
  --alias "$docker_plugin_reference" \
  "$docker_plugin_reference" \
    LOG_LEVEL=debug

And this is how the docker daemon contacts the docker hub registry and is able to download the image manifest:

And this is how this library GetManifestAsync does it but fails to download the image manifest:

The difference is, the docker daemon first manifest request uses a HEAD request to translate the manifest reference from the version 2.2.1 to the digest sha256:c7cf4544f8eee4e9d55e701e01f6ec5a290a807f0e75dfaf14ec463e3d973963 (which it obtained from the HEAD response docker-content-digest header), and only then, it uses a GET request to obtain the actual manifest.

For cross-reference, this is also being tracked at ChangemakerStudios/Docker.Registry.DotNet#12.

[milosgajdos]

Hi @rgl I can't comment on how are you getting these results based on your screenshots, but I can see you're getting back 401 post initial authentication -- this means you are not authorized to get the manifests. (just FYI: I can only speak on behalf of Docker registry and can't commend on the Changemaker project).

Registry should let you grab manifest by tag using GET, indeed. See the example below on how you can grab a tagged manifest for the popular ansible project using curl -- this should work on any other projects, indeed -- I'm using ansible here just for illustration purposes.

Get the registry authorization token:
IT'S IMPORTANT YOU GENERATE BASE AUTHENTICATION HEADER USING YOUR DOCKER HUB CREDENTIALS

$ export TOKEN=$(curl -H "Authorization: Basic YOUR_BASE_AUTH" "https://auth.docker.io/token?service=registry.docker.io&scope=repository:ansible/ansible:pull" | jq -r '.token')

Make sure the token you get back has the right scope -- in this case thats pull:

$ jwt decode $TOKEN
Token header
------------
{
  "typ": "JWT",
  "alg": "RS256"
}

Token claims
------------
{
  "access": [
    {
      "actions": [
        "pull"
      ],
      "name": "ansible/ansible",
      "type": "repository"
    }
  ],
...
...

Grab the tagged manifest (NOTE: I've picked one of the available tags, but the same should work for any of them):

curl -H "Authorization: Bearer $TOKEN" "https://registry-1.docker.io/v2/ansible/ansible/manifests/ubuntu1404" | jq '.fsLayers'

[
  {
    "blobSum": "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4"
  },
  {
    "blobSum": "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4"
  },
  {
    "blobSum": "sha256:7b2540a071f1176ec66cc694df8ef7fb8f23eaec96fbedd4bd3d6abc92ae81e0"
  },
  {
    "blobSum": "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4"
  },
  {
    "blobSum": "sha256:4bb9435deb32a5c3939f08e19e3706fe1b416d2dc1455c177ad888aa186e4833"
  },
  {
    "blobSum": "sha256:d2ce138cb0050d9b21d20a485eaa4e0e809add554417b47bfd21620b9e6ece5b"
  },
...
...

[rgl]

@milosgajdos, please, make sure you try that in a docker plugin image like the grafana/loki-docker-driver:2.2.1.

[milosgajdos]

@hi @rgl you are right. Plugin manifests are not accessible via registry API manifest tag endpoint -- apologies for my confusing answer. We will discuss this internally, but until we've taken an action on this, please do follow the docker client flow.

[github-actions]

We are clearing up our old issues and your ticket has been open for 6 months with no activity. Remove stale label or comment or this will be closed in 15 days.

[rgl]

@milosgajdos, did you get a chance to look into this?

[milosgajdos]

Unfortunately I have no update on this, I'm afraid. For the time being I'd recommend to continue to use the current flow.

[jonjohnsonjr]

Looks like hub is returning this when you attempt to fetch a plugin:

HTTP/1.1 401 Unauthorized
...
Www-Authenticate: Bearer realm="https://auth.docker.io/token",service="registry.docker.io",scope="repository:vieux/sshfs:pull",error="insufficient_scope"

Whereas it should be:

HTTP/1.1 401 Unauthorized
...
Www-Authenticate: Bearer realm="https://auth.docker.io/token",service="registry.docker.io",scope="repository(plugin):vieux/sshfs:pull",error="insufficient_scope"

(note the scope="repository(plugin):vieux/sshfs:pull")

Based on https://github.com/moby/moby/blob/fcb52454acdfd3de635aa2c17f72ca4ee7d526d1/plugin/registry.go#L23

@milosgajdos

[jcarter3]

This has been fixed - the issue was around the scope handling on the authorization server with how plugins required a "class" to be appended (repository(plugin)). This check/condition has been removed. I'll close for now but can be reopened if needed.

[rgl]

@jcarter3, thx for the follow up!

does that mean we no longer need to use the repository(plugin) scope? repository scope will be enough?

[jcarter3]

That's correct, the plugin part is no longer needed. It can still be sent so that existing tooling continues to work, but is ultimately ignored.