Error when listing library/* tags w/ fresh install

[domibarton]

Hi, I use this docker registry UI and I have an issue...

Bug description

When I do a pull (e.g. docker pull ubuntu), I can see the fetched image in the UI.
However, when I click on it, then I see a lot of errors.

How to Reproduce

For UI bug, steps to reproduce the behavior:

1. Run docker pull ubuntu
2. Go to the UI
3. Click on library/ubuntu
4. See the following errors

{"errors":[{"code":"UNKNOWN","message":"unknown error","detail":{}}]}

For service bug, steps to reproduce the behavior:

My docker-compose file

---
version: "3.8"

services:

  registry:
    image: registry:${REGISTRY_VERSION}
    container_name: registry
    environment:
      - REGISTRY_PROXY_REMOTEURL
      - REGISTRY_PROXY_USERNAME
      - REGISTRY_PROXY_PASSWORD
      - REGISTRY_STORAGE_DELETE_ENABLED="true"
    networks:
      - proxy
    volumes:
      - registry:/var/lib/registry
    labels:
      - traefik.http.routers.registry.rule=Host(`${REGISTRY_URL}`)
      - traefik.http.routers.registry.middlewares=private-service@file
    restart: always

  registry-ui:
    image: joxit/docker-registry-ui:static
    container_name: registry-ui
    environment:
      REGISTRY_URL: http://registry:5000
      PULL_URL: https://${REGISTRY_URL}
      DELETE_IMAGES: "true"
    networks:
      - proxy
    labels:
      - traefik.http.routers.registry-ui.rule=Host(`${REGISTRY_UI_URL}`)
      - traefik.http.routers.registry-ui.middlewares=private-service@file
    restart: always

volumes:
  registry:

networks:
  proxy:
    external: true

Expected behavior

No errors should occur.

Screenshots

System information

• OS: Debian Buster
• Docker registry UI:
  • Version: latest
  • Server: docker

- Docker version: 19.03.13
- Docker registry ui tag: static
- OS/Arch: linux/amd64
- Tools: docker-compose

Additional context

Appart from that, the UI works & looks fine 👍

[Joxit]

Hi, thank you for issue, I'm glad that you are using my project :).

Are you using a private registry as a docker hub mirror ? Like this example https://docs.docker.com/registry/recipes/mirror/ ?

[domibarton]

Hi @Joxit

Yes, exactly like in the example…
I just have a Traefik Proxy in front of it.

Cheers
Domi

[Joxit]

Hi,

Ok, I tried your example, and it seems that using the UI with a docker hub mirror is dangerous... You can reach your docker hub quota limit easily.

{"errors":[{"code":"UNKNOWN","message":"unknown error","detail":{"errors":[{"code":"TOOMANYREQUESTS","message":"You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit"}]}}]}

I couldn't analyze how the docker mirror actually works, but it seems to act as a proxy for the requests that are made. If docker hub gives you an error, your mirror will also give you an error.
When you want to look at the list of your tags via the UI, the mirror will ask for all the tags and make the requests for each of them. I know docker has recently added query restrictions, this could be the cause.

Since the response {"errors":[{"code":"UNKNOWN","message":"unknown error","detail":{}}]} comes from the mirror, I think this is a Docker Registry issue.

[gahoo]

My private registry also act as a mirror. And I ran into the same situation, showing {"errors":[{"code":"UNKNOWN","message":"unknown error","detail":{}}]}. But I think the problem is that the UI saw all the tags and request all their information, but not all the tags are available on my cache registry. Look at the request https://<mirror-registry>/v2/<repository>/manifests/<some-tag> under the hood, the error will show up when <some-tag> does not exists and will not show up when <some-tag> exists. curl https://<mirror-registry>/v2/<repository>/manifests/<some-tag> and replace the tags with exists and not exists ones you will know what I mean.

So the solution is clear, find out tags only exist on the mirror and do not request other tags detail. But here comes another problem, how can I tell a tag is exists on the mirror since https://<mirror-registry>/v2/<repository>/tags/list returns all the tags and does not provide any filtering function.

[Joxit]

Yes @gahoo you understood everything, that's why the UI can be dangerous when the proxy is on 😕

[Semmu]

I'm having the exact same problem, I thought I can easily spin up this frontend for my pull-through cache registry and would appreciate some solution regarding this.

Is there a way to have this UI only show images and tags which are present on the registry it is pointed at? (Also I find it a bit strange that it is doing client-side requests to the underlying registry, I thought the UI acts as a proxy for these requests.)

[Semmu]

After some further investigation, I came to the conclusion that my idea cannot be implemented. I guess the official docker registry in proxy mode is so transparent regarding API calls that there is no way to query data from it (e.g. what images did it already download to its own storage.) Every single API call ends up at the official Docker Hub.

I have another (stupid?) idea though: spin up a secondary registry in non-proxy and read-only mode, which uses the same storage as the pull-through cache. Hopefully, they won't conflict too much and I will have a registry and proper API to at least inspect my cached content. Will report back if it works.

[Semmu]

🎉 It works! 🎉

It's not perfect, because requesting extra information which is "mentioned" in manifests, but not actually downloaded to the mirror registry shows an error, but apart from this small issue you can browse your cached catalogs and tags.

🌐 My setup:

• Storage:
  • S3 bucket: mirror_registry_bucket
  • Custom CORS rules for the bucket! It is needed, as the client makes direct requests to the bucket. You need to allow your UI origin host and headers.
• First registry accessible on mirror.dockerhub.internal
  • ingress path prefix: /v2
  • storage backend: S3 bucket from above
  • runs in proxy mode of the official Docker Hub
• Second registry accessible on ui.mirror.dockerhub.internal
  • ingress path prefix: /v2
  • storage backend: shared with the first registry's S3 bucket
    • this should make it possible to inspect the mirror registry contents
  • runs in non-proxy, read-only mode
• Joxit UI accessible on ui.mirror.dockerhub.internal
  • same URL as the secondary registry, for CORS reasons
  • ingress path prefix: /
  • dockerRegistryUrl set to https://URL of second registry
  • single registry mode + disabled delete

🏁 Goal: to have some UI showing the contents of my own mirror registry, without acting as a completely transparent API proxy to the official DockerHub.

✅ And it seems to be working mostly fine!

⚠️ Though I won't guarantee that running 2 registries on the same storage backend won't cause problems, so use this setup at your own risk!

Proof:

The last screenshot shows the error message when requesting some data which is actually not mirrored from the official DockerHub, as it was not needed yet by any docker client.

[Joxit]

Hi @Semmu thank you for your deep investigation and idea ! I added a section for this in the FAQ and I added your name in the contribution list !