Proxy cache

[michmike]

enable proxy cache for Harbor.

This is very useful in situations of intermittent network connectivity where you still need access to local images. the expectation is that the network will go off in ROBO or IoT scenarios

Of course, the standard use case of reducing redundant image pulls across the network applies here

[xaleeks]

@michmike Is this referring to deploying a worker instance close to the edge nodes functioning specifically as a local registry mirror for a central Harbor and have the edge daemons pointing to it for local pulls? or is Harbor being used as proxy cache to pull from other registries like Docker hub and such? I mean is this this linking Harbor instances or w/ rest of the registry ecosystem.

somethings to think about

• once pulled, how does long is it kept in cache for other clients before cleaned, there should be some LRU (least recently used) algorithm for determination
• 'subsequent' pulls of any particular image can be based on just the image manifest
• user pulling image needs to be properly authenticated and authorization checked before redirected to the local cache
• will this be able to work w/ rbac
• when to use this vs deploying another harbor cluster and relying on regular replication

[salcinad]

@michmike
are you referring to the: https://github.com/goharbor/harbor/blob/master/contrib/Configure_mirror.md

our need is that we don't pull Images from "registry-1.docker.io" (from Docker hub in Internet) but from internal Harbor Register which will act as Proxy/Cache ( for example this Option is available in Nexus 3), which will Cache and if not cached pull Images from Docker hub and provide to user.

[michmike]

proxy cache is a feature of registry v2. this is what i meant - https://mtijhof.wordpress.com/2018/07/23/using-nexus-oss-as-a-proxy-cache-for-docker-images/

[michmike]

we of course don't have to restrict the Harbor capabilities, so it could pull images as a cache from any type of replication target we can support

[stonezdj]

proxy cache is a feature of registry v2. this is what i meant - https://mtijhof.wordpress.com/2018/07/23/using-nexus-oss-as-a-proxy-cache-for-docker-images/

The pull through proxy only works for docker hub, it doesn't work for other registry repository.

[xaleeks]

@stonezdj yes it seems the docker registry's own proxy mechanism still only works for docker hub. There's a pr on supporting other self hosted registries but still unmerged after 2 years
moby/moby#34319

[xaleeks]

PRD:
https://docs.google.com/document/d/15CRHZuxfw9cm3booRVIToZrkc3Emqi-6Z26-AlmAWuE/edit
ping me for access

[sadoMasupilami]

As docker hub is soon enforcing strict image pull limits this feature is very important I think.

https://www.docker.com/blog/scaling-docker-to-serve-millions-more-developers-network-egress/

[gregsidelinger]

Is there anyway to configure harbor 2.1 as a transparent docker hub mirror? I can only make 2.1 pull from the upstream by doing things like this docker pull mycache/library/tomcat:latest. I was hoping I could make it work like this docker pull tomcat:latest by pointing my registry-mirror setting to my harbor install.

[jgallucci32]

@xaleeks @gregsidelinger Using the example from @ChristianCiach I was able to get Harbor to act as both a transparent proxy for Docker Hub and a Private Registry. It involves 4 steps:

1. Create a Proxy Cache project (i.e. docker.io) as pull-through cache to Docker Hub
2. Create DNS alias (i.e. docker-mirror.domain.local) which points to the DNS name of Harbor
3. Modify the nginx.conf file for the harbor-nginx pod to rewrite the URLs for /v2 and /service when using the DNS alias
4. Configure Docker daemon to use DNS alias as registry-mirror

Modifications to nginx.conf

1. Put in http { block

  # Map for proxy cache
  map $upstream_http_www_authenticate $new_header {
    ~^(?<prefix1>.*https://).*(?<suffix1>/service/token.*)$     $prefix1$host$suffix1;
  }
  map $args $new_args {
    ~^(?<prefix2>.*scope=repository%3A)(?<suffix2>(?!docker.io).*)$     ${prefix2}docker.io%2F$suffix2;
  }

2. Put in second server { block

    # Rewrite for proxy cache
    if ($host = 'docker-mirror.domain.local') {
      rewrite ^/v2/((?!docker.io).+)$ /v2/docker.io/$1 last;
      set $args $new_args;
    }

3. Put in location /v2/ { block

      # Modify headers for proxy cache
      proxy_hide_header Www-Authenticate;
      add_header Www-Authenticate $new_header always;

I am thinking this really needs to be its own pod within Harbor so it can be decoupled from the primary service. Since this issue is closed out I will work on creating a new one with this information.

UPDATE: I made changes based on some research and testing where map tables are better than multiple if blocks

[bagel-dawg]

@xaleeks @gregsidelinger Using the example from @ChristianCiach I was able to get Harbor to act as both a transparent proxy for Docker Hub and a Private Registry. It involves 4 steps:

1. Create a Proxy Cache project (i.e. `docker.io`) as pull-through cache to Docker Hub

2. Create DNS alias (i.e. `docker-mirror.domain.local`) which points to the DNS name of Harbor

3. Modify the `nginx.conf` file for the harbor-nginx pod to rewrite the URLs for `/v2` and `/service` when using the DNS alias

4. Configure Docker daemon to use DNS alias as registry-mirror

Modifications to nginx.conf

1. Put in `http {` block

  # Map for proxy cache
  map $upstream_http_www_authenticate $new_header {
    ~^(?<prefix1>.*https://).*(?<suffix1>/service/token.*)$     $prefix1$host$suffix1;
  }
  map $args $new_args {
    ~^(?<prefix2>.*scope=repository%3A)(?<suffix2>(?!docker.io).*)$     ${prefix2}docker.io%2F$suffix2;
  }

1. Put in second `server {` block

    # Rewrite for proxy cache
    if ($host = 'docker-mirror.domain.local') {
      rewrite ^/v2/((?!docker.io).+)$ /v2/docker.io/$1 last;
      set $args $new_args;
    }

1. Put in `location /v2/ {` block

      # Modify headers for proxy cache
      proxy_hide_header Www-Authenticate;
      add_header Www-Authenticate $new_header always;

I am thinking this really needs to be its own pod within Harbor so it can be decoupled from the primary service. Since this issue is closed out I will work on creating a new one with this information.

UPDATE: I made changes based on some research and testing where map tables are better than multiple if blocks

Could you provide a more-complete example of this? I am trying to attempt this patch but I am unable to get my nginx.conf to work right. I'm also a bit confused because your transparent proxy also seems to be named docker.io so I am unsure of what needs to be docker.io literally and what needs to be $MY_HARBOR_PROJECT_NAME

[jgallucci32]

@bagel-dawg The name of the Harbor project is literally docker.io, though it can be anything you like (you will need to rename anything that says docker.io in my changes to the actual name of the project you chose if different). I prefer to name it the first part of the upstream repository to make it easy to remember where the cached images come from. I will grab the full copy of nginx.conf, but what I posted above are the only changes made to nginx.conf after deploy 2.1.1 of Harbor.

[bagel-dawg]

@jgallucci32 Thanks, I actually got it working. It seemed that I was running into this issue with the Host header.

The redirect kept occurring over and over again until it failed, and then docker would failback to the the non-mirror address.

[hwchiu]

@bagel-dawg @jgallucci32
I tried the steps shown above but it didn't work, I checked the proxy.log from the Harbor but didn't see any error, do you have any preferred way to debug this issue? how do I ensure my Nginx works as I expect?

For example, I tried to pull the busybox image from my docker side and I can see HTTP requests in Harbor's proxy.log. so I think the registry-mirror setting in docker-client is correct.
Following is the log message from the proxy.log but I don't know how to debug it.

GET /v2/ HTTP/1.1" 401 76 "-" "docker/19.03.5 go/go1.12.12 git-commit/633a0ea838 kernel/4.15.0-72-generic os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.13 \x5C(linux\x5C))" 0.002 0.000 .

GET /service/token?scope=repository%3Alibrary%2Fbusybox%3Apull&service=harbor-registry HTTP/1.1" 200 948 "-" "docker/19.03.5 go/go1.12.12 git-commit/633a0ea838 kernel/4.15.0-72-generic os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.13 \x5C(linux\x5C))" 0.020 0.020 .

GET /v2/library/busybox/manifests/latest HTTP/1.1" 404 83 "-" "docker/19.03.5 go/go1.12.12 git-commit/633a0ea838 kernel/4.15.0-72-generic os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.13 \x5C(linux\x5C))" 0.005 0.000 .

Thanks

[jgallucci32]

@hwchiu The first 2 lines look correct. You get HTTP/401 for /v2 which then redirects to get the bearer token which returns HTTP/200. The third request is HTTP/404 meaning URL is not found.

You will need to first verify your proxy cache is working by attempting a pull-through request like this

docker pull harbor.domain.local/docker.io/library/busybox:latest

If that is successful, then try to get the manifest directly using a web browser with the DNS alias you use for the mirror

https://harbor-mirror.domain.local/docker.io/library/busybox/manifests/latest

This will test if you can get to the image in the proxy cache using the DNS alias with the full path to the image. Unfortunately nginx logging in K8s is not very good (or at least I haven't figured out how to get good debugging yet)

[hwchiu]

@jgallucci32
Thanks for your help, I follow your steps to verify the function, yes, I can pull the image in step (1) and I can see the image is created in the Harbor's UI.
However, when I open the URL on the second step, it shows the 404 Page not found.
but, at least I have a new way to test my nginx config.
Thanks for your help.

[hwchiu]

@jgallucci32
Fixed, I put the config in the wrong server section.
Thanks for your help again.

[jgallucci32]

@hwchiu @bagel-dawg I did some refactoring to use map variables to make it more clear what you need to set and reduces the changes made. I also added an improvement where the DNS suffix can be used in the form of <harbor_project>.harbor-mirror.domain.local so you only have to modify nginx.conf once and can configure it for X number of projects (this is very helpful with deployments with containerd which can support multiple mirrors)

Harbor can act as both a transparent proxy and Private Registry for multiple repositories:

1. Create a Proxy Cache project (i.e. docker.io) as pull-through cache to Docker Hub
2. Create DNS alias (i.e. docker-mirror.domain.local) which points to the DNS name of Harbor
3. Create wildcard DNS alias (i.e. *.docker-mirror.domainlocal) which points to the DNS name of Harbor
4. Modify the nginx.conf file for the harbor-nginx pod to rewrite the URLs for /v2 and /service when using the DNS alias
5. Configure Docker daemon to use DNS alias as registry-mirror (or use DNS subdomain of Harbor project) (NOTE: You must specify "insecure-registries" if your SSL cert is not configured for wildcard DNS subdomains)

Modifications to nginx.conf

1. Put in http { block

  # Map for proxy cache
  map $host $harbor_project {
    default 0;
    ~^docker-mirror.domain.local$ docker.io;
    ~^(?<project>.+).docker-mirror.domain.local$ $project;
  }
  map $request_uri $new_uri {
    ~^/v2/(.+)$ /v2/$harbor_project/$1;
  }
  map $args $new_args {
    ~^(?<prefix2>.*scope=repository%3A)(?<suffix2>.*)$     ${prefix2}${harbor_project}%2F$suffix2;
  }
  map $upstream_http_www_authenticate $new_header {
    ~^(?<prefix1>.*https://).*(?<suffix1>/service/token.*)$     $prefix1$host$suffix1;
  }

2. Put in second server { block

    # Rewrite for proxy cache
    if ($harbor_project != 0) {
      rewrite ^/v2/(.+)$ $new_uri;
      set $args $new_args;
    }

3. Put in location /v2/ { block

      # Modify headers for proxy cache
      proxy_hide_header Www-Authenticate;
      add_header Www-Authenticate $new_header always;

Please test this out and let me know if you run into any issues.

[jgallucci32]

@hwchiu @bagel-dawg I created a new issue (feature request) to track this and plan to work this into a PR for review #13579

[hwchiu]

@hwchiu @bagel-dawg I did some refactoring to use map variables to make it more clear what you need to set and reduces the changes made. I also added an improvement where the DNS suffix can be used in the form of <harbor_project>.harbor-mirror.domain.local so you only have to modify nginx.conf once and can configure it for X number of projects (this is very helpful with deployments with containerd which can support multiple mirrors)

Harbor can act as both a transparent proxy and Private Registry for multiple repositories:

1. Create a Proxy Cache project (i.e. docker.io) as pull-through cache to Docker Hub
2. Create DNS alias (i.e. docker-mirror.domain.local) which points to the DNS name of Harbor
3. Create wildcard DNS alias (i.e. *.docker-mirror.domainlocal) which points to the DNS name of Harbor
4. Modify the nginx.conf file for the harbor-nginx pod to rewrite the URLs for /v2 and /service when using the DNS alias
5. Configure Docker daemon to use DNS alias as registry-mirror (or use DNS subdomain of Harbor project) (NOTE: You must specify "insecure-registries" if your SSL cert is not configured for wildcard DNS subdomains)

Modifications to nginx.conf

1. Put in http { block

  # Map for proxy cache
  map $host $harbor_project {
    default 0;
    ~^docker-mirror.domain.local$ docker.io;
    ~^(?<project>.+).docker-mirror.domain.local$ $project;
  }
  map $request_uri $new_uri {
    ~^/v2/(.+)$ /v2/$harbor_project/$1;
  }
  map $args $new_args {
    ~^(?<prefix2>.*scope=repository%3A)(?<suffix2>.*)$     ${prefix2}${harbor_project}%2F$suffix2;
  }
  map $upstream_http_www_authenticate $new_header {
    ~^(?<prefix1>.*https://).*(?<suffix1>/service/token.*)$     $prefix1$host$suffix1;
  }

1. Put in second server { block

    # Rewrite for proxy cache
    if ($harbor_project != 0) {
      rewrite ^/v2/(.+)$ $new_uri;
      set $args $new_args;
    }

1. Put in location /v2/ { block

      # Modify headers for proxy cache
      proxy_hide_header Www-Authenticate;
      add_header Www-Authenticate $new_header always;

Please test this out and let me know if you run into any issues.

Thanks for your help, I will run it again when I have the time and let you know if I have any issues.

[ChristianCiach]

As shown by @jgallucci32, you could rewrite the Www-authenticate header like this:

# Map for proxy cache
map $upstream_http_www_authenticate $new_header {
    ~^(?<prefix1>.*https://).*(?<suffix1>/service/token.*)$     $prefix1$host$suffix1;
}

But this could fail if your nginx reverse proxy runs on a different port than the docker client uses to connect to it. In my example, my nginx runs at port 443 inside the container, but the port is mapped to 5000 on the host (-p 5000:443). In this case, the port (5000 in my case) will be missing from the Www-authenticate header, which causes the docker client to connect to the /service-endpoint using the wrong port (443 by default).

To fix this, we can just use the variable $http_host instead of $host, which will include the port of the request:

map $upstream_http_www_authenticate $new_header {
    ~^(?<prefix1>.*https://).*(?<suffix1>/service/token.*)$     $prefix1$http_host$suffix1;
}

[aSauerwein]

I've tried so setup a transparent registry with Harbor 2.3.2 and editing the nginx.conf but it does not work. I always get this error message

Error response from daemon: unauthorized: authorize header needed to send HEAD to repository: authorize header needed to send HEAD to repository

my modified nginx.conf: https://gist.github.com/aSauerwein/424ac58e13c7aadd619b3c75461e555f

debugging with curl shows me that the Www-Authenticateheader is missing when using my docker-mirror instead of harbor directly

[ricardojdsilva87]

Hello all, just an update with the code for nginx that enabled us to use Harbor as a dockerproxy with containerd and docker.

We are using NGINX helm chart, and added the following in the http:

    http-snippet: |
      # Map for docker.io proxy cache
      map $host $harbor_project {
        default 0;
        ~^dockerproxy(.*)xxxxxx$ proxy;
      }
      map $request_uri $new_harbor_uri {
        ~^/v2/(.+)$ /v2/$harbor_project/$1;
      }
      map $new_harbor_uri $new_containerd_uri {
        default $new_harbor_uri;
        ~^/v2/(.+)\?ns=docker\.io(.*)$ /v2/$1$2;
      }
      map $args $new_harbor_args {
        ~^(?<prefix2>.*scope=repository%3A)(?<suffix2>.*)$     ${prefix2}${harbor_project}%2F$suffix2;
      }
      map $upstream_http_www_authenticate $new_harbor_header {
        ~^(?<prefix1>.*https://).*(?<suffix1>/service/token.*)$     $prefix1$http_host$suffix1;
      }

Added after the following annotations on the proxy ingress:

  annotations:
    nginx.ingress.kubernetes.io/server-snippet: |
      # Rewrite for docker.io proxy cache
      if ($harbor_project != 0) {
        rewrite ^/v2/(.+)$ $new_harbor_uri;
        rewrite ^/v2/(.+)$ $new_containerd_uri;
        set $args $new_harbor_args;
      }
    nginx.ingress.kubernetes.io/configuration-snippet: |
      # Modify headers for proxy cache  
      proxy_hide_header Www-Authenticate;
      add_header Www-Authenticate $new_harbor_header always;

Hope it helps

[alexdepalex]

Ran into the authenticate issue with both go-containerregistry and containerd. Isn't this something that Harbor could support natively, instead of these nginx reverse proxy hacks?

[wolfganghuse]

I am also seeing 401 unauth when trying to use docker-io.docker-registry.mydomain.local like described above.
https is configured and Cert is valid.
Harbor is 2.4.1 offline install running with docker-compose.

[pedroosorio]

If anyone is still looking, here is an approach based of @jgallucci32 and @ricardojdsilva87 with no if's:

1. Add to http{ block:

map $host $harbor_project {
  default 0;
  ~^(?<project>.+).harbor.local$ $project;
}

map "$harbor_project#$request_uri" $harbor_registry_uri {
  default $request_uri;
  "~^0#.*$" $request_uri;
  "~^.+#/v2/_catalog.*$" $request_uri;
  "~^.+#/v2/(.+)(/tags/list.*)$" /v2/$harbor_project/$1$2;
  "~^.+#/v2/(.+)(/blobs/uploads.*)$" /v2/$harbor_project/$1$2;
  "~^.+#/v2/([^?]+)(.*)$" /v2/$harbor_project/$1;
}

map $args $harbor_args{
  default $args;
  ~^(?<arg_prefix>.*scope=repository%3A)(?<arg_suffix>.*)$ $arg_prefix$harbor_project%2F$arg_suffix;
}

map $upstream_http_www_authenticate $harbor_authenticate_header {
  default $upstream_http_www_authenticate;
  ~^(?<header_prefix>.*http.+//).*(?<header_suffix>/service/token.*)$ $header_prefix$http_host$header_suffix;
}

2. Add to location /v2/ block:

rewrite ^/v2/(.+)$ $harbor_registry_uri break;
proxy_hide_header Www-Authenticate;
add_header Www-Authenticate $harbor_authenticate_header always;

3. Add to location /service/ block:

set $args $harbor_args;

• Adds support for http and https installations
• Does not use if's
• Supports docker, podman and containerd
• Removes query parameters when they are not expected - https://docs.docker.com/registry/spec/api/

How to use:

1. Do this changes to nginx.conf
2. Create a registry to the given target registry
3. Create a project (proxy-cache type) using the registry named 'xyz'
4. In the machines using Harbor as the proxy, add an host 'xyz.harbor.local' pointing to the harbor IP
5. Repeat 2,3 and 4 for each registry to be mirrored

If 'harbor.local' is not a good suffix for you, make sure you change it in the hosts file as well as in the nginx configuration at map $host $harbor_project

[hiddenmarten]

If anybody has issues with their non-nginx-based ingress controller, welcome to this simple chart.

Thanks for the inspiration:
@pedroosorio, @jgallucci32 and @ricardojdsilva87

[asdorsey]

EDIT 20240528: Please see my comment below with an updated solution

@pedroosorio Thank you for your solution for this problem. I've made one slight modification to better handle images like python:3.12 (which is actually library/python:3.12, apparently). I've modified the $harbor_registry_uri map as follows:

  map "$harbor_project#$request_uri" $harbor_registry_uri {
    default $request_uri;
    "~^0#.*$" $request_uri;
    "~^.+#/v2/_catalog.*$" $request_uri;
    "~^.+#/v2/(.+)(/tags/list.*)$" /v2/$harbor_project/$1$2;
    "~^.+#/v2/(.+)(/blobs/uploads.*)$" /v2/$harbor_project/$1$2;
    "~^.+#/v2/([^/]+)(.*)$" /v2/$harbor_project/library/$1$2;    # added this line - MODIFIED 20240523
    "~^.+#/v2/([^?]+)(.*)$" /v2/$harbor_project/$1;
  }

Without the extra line that I added, I was having issues pulling "official images" that don't have a publisher name, like python:3.12.

$ docker pull docker.example.com/python:3.12
Error response from daemon: unknown: repository docker/docker/library/python not found

The line I added makes pulling official images work as expected:

$ docker pull docker.example.com/python:3.12
3.12: Pulling from python
Digest: sha256:3966b81808d864099f802080d897cef36c01550472ab3955fdd716d1c665acd6
Status: Image is up to date for docker.walkingwire.net/python:3.12
docker.walkingwire.net/python:3.12

[asdorsey]

The following is a further modification of @pedroosorio 's solution that adds the capability to bypass the mapping and rewriting process for certain domains.

I needed both transparent proxy caches and the ability to upload to other projects, but I had issues with the /blobs/uploads map pattern applying multiple times (probably because of a redirect somewhere), so I'd end up with the project name in the URI multiple times. With this change, I can instead exclude one of the domain names pointing to this Harbor instance, so when using that excluded domain, Harbor acts normally (without the rewrites and stuff for the transparent proxy cache).

Unfortunately I did have to use an if for the rewrite in the location /v2/ block to get this to work. Someone better at NGINX than me could probably eliminate that if.

1. Add to http{ block:

  map $host $harbor_project {
    default 0;
    ~^harbor.example.com$ 0;        # this is how to specify a domain name to exclude from processing
    ~^(?<project>.+).example.com$ $project;
  }

  map "$harbor_project#$request_uri" $harbor_registry_uri {
    default $request_uri;
    "~^0#.*$" $request_uri;
    "~^.+#/v2/_catalog.*$" $request_uri;
    "~^.+#/v2/(.+)(/tags/list.*)$" /v2/$harbor_project/$1$2;
    "~^.+#/v2/(.+)(/blobs/uploads.*)$" /v2/$harbor_project/$1$2;
    "~^.+#/v2/([^/]+)(.*)$" /v2/$harbor_project/library/$1$2;
    "~^.+#/v2/([^?]+)(.*)$" /v2/$harbor_project/$1;
  }
  
  map "$harbor_project#$args" $harbor_args {
    default $args;
    "~^0#.*$" $args;
    "~^.+#(?<arg_prefix>.*scope=repository%3A)(?<arg_suffix>.*)$" $arg_prefix$harbor_project%2F$arg_suffix;
  }
  
  map "$harbor_project#$upstream_http_www_authenticate" $harbor_authenticate_header {
    default $upstream_http_www_authenticate;
    "~^0#.*$" $upstream_http_www_authenticate;
    "~^.+#(?<header_prefix>.*http.+//).*(?<header_suffix>/service/token.*)$" $header_prefix$http_host$header_suffix;
  }

2. Add to location /v2/ block:

      if ($harbor_project) {
        rewrite ^/v2/(.+)$ $harbor_registry_uri break;
      }
      proxy_hide_header Www-Authenticate;
      add_header Www-Authenticate $harbor_authenticate_header always;

3. Add to location /service/ block:

    set $args $harbor_args;