Did Dockerhub rate limit affect Github Action?

[BrightRan]

Associated community ticket: https://git.luolix.topmunity/t/did-dockerhub-rate-limit-affect-github-action/128158

The customer noticed that Docker hub has updated their pricing, Download rate limit and retention policy recently. He is wondering if the download rate limit affects on Github Action.
Will the steps like below are affected?

jobs:
  my_first_job:
    steps:
      - uses: docker://alpine:3.11

jobs:
  my_first_job:
    steps:
      - run: docker pull alpine:3.11

According to the introductions from the docs about "Download rate limit", the limit seems only applies for the anonymous users with an eventual limit of 100 pulls per six hours. Logged in users will not be affected at this time.

I setup a workflow with the following step that pull an image 101 times in a loop.

- name: Test Docker download rate limits
  run: |
    docker image ls
    index=1
    while [ $index -le 101 ]
    do
      echo "---------------------------------------------------------------------------------------"
      echo "Pull image $index times"
      docker pull alpine:3.11
      docker image ls
      index=$(( $index + 1 ))
      docker image rm alpine:3.11
    done
    docker image ls

It can successfully download the image 101 times without any warning or error messages about the download rate limit.
Note: Consider there may be image cache, I have used the "docker image rm" command to remove the image every time before downloading it again.

Question:
When we try to pull some public images in the workflow, if we do not login with any of our accounts, whether GitHub has a default account to login the docker?
If not, in my above example, why it is not hit the download rate limit?

[maxim-lobanov]

Hello @BrightRan , thank you for raising it.
The new policy will be applied starting from November 1st, 2020. It is the reason why we can't reproduce it right now.
Docker will gradually introduce these rate limits, with full effects starting from November 1st, 2020.
GitHub doesn't have default account to login in the docker. I am not sure if it is possible because of security reasons.

I have found the following Docker use-cases in GA (I could miss something):
1.

jobs:
  my_first_job:
    steps:
      - run: docker pull alpine:3.11

This use-case allow customers to do login by theirselves:

steps:
  - run: |
      docker login --username ${{ secrets.docker_username }} --password ${{ secrets.docker_password }}
      docker pull alpine:3.11

jobs:
  container:
    runs-on: ubuntu-latest
    container: node:10.16-jessie
    steps:
      - run: |
          echo This job does specify a container.
          echo It runs in the container instead of the VM.
        name: Run in container

Looks like pull of such container is happening at the first step of pipeline and we can't login before somehow.

jobs:
  my_first_job:
    runs-on: ubuntu-latest
    steps:
      - uses: docker://alpine:3.11

this use case will be affected too
because even if we invoke

steps:
  - run: docker login
  - uses: docker://alpine:3.11

Container pull happens at the first step of the build before docker login

jobs:
  my_first_job:
    runs-on: ubuntu-latest
    steps:
      - uses: my_custom_action_that_uses_docker_inside@v1

In the first use-case, customer can overcome limit by login to docker as a first step of pipeline: docker login --username ${{ secrets.docker_username }} --password ${{ secrets.docker_password }}.
But looks like use-cases 2,3,4 will be broken.

DockerHub will use IP rate limit so our customers could be affected. Docker will gradually impose download rate limits with an eventual limit of 100 pulls per six hours for anonymous users.
As I can see, DockerHub even provides separate documentation section about rate limits in GitHub Actions. But actually, they are asking action owners to build-in docker login as a part of their actions and it will fix use case 4.

I don't have much experience with docker so could be wrong, may be actions/runner team can help us to understand if there is a way to login in use-cases 2,3,4. cc: @hross , @ericsciple , @TingluoHuang

cc: @thejoebourneidentity @alepauly @bryanmacfarlane

[BrightRan]

@maxim-lobanov ,
Thanks for your reply.
For the use case 3 and 4, can we add a step to login docker before the 'uses' step? Similar to the use case 1 you mentioned.

[maxim-lobanov]

@BrightRan , for 4 - yes.
For 2 and 3 it is not possible because looks like pulling of all images happen before all steps.
This yaml produces the following steps:

steps:
  - run: docker login
  - uses: docker://alpine:3.11

[BrightRan]

@maxim-lobanov ,
Yes, you're right.
I just have tried it, and the results confirm your statement.
Thanks for your detailed explanation.

[hross]

The great thing about this thread is...

We are actually actively working on providing more native login capabilities in actions. See this ADR. This should make it easier for customers to set up authentication for scenarios 2 and 3.

@dakale is adjusting the syntax with @chrispat (today, actually). We should have this ready to go by the end of the month at the latest, which should mean better support before the docker limits are imposed.

Hey @dakale can you add each of these examples to your ADR (1-4) and explain how they can be addressed with the new syntax?

[maxim-lobanov]

@hross , Happy to hear that we are working on feature that will fix the issue.
Also I think we will need a good communication with our customers to migrate them to new approach with auth.
We should avoid case when all docker related builds start to fail one day, and customers will have to fix their builds in rush to unblock builds.

[dakale]

I agree with the scenarios as described by @maxim-lobanov , I think it accurately reflects all the ways we support containers. This only affects 2 and 3. Those are where we pre-pull containers at the start of the job, thus giving no place to run commands to log in. 1 and 4 will be fine as users can run docker login in step if needed

For 2 and 3, we dont currently support pulling with authentication. That previously only meant you couldnt use "private" images in those scenarios, but now it means those scenarios may be impacted by this rate limiting issue. A single solution that allows the pull to be authenticated solves both

I should note that (due to my mistake), the link to the ADR @hross posted is out of date. We have iterated on it since, and the change is significant relative to this issue. Check the PR for the most up to date document: https://github.com/github/c2c-actions-runtime/pull/813/files

That ADR, as currently designed, only solves 2, which currently is impossible to work around (except on self-hosted). It does not address 3 at all, nor does it add anything for 1, 4, but those have viable workarounds. Thats because its designed to solve a different problem that would have accidentally helped the rate limiting problem too

I have no way to gauge what the impact of the rate limiting will be, since I dont know the range of IPs used for the hosted runner vms or how much of the available rate they might consume, but 100 per 6 hours is very very low. Its hard to make a judgement on whether its worth revising the ADR again to account for rate limiting, but I do think its worth considering given how low the limit is.

[vtbassmatt]

We're also thinking about this in Azure Pipelines. We already have a technical solution for scenarios 2 and 3 (container resources can use a service connection for authentication). Most of our current usage is anonymous, though, so we'll have to help customers understand the impact and adapt.

[nihaals]

Judging from the ADRs being private, will GitHub users not get to see the syntax until it's out (and can't have breaking changes)? Since the rate limit would almost definetely affect users, would it be worth adding a deprecation notice once a fix is available eventually forcing all users to pull while being logged in? Could help solve problems before they arise if people new to GitHub Actions start using it and expect it to work without knowing about the limit or that public CIs don't get some kind of whitelist

[chrispat]

@nihaals the syntax is very simple as we are simply adding a credentials element to the job.container and job.services key. We are also trying to understand from Docker if the library images are impacted by these changes. Based on our telemetry we believe that will have the biggest impact in how many users are immediately impacted.

[nihaals]

I made what I think is a related feature request that may make it nicer to login with Actions which sounds like will work with the new syntax

What do you think about the deprecation notice on unauthorised Docker Hub images? Probably blocked by finding out if official images are exempt and the new syntax

[rnorth]

As I understand it one scary and unresolved problem with (1) is going to be sharing Hub credentials with forked repos. Any solution that involves a docker login execution will not work, because (a) secrets are not available to forked repos, and (b) we have no other safe way to provide credentials to a docker login invocation.

Does the solution that's being worked on address this aspect?

[chrispat]

At the moment we do not have a solution to that particular problem but we are investigating options.

[maxim-lobanov]

After the latest GitHub update, you can specify credentials for container downloading: https://github.blog/changelog/2020-09-24-github-actions-private-registry-support-for-job-and-service-containers/

[rnorth]

@chrispat sorry to press on this, but is there any news on solutions to the forked repo problem I mentioned above?

[mattwelke]

@maxim-lobanov

After the latest GitHub update, you can specify credentials for container downloading

I read the blog post. The new feature, being able to authenticate with registries, looks like it will have the nice side effect of solving the rate limiting problem for job containers and service containers by allowing people to authenticate, which would stop them from being forced to share the rate limit pool with other GitHub Actions users.

However, do you know whether it will work for Docker container actions? For example, from the docs (https://docs.github.com/en/free-pro-team@latest/actions/reference/workflow-syntax-for-github-actions#example-using-a-docker-hub-action):

jobs:
  my_first_job:
    steps:
      - name: My first step
        uses: docker://alpine:3.8

In this example, the image used for the Docker container action is being pulled by the GitHub Actions runner from Docker Hub. Because credentials aren't specified for the step, I'm assuming this is being done without authenticating with Docker Hub.

The images used for Docker container actions must be public, which means people don't have to authenticate to use them, but if they choose not to authenticate as they use them, they'll be subject to the IP-based rate limiting, affected by other GitHub Actions users. Is this correct?

[rnorth]

@chrispat sorry to ask again, but it's getting worryingly close to November and, as far as I can tell, there's still no safe way for open source projects to use authenticated access to Docker Hub.

Has there been any progress in finding a way for forks of repos to be able to inherit Docker Hub authentication from the original repo?

[chrispat]

For publicly accessible containers we are working with docker hub to make sure you will not be impacted by the new rate limits. If you need private containers we still do not have a solution for that problem for forks of public repos.

[rnorth]

Thank you @chrispat! As a maintainer of an open source project that uses public Docker images, that's really good news.

[mdeggies]

For publicly accessible containers we are working with docker hub to make sure you will not be impacted by the new rate limits.

@chrispat just to confirm, does this mean we will NOT need to add dockerhub auth to all of our jobs that pull public dockerhub images?

[chrispat]

That is correct. It is important to note that this will only apply to the runners hosted by GitHub, if you are using self-hosted runners you will need to configure your own credentials.

[mdeggies]

Thank you for the confirmation! :)

[sandstrom]

@smcgivern Did they share any timeline? Is this a known issue they are working to fix?

We started using the ARM runners and ran into this problem.

[smcgivern]

They did not, sorry. They said they had no ETA currently.

[benjie]

Seem to be hitting this today; our runners are not special (not ARM, not GPU). Hoping it's just a temporary blip, but raising it in case others are also impacted.

[mvdan]

We are hitting this error today on public shared runners trying to pull a public Docker Hub image, https://hub.docker.com/_/postgres, so the rate limits are still in effect as far as we can see. Pretty annoying given that there's no easy way to cache pulled images on GitHub Actions either. We're having to resort to workarounds like authenticating with Docker Hub with a bot account, or trying to move off of Docker Hub as much as possible.

[mattwelke]

I don't think it's unreasonable to be required to authenticate with Docker Hub. If I were hosting a free service providing Docker images, I would find it helpful to know who's using up my bandwidth!

I wonder if GitHub could provide credentials for a "GitHub Actions" Docker Hub account to all GitHub Actions workflows like they do with GITHUB_TOKEN. This account would represent all GitHub Actions users authenticating only as a GitHub Actions user instead of more specifically, as themself.

GitHub could even coordinate with Docker so that Docker could impose a rate limit on this account if they want to treat it differently, considering it would represent multiple users instead of just one user.

This would allow users to start using Docker Hub with less of a chance of running into this issue without having to set up their own Docker Hub credentials in their workflows.

[mvdan]

@mattwelke I would not have an issue with reasonable rate limits if GitHub Actions made it reasonably easy to cache pulled images. Fifteen minutes of searching gave me no results other than https://github.com/marketplace/actions/docker-cache, which is an unofficial action that I would rather not have to trust.

I'm not assigning blame here, as both GitHub Actions and Docker Hub are largely free services, but it's a bit silly that their interaction runs into rate limits for no reason.

[chrispat]

I don't think it's unreasonable to be required to authenticate with Docker Hub. If I were hosting a free service providing Docker images, I would find it helpful to know who's using up my bandwidth!

I wonder if GitHub could provide credentials for a "GitHub Actions" Docker Hub account to all GitHub Actions workflows like they do with GITHUB_TOKEN. This account would represent all GitHub Actions users authenticating only as a GitHub Actions user instead of more specifically, as themself.

GitHub could even coordinate with Docker so that Docker could impose a rate limit on this account if they want to treat it differently, considering it would represent multiple users instead of just one user.

This would allow users to start using Docker Hub with less of a chance of running into this issue without having to set up their own Docker Hub credentials in their workflows.

All of the shared public runners are automatically logged in to docker hub with a shared account that has significantly higher rate limits on Docker Hub. I just ran a job that pulled that image and was not able to reproduce the issue.

[rustamabd2]

All of the shared public runners are automatically logged in to docker hub with a shared account that has significantly higher rate limits on Docker Hub. I just ran a job that pulled that image and was not able to reproduce the issue.

This issue is sporadic, it "fixes" itself after re-running the job, so it's to be expected that it can't be easily reproduced.
Could it be that the higher limit is being reached? Is it being monitored anywhere?

[ItsReddi]

Is there a way to authenticate?
We are using GitHub Runners and getting it since today.
We use "services:"

services:
      # Label used to access the service container
      local-mysql:
        image: mysql
        ports:
          - 3306:3306
        env:
          MYSQL_ROOT_PASSWORD: dev
          MYSQL_USER: dev
          MYSQL_PASSWORD: dev
          MYSQL_DATABASE: test
        options: --health-cmd "mysqladmin ping" --health-interval 10s --health-timeout 5s --health-retries 10
      redis:
        image: redis:6.0.8-alpine
        ports:
          - 6379:6379

[sileht]

We have seen this issue sporadicaly and regularly over the last few months.

[nebuk89]

If you are seeing this issue and it's in a public repo could you link me to workflow runs please so I can see date/time/other run info so we can dig in <3

[ramondelafuente]

We see this too, unfortunately not on a public repo... this is the raw log from the run:

2024-09-11T15:24:30.8594587Z Current runner version: '2.319.1'
2024-09-11T15:24:30.8619384Z ##[group]Operating System
2024-09-11T15:24:30.8620159Z Ubuntu
2024-09-11T15:24:30.8620571Z 22.04.4
2024-09-11T15:24:30.8620905Z LTS
2024-09-11T15:24:30.8621303Z ##[endgroup]
2024-09-11T15:24:30.8621707Z ##[group]Runner Image
2024-09-11T15:24:30.8622159Z Image: ubuntu-22.04
2024-09-11T15:24:30.8622601Z Version: 20240901.1.0
2024-09-11T15:24:30.8623675Z Included Software: https://github.com/actions/runner-images/blob/ubuntu22/20240901.1/images/ubuntu/Ubuntu2204-Readme.md
2024-09-11T15:24:30.8625378Z Image Release: https://github.com/actions/runner-images/releases/tag/ubuntu22%2F20240901.1
2024-09-11T15:24:30.8626343Z ##[endgroup]
2024-09-11T15:24:30.8626759Z ##[group]Runner Image Provisioner
2024-09-11T15:24:30.8627256Z 2.0.384.1
2024-09-11T15:24:30.8627658Z ##[endgroup]
2024-09-11T15:24:30.8643667Z ##[group]GITHUB_TOKEN Permissions
2024-09-11T15:24:30.8645608Z Actions: write
2024-09-11T15:24:30.8646224Z Attestations: write
2024-09-11T15:24:30.8646860Z Checks: write
2024-09-11T15:24:30.8647257Z Contents: write
2024-09-11T15:24:30.8647708Z Deployments: write
2024-09-11T15:24:30.8648095Z Discussions: write
2024-09-11T15:24:30.8648503Z Issues: write
2024-09-11T15:24:30.8648966Z Metadata: read
2024-09-11T15:24:30.8649317Z Packages: write
2024-09-11T15:24:30.8649715Z Pages: write
2024-09-11T15:24:30.8650160Z PullRequests: write
2024-09-11T15:24:30.8650559Z RepositoryProjects: write
2024-09-11T15:24:30.8651043Z SecurityEvents: write
2024-09-11T15:24:30.8651533Z Statuses: write
2024-09-11T15:24:30.8651904Z ##[endgroup]
2024-09-11T15:24:30.8654875Z Secret source: Actions
2024-09-11T15:24:30.8655603Z Prepare workflow directory
2024-09-11T15:24:30.9561395Z Prepare all required actions
2024-09-11T15:24:30.9723539Z Getting action download info
2024-09-11T15:24:31.1511156Z Download action repository 'actions/checkout@v4.1.7' (SHA:692973e3d937129bcbf40652eb9f2f61becf3332)
2024-09-11T15:24:31.2764167Z Download action repository 'shivammathur/setup-php@2.31.1' (SHA:c541c155eee45413f5b09a52248675b1a2575231)
2024-09-11T15:24:31.4537468Z Download action repository 'actions/cache@v4' (SHA:0c45773b623bea8c8e75f6c82b208c3cf94ea4f9)
2024-09-11T15:24:31.7014201Z Complete job name: integration (8.3, false)
2024-09-11T15:24:31.7901518Z ##[group]Checking docker version
2024-09-11T15:24:31.7916495Z ##[command]/usr/bin/docker version --format '{{.Server.APIVersion}}'
2024-09-11T15:24:31.8915903Z '1.45'
2024-09-11T15:24:31.8949226Z Docker daemon API version: '1.45'
2024-09-11T15:24:31.8950982Z ##[command]/usr/bin/docker version --format '{{.Client.APIVersion}}'
2024-09-11T15:24:31.9179685Z '1.45'
2024-09-11T15:24:31.9185739Z Docker client API version: '1.45'
2024-09-11T15:24:31.9192124Z ##[endgroup]
2024-09-11T15:24:31.9196456Z ##[group]Clean up resources from previous jobs
2024-09-11T15:24:31.9204020Z ##[command]/usr/bin/docker ps --all --quiet --no-trunc --filter "label=019bbc"
2024-09-11T15:24:31.9409362Z ##[command]/usr/bin/docker network prune --force --filter "label=019bbc"
2024-09-11T15:24:31.9547253Z ##[endgroup]
2024-09-11T15:24:31.9548037Z ##[group]Create local container network
2024-09-11T15:24:31.9562186Z ##[command]/usr/bin/docker network create --label 019bbc github_network_cf6957732b494bb6ae13d16c7f4baea1
2024-09-11T15:24:32.0297875Z 07d2aa32c8271575dadfe2d18188db4ec4cd8fa48b54707034fc218cb7be4f1c
2024-09-11T15:24:32.0327450Z ##[endgroup]
2024-09-11T15:24:32.0446731Z ##[group]Starting mailer service container
2024-09-11T15:24:32.0515749Z ##[command]/usr/bin/docker pull axllent/mailpit
2024-09-11T15:24:32.0614358Z Using default tag: latest
2024-09-11T15:24:37.2900135Z latest: Pulling from axllent/mailpit
2024-09-11T15:24:37.3058632Z toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit
2024-09-11T15:24:37.3085866Z ##[warning]Docker pull failed with exit code 1, back off 3.575 seconds before retry.
2024-09-11T15:24:40.8847426Z ##[command]/usr/bin/docker pull axllent/mailpit
2024-09-11T15:24:40.8950838Z Using default tag: latest
2024-09-11T15:24:41.0395882Z Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit
2024-09-11T15:24:41.0415680Z ##[warning]Docker pull failed with exit code 1, back off 5.34 seconds before retry.
2024-09-11T15:24:46.3798025Z ##[command]/usr/bin/docker pull axllent/mailpit
2024-09-11T15:24:46.3910865Z Using default tag: latest
2024-09-11T15:24:46.5341928Z Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit
2024-09-11T15:24:46.5442001Z ##[error]Docker pull failed with exit code 1
2024-09-11T15:24:46.5572271Z Remove container network: github_network_cf6957732b494bb6ae13d16c7f4baea1
2024-09-11T15:24:46.5577160Z ##[command]/usr/bin/docker network rm github_network_cf6957732b494bb6ae13d16c7f4baea1
2024-09-11T15:24:46.7412872Z github_network_cf6957732b494bb6ae13d16c7f4baea1
2024-09-11T15:24:46.7597309Z Cleaning up orphan processes

And a relevant part of the workflow:

 integration:
    runs-on: ubuntu-latest
    continue-on-error: ${{ matrix.allow-failure }}
    strategy:
      matrix:
        php-versions: ['8.3']
        allow-failure: [false]

    services:
      mailer:
        image: axllent/mailpit # Docker Hub image: https://hub.docker.com/r/axllent/mailpit
        ports:
          - 1025:1025
          - 8025:8025
      mariadb:
        image: mariadb:10.11 # https://hub.docker.com/_/mariadb
        env:
          MYSQL_ALLOW_EMPTY_PASSWORD: yes
        ports:
          - 3306:3306
        options: --health-cmd="mysqladmin ping" --health-interval=10s --health-timeout=5s --health-retries=3

    steps:
      - uses: actions/checkout@v4.1.7
      ...etc

On a side-note:
Is there a better place for a renewed discussion on this topic than a closed issie from 2020? (open question, since research did bring me here).

[ramondelafuente]

@nebuk89 Was the comment above useful? (i.e. should I post the moar logs if we see this happening again)
We've only had two instances (in the past 2 weeks) so not a frequent occurrence for us.

[Cactusbone]

Got the `toomanyrequests' error on a dokku Github actions pipeline:
On the same PR:

• https://github.com/dokku/dokku/actions/runs/10880985283/job/30222976381?pr=7147
• https://github.com/dokku/dokku/actions/runs/10880985283/job/30222979428?pr=7147

Hope this helps :)

[y-nk]

this hitting me on a private repo too. impossible to know exactly why/how.

[jon-nfc]

If you are seeing this issue and it's in a public repo could you link me to workflow runs please so I can see date/time/other run info so we can dig in <3

@nebuk89, Seen this today too when attempting to pull docker.io/library/python:3.11-alpine3.19.

job link -> https://github.com/nofusscomputing/centurion_erp/actions/runs/11330393667/job/31508120435.

[vbalumuri]

Same here, getting the rate limit error on our private repo while trying to pull python:3.7-slim

429 Too Many Requests - Server message: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading

[drcongo]

Also just started getting this today.

[nvuillam]

I had docker pull rate llimit issues when I was authenticated to Docker Hub.
So I authenticated only when necessary, to remain unauthenticated most of the time -> no more docker pull issues

Since today: Pull issues again... i'd never seen rate limit issue from GitHub Actions with unauthenticated user :/

[MikeSchlosser16]

Also getting this today on a private repo!

[carloseduardosx]

+1. I'm facing the same issue while fetching public images from the Docker Registry on my GHA Workflows.

[KlausVii]

This was an issue on docker's side docker/hub-feedback#2414