Add option to skip getLabels for index.docker.io to reduce impact on Docker Hub rate limit #29582
Replies: 2 comments 7 replies
-
I recommend that you set https://docs.renovatebot.com/self-hosted-experimental/#renovate_x_docker_hub_tags in order to reduce the limit from index.docker.io. It's still got rate limits but we don't seem to hit them as much due to its more efficient API results and it will be the default for Renovate v38 You're welcome to submit a PR for your idea too though, and report back on the findings. If the only queries which affect rate limits are for the URLs and we cache them heavily then I wouldn't expect it to use up much API calls anyway. |
Beta Was this translation helpful? Give feedback.
-
I can confirm that caching keeps these requests at a minimum. So far working with Renovate is great! Thanks a lot for this great work! In scenarios where the label information might not be needed at all (e.g. frequent dry-run mode lookups) especially when multiple independent systems that are running Renovate share the same IP towards DockerHub, it would be nice to opt out of the rate limited requests. I was able to confirm that the label requests mentioned above were counted against the DockerHub rate limit. I've added the proposed variable and tested the changes of #29624 on a test repository containing the following Dockerfile. FROM sonarsource/sonar-scanner-cli:4.8.0 I've tested the changes by running Renovate against the repository in dry-run=lookup mode. Without the variable
After querying the DockerHub Rate Limit I noticed a decrease in remaining. After setting the variable
After querying the DockerHub Rate Limit time I noticed no decrease. |
Beta Was this translation helpful? Give feedback.
-
Tell us more.
DockerHub defines a rate limit on GET
/v2/*/manifests/*
requests.Renovate uses such requests to identify gitRef, sourceUrl and homepage for the release.
renovate/lib/modules/datasource/docker/index.ts
Lines 1098 to 1115 in 7516af6
The labels seem to be already considered optional for docker library images.
https://github.com/renovatebot/renovate/blob/37.401.1/lib/modules/datasource/docker/index.ts#L459-L465
I suggest to add a self-hosted experimental environment variable
RENOVATE_X_DISABLE_DOCKER_HUB_LABELS
which disables all label requests to https://index.docker.io to optinally reduce the impact on the DockerHub rate limit further.https://docs.renovatebot.com/self-hosted-experimental/#self-hosted-experimental-environment-variables
related to:
#17702
#19036
#21964
Beta Was this translation helpful? Give feedback.
All reactions