Ratelimiting error when downloading vulnerability db from ghcr.io #389

HenrikDK · 2024-09-18T11:01:34Z

Hi, we're using trivy to scan our containers, lately we've been seeing an increase number of rate-limiting errors when trivy is downloading the vulnerability database.

"2024-09-18T10:40:44Z FATAL Fatal error init error: DB error: failed to download vulnerability DB: database download error: oci download error: failed to fetch the layer: GET https://ghcr.io/v2/aquasecurity/trivy-db/blobs/sha256:11c57f2012b2ac112256f94aa404e1feb7e1b7a5787598946b87149115cdb43d: TOOMANYREQUESTS: retry-after: 129.163µs, allowed: 44000/minute"

My guess is this is a global ratelimit as i can't imagine our low number of devs are causing 700+ requests a second.

I have in the meantime discovered that these scans are only used for SBOM generation on our end so we don't need to download the vulnerability database everytime, but i though this issue should be raised as i can't imagine we are the only ones seeing these errors.

simar7 · 2024-09-18T18:57:54Z

Thanks for the report, we will look into it.

benglewis · 2024-09-19T12:41:13Z

I also saw this right now :/ Any ideas why?

aquasecurity/trivy-action#389

billhammond-dev · 2024-09-19T16:53:40Z

I believe this is currently causing problems with anyone using the trivy action. We have had to turn it off on some workflows. I'm not sure what the long term solution might be - if GH cannot increase the global rate limit for the artifact pull then maybe it needs to be in a public AWS S3 bucket or something similar?

billhammond-dev · 2024-09-19T16:57:49Z

Also see https://github.com/orgs/community/discussions/139074

billhammond-dev · 2024-09-20T14:08:16Z

From My PR above, a workaround suggested by someone else:

- uses: aquasecurity/trivy-action@0.24.0
  with:
    ...
  env:
    TRIVY_DB_REPOSITORY: <something else than ghcr.io>
    TRIVY_JAVA_DB_REPOSITORY: <something else than ghcr.io>```

nelsonleblanc-rl · 2024-09-20T16:48:16Z

Does anyone know how to get trivy-action to auth with a privately hosted trivy-db repo? I can get it working fine with normal trivy on local, but trivy-action does not work with either docker/login-action or the usual echo $GITHUB_TOKEN | docker login ghcr.io -u USERNAME --password-stdin

2024-09-20T16:39:01Z FATAL Fatal error init error: DB error: failed to download vulnerability DB: database download error: OCI repository error: 1 error occurred: * GET https://ghcr.io/token?scope=repository%3Aprivate-github-org%2Ftrivy-db%3Apull&service=ghcr.io: UNAUTHORIZED: authentication required

billhammond-dev · 2024-09-20T17:19:54Z

I was able to get it to work with ECR only using an OIDC login via the configure-aws-credentials action used right before the trivy action. It is not using docker to pull the artifact as it is not a docker image.

9838183063 · 2024-09-22T05:23:38Z

I am poor student

srenatus · 2024-09-23T09:25:20Z

I have no long-term tests yet, but from my understanding of GH's rate limiting, just providing a token of any sort will give you higher quotas? If that's the case, the following should help:

      - name: Run Trivy scan on image
        uses: aquasecurity/trivy-action@0.24.0
        with:
          [... your config ...]
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

baksetercx · 2024-09-23T12:09:27Z

I have no long-term tests yet, but from my understanding of GH's rate limiting, just providing a token of any sort will give you higher quotas? If that's the case, the following should help:
      - name: Run Trivy scan on image
        uses: aquasecurity/trivy-action@0.24.0
        with:
          [... your config ...]
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

I've tried logging in to GHCR via docker/login-action before running Trivy CLI (not action), and I am still getting lots of 429 errors.

nnellanspdl · 2024-09-23T13:24:02Z

From My PR above, a workaround suggested by someone else:

- uses: aquasecurity/trivy-action@0.24.0
  with:
    ...
  env:
    TRIVY_DB_REPOSITORY: <something else than ghcr.io>
    TRIVY_JAVA_DB_REPOSITORY: <something else than ghcr.io>```

So, if I understand this correctly:

I, as the consumer of this action, must download copies of these DBs and store them on my own registry. Then, I must pass environment variables to the action which point at my copies of the DBs. Is that correct?

How often are these DBs updated?

ybelMekk · 2024-09-23T14:06:32Z

@nnellanspdl think its at 00:00 every day? but im not sure.

But anyway this workaround is a hustle to host them self if u need to update them every day

NicholasFiorentini · 2024-09-23T15:24:55Z

I have no long-term tests yet, but from my understanding of GH's rate limiting, just providing a token of any sort will give you higher quotas? If that's the case, the following should help:
      - name: Run Trivy scan on image
        uses: aquasecurity/trivy-action@0.24.0
        with:
          [... your config ...]
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
I've tried logging in to GHCR via docker/login-action before running Trivy CLI (not action), and I am still getting lots of 429 errors.

Same for me, it doesn't seem to have significant effects.

NicholasFiorentini · 2024-09-23T15:35:10Z

I have no long-term tests yet, but from my understanding of GH's rate limiting, just providing a token of any sort will give you higher quotas? If that's the case, the following should help:
      - name: Run Trivy scan on image
        uses: aquasecurity/trivy-action@0.24.0
        with:
          [... your config ...]
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

I'm trying with:

env:
    ACTIONS_RUNTIME_TOKEN: ${{ secrets.GITHUB_TOKEN }}

I spawned multiple parallel ci/cd actions, and this seems more reliable.

BRONSOLO · 2024-09-23T16:57:30Z

If anyone is going the route of uploading the Trivy DB to their own registry, I've had success using https://github.com/oras-project/setup-oras

Something like:

  vendor-trivy-db:
    runs-on: ubuntu-latest
    steps:
      - name: Vendor latest trivy db
        uses: oras-project/setup-oras@v1
      - run: |
          oras pull ghcr.io/aquasecurity/trivy-db:2
          oras login -u ${{ secrets.REGISTRY_USERNAME }} -p ${{ secrets.REGISTRY_TOKEN }} YOUR_REGISTRY
          oras push YOUR_REGISTRY \
            db.tar.gz:application/vnd.aquasec.trivy.db.layer.v1.tar+gzip \
            --artifact-type application/vnd.aquasec.trivy.config.v1+json

eugentius · 2024-09-23T17:57:44Z

I setup AWS ECR pull-throuhg cache for trivy-db and trivy-java-db , modified action:

      - name: Run Trivy vulnerability scanner
        uses: aquasecurity/trivy-action@0.24.0
        with:
          image-ref: ${{ env.DOCKER_IMAGE_TO_SCAN }}
          format: 'table'
          exit-code: '1' 
          ignore-unfixed: true
          vuln-type: 'os,library'
          severity: 'CRITICAL,HIGH'
        env:
          TRIVY_DB_REPOSITORY: <ECR_ID>.dkr.ecr.us-east-1.amazonaws.com/github/ghcr.io/aquasecurity/trivy-db
          TRIVY_JAVA_DB_REPOSITORY: <ECR_ID>.dkr.ecr.us-east-1.amazonaws.com/github/ghcr.io/aquasecurity/trivy-java-db
          TRIVY_DEBUG: true

but pulling of trivy-db fails with:

2024-09-23T16:16:12Z	INFO	Downloading DB...	repository="<ECR_ID>.dkr.ecr.us-east-1.amazonaws.com/github/ghcr.io/aquasecurity/trivy-db"
2024-09-23T16:16:12Z	DEBUG	No metadata file
2024-09-23T16:16:17Z	DEBUG	Credential error	err="failed to get authorization token: operation error ECR: GetAuthorizationToken, get identity: get credentials: failed to refresh cached credentials, no EC2 IMDS role found, operation error ec2imds: GetMetadata, canceled, context deadline exceeded"
2024-09-23T16:16:17Z	FATAL	Fatal error	init error: DB error: failed to download vulnerability DB: database download error: OCI repository error: 1 error occurred:
	* GET https://<ECR_ID>.dkr.ecr.us-east-1.amazonaws.com/v2/github/ghcr.io/aquasecurity/trivy-db/manifests/2: unexpected status code 401 Unauthorized: Not Authorized

Docker is logged-in.
If I run trivy binary locally or on runner, it works fine:

runner@runner-set-xs-djfqb-0:/tmp$ export TRIVY_DB_REPOSITORY=<ECR_ID>.dkr.ecr.us-east-1.amazonaws.com/github/ghcr.io/aquasecurity/trivy-db
runner@runner-set-xs-djfqb-0:/tmp$ export TRIVY_JAVA_DB_REPOSITORY=<ECR_ID>.dkr.ecr.us-east-1.amazonaws.com/github/ghcr.io/aquasecurity/trivy-java-db
runner@runner-set-xs-djfqb-0:/tmp$ trivy image    --format table --exit-code  1 --ignore-unfixed --vuln-type  os,library --severity  CRITICAL,HIGH  <ECR_ID>.dkr.ecr.us-east-1.amazonaws.com/my-awesome-app:1.23.0
2024-09-23T16:35:04Z	WARN	'--vuln-type' is deprecated. Use '--pkg-types' instead.
2024-09-23T16:35:04Z	INFO	Adding schema version to the DB repository for backward compatibility	repository="<ECR_ID>.dkr.ecr.us-east-1.amazonaws.com/github/ghcr.io/aquasecurity/trivy-db:2"
2024-09-23T16:35:04Z	INFO	Adding schema version to the Java DB repository for backward compatibility	repository="<ECR_ID>.dkr.ecr.us-east-1.amazonaws.com/github/ghcr.io/aquasecurity/trivy-java-db:1"
2024-09-23T16:35:04Z	INFO	[db] Need to update DB
2024-09-23T16:35:04Z	INFO	[db] Downloading DB...	repository="<ECR_ID>.dkr.ecr.us-east-1.amazonaws.com/github/ghcr.io/aquasecurity/trivy-db:2"
53.56 MiB / 53.56 MiB [------------------------------------------------------------------------------------------------------

Has somebody tried to pull trivy-db from AWS ECR using action?

billhammond-dev · 2024-09-23T20:23:49Z

Yes so you can pull from ECR pull through but only if you do an OIDC set-aws-credentials action first before the trivy action. Im not sure why yet that you cannot use anything but OIDC, or at least I can't seem to get regular role assumption to work. Docker login doesnt help you as the container doesnt try to pull the DB using docker commands.

If you try a docker pull you will get the unsupported media type error as the above post, as the artifact isnt an 'image'

nbenmoody-tesouro · 2024-09-23T20:25:06Z

Ah, thanks. I was logged in under the incorrect account when I posted originally. That's what I was wondering, @billhammond-dev !

nbenmoody-tesouro · 2024-09-23T20:25:21Z

This was my error, for anyone else who runs into it:

latest: Pulling from github/ghcr.io/aquasecurity/trivy-db
unsupported media type application/vnd.aquasec.trivy.config.v1+json

nnellanspdl · 2024-09-23T20:35:38Z

@nnellanspdl think its at 00:00 every day? but im not sure.

But anyway this workaround is a hustle to host them self if u need to update them every day

Thanks. Yes, this is a lot to ask of consumers of your action.

Nava-JoshLong · 2024-09-23T21:53:45Z

I'm guessing it would be too much work to update the logic for pulling the file to allow passing it the file directly? We could setup a workflow to pull and stash the image every X hours, and then in the workflow that uses the image, we pull the file from the stash to use. It'd lower the amount of hits by users, and we wouldn't need to host it in AWS and pay

simar7 · 2024-09-24T04:42:50Z

ACTIONS_RUNTIME_TOKEN

@NicholasFiorentini that's interesting, would you mind creating a PR to document this in the repo? If possible, could you also reference where this environment variable is documented?

jpalomaki · 2024-09-24T07:16:15Z

FWIW, here's a sample snippet for using AWS ECR pull through cache repositories using OIDC for AWS auth.

Pull through cache ECR repositories (for hosting the cached trivy DB artifacts) must be configured prior to running this workflow, see documentation.

- name: Setup AWS credentials
  uses: aws-actions/configure-aws-credentials@v4
  with:
    aws-region: ...
    role-to-assume: <role, assumable through OIDC, that can pull from the cache ECR repositories>

- id: ecr-login
  name: Login to ECR
  uses: aws-actions/amazon-ecr-login@v2

...

- name: Run trivy scan
  uses: aquasecurity/trivy-action@0.24.0
  with:
    ...
  env:
    TRIVY_DB_REPOSITORY: ${{ steps.ecr-login.outputs.registry }}/github/aquasecurity/trivy-db:2
    TRIVY_JAVA_DB_REPOSITORY: ${{ steps.ecr-login.outputs.registry }}/github/aquasecurity/trivy-java-db:1

Per AWS documentation:

When a cached image is pulled through the Amazon ECR private registry URI, Amazon ECR checks the upstream repository at least once every 24 hours to verify whether the cached image is the latest version. If there is a newer image in the upstream registry, Amazon ECR attempts to update the cached image. This timer is based off the last pull of the cached image.

nvuillam · 2024-11-11T17:02:21Z

Is is the same with trivy java db ?
All my trivy actions passes exept those who contain java , I spend my days relaunching them >_<

Darwiner · 2024-11-11T17:34:26Z

Is is the same with trivy java db ? All my trivy actions passes exept those who contain java , I spend my days relaunching them >_<

Yes, the same logic seems to be used for trivy-java-db (updates going to all 3 repositories) - https://github.com/aquasecurity/trivy-java-db/blob/main/.github/workflows/cron.yml.

With the small difference of the workflow running once every day at midnight, instead of every 6 hours.

nvuillam · 2024-11-11T20:06:13Z

Cache is is restored, but Java DB still unavailable and provoking the crash

Why ?

I love trivy when it works, but since september it became an unreliable tool randomly failing CI jobs, I'm afraid you are slowly killing your great product by doing nothing permanent to solve the issue :/

Cache Size: ~36 MB (37525328 B)
/usr/bin/tar -xf /home/runner/work/_temp/66eda285-1ded-4727-8d16-b1924fbba231/cache.tzst -P -C /home/runner/work/megalinter/megalinter --use-compress-program unzstd
Cache restored successfully
Cache restored from key: trivy-binary-v0.56.1-Linux-X64
Run echo /home/runner/.local/bin/trivy-bin >> $GITHUB_PATH
  echo /home/runner/.local/bin/trivy-bin >> $GITHUB_PATH
  shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
  env:
    DOCKER_METADATA_OUTPUT_VERSION: beta
    DOCKER_METADATA_OUTPUT_TAGS: ghcr.io/oxsecurity/megalinter-worker-ci_light:beta
    DOCKER_METADATA_OUTPUT_LABELS: org.opencontainers.image.created=2024-11-11T19:59:35.390Z
  org.opencontainers.image.description=🦙 MegaLinter analyzes 50 languages, 22 formats, 21 tooling formats, excessive copy-pastes, spelling mistakes and security issues in your repository sources with a GitHub Action, other CI tools or locally.
  org.opencontainers.image.licenses=AGPL-3.0
  org.opencontainers.image.revision=ba3542f4a4e7a5848cb09b030dcd919d8acfcb9f
  org.opencontainers.image.source=https://github.com/oxsecurity/megalinter
  org.opencontainers.image.title=megalinter
  org.opencontainers.image.url=https://github.com/oxsecurity/megalinter
  org.opencontainers.image.version=beta
    DOCKER_METADATA_OUTPUT_ANNOTATIONS: manifest:org.opencontainers.image.created=2024-11-11T19:59:35.390Z
  manifest:org.opencontainers.image.description=🦙 MegaLinter analyzes 50 languages, 22 formats, 21 tooling formats, excessive copy-pastes, spelling mistakes and security issues in your repository sources with a GitHub Action, other CI tools or locally.
  manifest:org.opencontainers.image.licenses=AGPL-3.0
  manifest:org.opencontainers.image.revision=ba3542f4a4e7a5848cb09b030dcd919d8acfcb9f
  manifest:org.opencontainers.image.source=https://github.com/oxsecurity/megalinter
  manifest:org.opencontainers.image.title=megalinter
  manifest:org.opencontainers.image.url=https://github.com/oxsecurity/megalinter
  manifest:org.opencontainers.image.version=beta
    DOCKER_METADATA_OUTPUT_JSON: {"tags":["ghcr.io/oxsecurity/megalinter-worker-ci_light:beta"],"labels":{"org.opencontainers.image.created":"2024-11-11T19:59:35.390Z","org.opencontainers.image.description":"🦙 MegaLinter analyzes 50 languages, 22 formats, 21 tooling formats, excessive copy-pastes, spelling mistakes and security issues in your repository sources with a GitHub Action, other CI tools or locally.","org.opencontainers.image.licenses":"AGPL-3.0","org.opencontainers.image.revision":"ba3542f4a4e7a5848cb09b030dcd919d8acfcb9f","org.opencontainers.image.source":"https://github.com/oxsecurity/megalinter","org.opencontainers.image.title":"megalinter","org.opencontainers.image.url":"https://github.com/oxsecurity/megalinter","org.opencontainers.image.version":"beta"},"annotations":["manifest:org.opencontainers.image.created=2024-11-11T19:59:35.390Z","manifest:org.opencontainers.image.description=🦙 MegaLinter analyzes 50 languages, 22 formats, 21 tooling formats, excessive copy-pastes, spelling mistakes and ..
    DOCKER_METADATA_OUTPUT_BAKE_FILE_TAGS: /home/runner/work/_temp/docker-actions-toolkit-hg7FBQ/docker-metadata-action-bake-tags.json
    DOCKER_METADATA_OUTPUT_BAKE_FILE_LABELS: /home/runner/work/_temp/docker-actions-toolkit-hg7FBQ/docker-metadata-action-bake-labels.json
    DOCKER_METADATA_OUTPUT_BAKE_FILE_ANNOTATIONS: /home/runner/work/_temp/docker-actions-toolkit-hg7FBQ/docker-metadata-action-bake-annotations.json
    DOCKER_METADATA_OUTPUT_BAKE_FILE: /home/runner/work/_temp/docker-actions-toolkit-hg7FBQ/docker-metadata-action-bake.json
    ACTIONS_RUNTIME_TOKEN: ***
Run echo "date=$(date +'%Y-%m-%d')" >> $GITHUB_OUTPUT
Run actions/cache@v4
  
Received 197132288 of 686605845 (28.7%), 188.0 MBs/sec
Received 507510784 of 686605845 (73.9%), 241.9 MBs/sec
Cache Size: ~655 MB (686605845 B)
/usr/bin/tar -xf /home/runner/work/_temp/d3fa340d-df56-445e-852c-eb69c086518a/cache.tzst -P -C /home/runner/work/megalinter/megalinter --use-compress-program unzstd
Received 686605845 of 686605845 (100.0%), 218.1 MBs/sec
Cache restored successfully
Cache restored from key: cache-trivy-2024-11-11
Run echo "$GITHUB_ACTION_PATH" >> $GITHUB_PATH
Run # Note: There is currently no way to distinguish between undefined variables and empty strings in GitHub Actions.
Run entrypoint.sh
Running Trivy with options: trivy image ghcr.io/oxsecurity/megalinter-ci_light:beta
2024-11-11T20:03:11Z	INFO	[vuln] Vulnerability scanning is enabled
2024-11-11T20:03:12Z	INFO	[python] License acquired from METADATA classifiers may be subject to additional terms	name="pip" version="24.3.1"
2024-11-11T20:03:12Z	INFO	[python] License acquired from METADATA classifiers may be subject to additional terms	name="distlib" version="0.3.9"
2024-11-11T20:03:12Z	INFO	[python] License acquired from METADATA classifiers may be subject to additional terms	name="pip" version="24.3.1"
2024-11-11T20:03:12Z	INFO	[python] License acquired from METADATA classifiers may be subject to additional terms	name="PyYAML" version="6.0.2"
2024-11-11T20:03:12Z	INFO	[python] License acquired from METADATA classifiers may be subject to additional terms	name="pip" version="24.3.1"
2024-11-11T20:03:12Z	INFO	[python] License acquired from METADATA classifiers may be subject to additional terms	name="yamllint" version="1.35.1"
2024-11-11T20:03:13Z	INFO	[javadb] Downloading Java DB...
2024-11-11T20:03:13Z	INFO	[javadb] Downloading artifact...	repo="ghcr.io/aquasecurity/trivy-java-db:1"
2024-11-11T20:03:14Z	INFO	[python] License acquired from METADATA classifiers may be subject to additional terms	name="DataProperty" version="1.0.1"
2024-11-11T20:03:14Z	INFO	[python] License acquired from METADATA classifiers may be subject to additional terms	name="Deprecated" version="1.2.14"
2024-11-11T20:03:14Z	INFO	[python] License acquired from METADATA classifiers may be subject to additional terms	name="GitPython" version="3.1.43"
2024-11-11T20:03:14Z	INFO	[python] License acquired from METADATA classifiers may be subject to additional terms	name="PyJWT" version="2.9.0"
2024-11-11T20:03:14Z	INFO	[python] License acquired from METADATA classifiers may be subject to additional terms	name="PyYAML" version="6.0.2"
2024-11-11T20:03:14Z	INFO	[python] License acquired from METADATA classifiers may be subject to additional terms	name="azure-devops" version="6.0.0b4"
2024-11-11T20:03:14Z	INFO	[python] License acquired from METADATA classifiers may be subject to additional terms	name="certifi" version="2024.8.30"
2024-11-11T20:03:14Z	INFO	[python] License acquired from METADATA classifiers may be subject to additional terms	name="cffi" version="1.17.1"
2024-11-11T20:03:14Z	INFO	[python] License acquired from METADATA classifiers may be subject to additional terms	name="chardet" version="5.2.0"
2024-11-11T20:03:14Z	INFO	[python] License acquired from METADATA classifiers may be subject to additional terms	name="charset-normalizer" version="3.4.0"
2024-11-11T20:03:14Z	ERROR	[javadb] Failed to download artifact	repo="ghcr.io/aquasecurity/trivy-java-db:1" err="oci download error: failed to fetch the layer: GET https://ghcr.io/v2/aquasecurity/trivy-java-db/blobs/sha256:b5c382f3eeef336fb5fa8c7eeacb9e4b3ef39348c5821f45[157](https://github.com/oxsecurity/megalinter/actions/runs/11785106494/job/32826338548#step:12:161)49f34feaef83a: TOOMANYREQUESTS: retry-after: 508.189µs, allowed: 44000/minute"
2024-11-11T20:03:14Z	INFO	[python] License acquired from METADATA classifiers may be subject to additional terms	name="cryptography" version="43.0.3"
2024-11-11T20:03:14Z	INFO	[python] License acquired from METADATA classifiers may be subject to additional terms	name="gitdb" version="4.0.11"
2024-11-11T20:03:14Z	INFO	[python] License acquired from METADATA classifiers may be subject to additional terms	name="isodate" version="0.7.2"
2024-11-11T20:03:14Z	INFO	[python] License acquired from METADATA classifiers may be subject to additional terms	name="lark-parser" version="0.7.8"
2024-11-11T20:03:14Z	INFO	[python] License acquired from METADATA classifiers may be subject to additional terms	name="mbstrdecoder" version="1.1.3"
2024-11-11T20:03:14Z	INFO	[python] License acquired from METADATA classifiers may be subject to additional terms	name="msrest" version="0.6.21"
2024-11-11T20:03:14Z	INFO	[python] License acquired from METADATA classifiers may be subject to additional terms	name="multiprocessing-logging" version="0.3.4"
2024-11-11T20:03:14Z	INFO	[python] License acquired from METADATA classifiers may be subject to additional terms	name="oauthlib" version="3.2.2"
2024-11-11T20:03:14Z	INFO	[python] License acquired from METADATA classifiers may be subject to additional terms	name="pathvalidate" version="3.2.1"
2024-11-11T20:03:14Z	INFO	[python] License acquired from METADATA classifiers may be subject to additional terms	name="pychalk" version="2.0.1"
2024-11-11T20:03:14Z	INFO	[python] License acquired from METADATA classifiers may be subject to additional terms	name="pycparser" version="2.22"
2024-11-11T20:03:14Z	INFO	[python] License acquired from METADATA classifiers may be subject to additional terms	name="pytablewriter" version="1.2.0"
2024-11-11T20:03:14Z	INFO	[python] License acquired from METADATA classifiers may be subject to additional terms	name="python-dateutil" version="2.9.0.post0"
2024-11-11T20:03:14Z	INFO	[python] License acquired from METADATA classifiers may be subject to additional terms	name="python-gitlab" version="5.0.0"
2024-11-11T20:03:14Z	INFO	[python] License acquired from METADATA classifiers may be subject to additional terms	name="pytz" version="2024.2"
2024-11-11T20:03:14Z	INFO	[python] License acquired from METADATA classifiers may be subject to additional terms	name="redis" version="5.2.0"
2024-11-11T20:03:14Z	INFO	[python] License acquired from METADATA classifiers may be subject to additional terms	name="regex" version="2024.11.6"
2024-11-11T20:03:14Z	INFO	[python] License acquired from METADATA classifiers may be subject to additional terms	name="requests" version="2.32.3"
2024-11-11T20:03:14Z	INFO	[python] License acquired from METADATA classifiers may be subject to additional terms	name="requests-oauthlib" version="2.0.0"
2024-11-11T20:03:14Z	INFO	[python] License acquired from METADATA classifiers may be subject to additional terms	name="requests-toolbelt" version="1.0.0"
2024-11-11T20:03:14Z	INFO	[python] License acquired from METADATA classifiers may be subject to additional terms	name="autocommand" version="2.2.2"
2024-11-11T20:03:14Z	INFO	[python] License acquired from METADATA classifiers may be subject to additional terms	name="typeguard" version="4.3.0"
2024-11-11T20:03:14Z	INFO	[python] License acquired from METADATA classifiers may be subject to additional terms	name="six" version="1.16.0"
2024-11-11T20:03:14Z	INFO	[python] License acquired from METADATA classifiers may be subject to additional terms	name="smmap" version="5.0.1"
2024-11-11T20:03:14Z	INFO	[python] License acquired from METADATA classifiers may be subject to additional terms	name="tabledata" version="1.3.3"
2024-11-11T20:03:14Z	INFO	[python] License acquired from METADATA classifiers may be subject to additional terms	name="tcolorpy" version="0.1.6"
2024-11-11T20:03:14Z	INFO	[python] License acquired from METADATA classifiers may be subject to additional terms	name="terminaltables" version="3.1.10"
2024-11-11T20:03:14Z	INFO	[python] License acquired from METADATA classifiers may be subject to additional terms	name="typepy" version="1.3.2"
2024-11-11T20:03:14Z	INFO	[python] License acquired from METADATA classifiers may be subject to additional terms	name="wrapt" version="1.16.0"
2024-11-11T20:03:15Z	FATAL	Fatal error	image scan error: scan error: scan failed: failed analysis: analyze error: pipeline error: failed to analyze layer (sha256:d7b18301204574bcfb563ac3192b4a13c33a82dc0d54be8314ec1d793a965c7f): post analysis error: post analysis error: Unable to initialize the Java DB: Java DB update failed: OCI artifact error: failed to download Java DB: failed to download artifact from any source

More info at: ballerina-platform/ballerina-library#7357 that fixes: aquasecurity/trivy-action#389

nnellanspdl · 2024-11-18T20:54:55Z

Trivy was recently updated to v0.57.1 and it now specifies multiple registry fallbacks, by default.

Edit, specifically it uses these:
db: [mirror.gcr.io/aquasec/trivy-db:2,ghcr.io/aquasecurity/trivy-db:2]
java-db: [mirror.gcr.io/aquasec/trivy-java-db:1,ghcr.io/aquasecurity/trivy-java-db:1]

However, the trivy action has not been updated quite yet. The latest action version at this time is v0.28.0 which still uses a default trivy version of v0.56.1. But, you can override the version of trivy by specifying the version input on the action:

- name: 'Run Trivy'
  uses: aquasecurity/trivy-action@0.28.0
  with:
    version: 'v0.57.1'

This worked successfully for me. I did not have to specify any custom registries by env variables. In my test, the vulnerability database was automatically downloaded from mirror.gcr.io/aquasec/trivy-db:2

askpatrickw · 2024-11-19T20:34:05Z

There is a PR in progress for this ... but it needs a little love and then approval.
#433

mythi · 2024-11-20T09:22:15Z

FWIW, with v0.29.0 of this action (that uses v0.57.1 by default), I get:

[vulndb] Artifact successfully downloaded	repo="mirror.gcr.io/aquasec/trivy-db:2"

Pulling the trivy DB of CVE leaded to several of failures due to ghcr limitations [1]. trivy was modified to use other repositories as mirrors for the DB, this change landed in 0.57.1 [2]. The trivy-action was updated recently to 0.57.1, thus pulling from the mirrors [3]. Signed-off-by: Francis Laniel <flaniel@linux.microsoft.com> [1]: aquasecurity/trivy-action#389 [2]: aquasecurity/trivy#7951 [3]: aquasecurity/trivy-action@18f2510ee396

danjeffery · 2024-11-25T20:57:42Z

We've still seen lots of timeouts using v0.57.1. Since we rebuild our image every week, I've shifted to a loop going through all three repos here in the dockerfile so we can get both DBs at build time. It still attempts to get the latest vuln DB in a loop several times before running, but gives up after too many failures and relies on the one it has in the image if it keeps getting rate-limited.

Trivy is a great open source project providing awesome value to the community. Have you considered something like Fast Forward?

ktzsolt · 2024-11-26T09:52:47Z

@danjeffery are you sure you don't override the default TRIVY_DB_REPOSITORY and TRIVY_JAVA_DB_REPOSITORY env vars in your dockerfile? We don't have any issue since 0.57.1 using the default values for repositories that point to mirror.gcr.io (not ghcr but gcr!), what url do you see in the logs regarding the timeout?

nvuillam · 2024-11-26T16:42:10Z

I confirm the issues from time to time with gcr.io ^^
It's not TOOMANYREQUESTS, but BLOB_UNKNOWN issues with gcr.io

danjeffery · 2024-11-27T18:58:01Z

@danjeffery are you sure you don't override the default TRIVY_DB_REPOSITORY and TRIVY_JAVA_DB_REPOSITORY env vars in your dockerfile? We don't have any issue since 0.57.1 using the default values for repositories that point to mirror.gcr.io (not ghcr but gcr!), what url do you see in the logs regarding the timeout?

I was not packaging the DB in the dockerfile (keep it minimal and all) until the rate limiting issues started cropping up. Now I override deliberately in order to loop through all three endpoints and make sure I've got the db in the dockerfile instead of waiting until runtime. Then I never pull the java DB at runtime and run a limited sequence of loops to try and get a fresh vuln DB at each runtime.

From gcr we get:
Fatal error#011init error: DB error: failed to download vulnerability DB: OCI artifact error: failed to download vulnerability DB: failed to download artifact from mirror.gcr.io/aquasec/trivy-db:2: OCI repository error: 1 error occurred:
From ghcr we get:
[vulndb] Failed to download artifact#011repo="ghcr.io/aquasecurity/trivy-db:2" err="OCI repository error: 1 error occurred:\n\t* GET https://ghcr.io/v2/aquasecurity/trivy-db/manifests/2: TOOMANYREQUESTS: retry-after: 83.029µs, allowed: 44000/minute\n\n"

Both are still occurring intermittently, but with the retry loop and hitting all three endpoints we get the db. I'm probably going to automate pulling the dbs into a private repo and then just point there at runtime.

Regarding my suggestion about Fast Forward, I spoke to some of the folks behind it and they thought it could be a good fit. Putting this behind Fastly, under their open source outreach, might be fantastic and the process to apply is pretty painless.

simar7 · 2024-12-02T22:05:36Z

@danjeffery based on your suggestion, I have submitted Trivy as part of the fast forward program and I'm looking forwarding to hearing back on it. If you know the folks behind it, would you mind bringing it up again? Thank you kindly.

Trivy DBの取得に失敗する問題の対策 aquasecurity/trivy-action#389

adambirds · 2024-12-08T15:49:48Z

Thanks for fixing this @simar7, since downloading the new version and applying that to my repositories I have had no further issues yet and all my CI is now passing great.

baksetercx added a commit to 3lvia/core-github-actions-templates that referenced this issue Sep 19, 2024

Skip updating Trivy DB because of 429 issue

0c439a0

aquasecurity/trivy-action#389

baksetercx mentioned this issue Sep 19, 2024

Skip updating Trivy DB because of 429 issue 3lvia/core-github-actions-templates#126

Closed

baksetercx added a commit to 3lvia/core-github-actions-templates that referenced this issue Sep 19, 2024

Skip updating Trivy DB because of 429 issue

50da100

aquasecurity/trivy-action#389

baksetercx added a commit to 3lvia/core-github-actions-templates that referenced this issue Sep 19, 2024

Skip updating Trivy DB because of 429 issue

65972ee

aquasecurity/trivy-action#389

billhammond-dev mentioned this issue Sep 19, 2024

Trivy allow alt db #391

Closed

ppawlowski mentioned this issue Sep 22, 2024

ci: temporarily disable container scan with Trivy FlowFuse/helm#468

Merged

10 tasks

tomuben mentioned this issue Sep 23, 2024

Implement a caching mechanism for Trivy DB exasol/script-languages-release#961

Closed

naddeoa mentioned this issue Nov 12, 2024

There is no db-all make target aquasecurity/trivy-db#466

Closed

ravisingal mentioned this issue Nov 12, 2024

chore: use public ecr registry for downloading trivy databases hypertrace/github-actions#69

Merged

DavSanchez mentioned this issue Nov 12, 2024

ci: pull trivy DB from public AWS ECR newrelic/coreint-automation#84

Merged

TharmiganK mentioned this issue Nov 13, 2024

Fix TOOMANYREQUESTS failure in Trivy Action ballerina-platform/ballerina-library#7357

Merged

lou-lan mentioned this issue Nov 13, 2024

Fix TOOMANYREQUESTS failure in Trivy Action spidernet-io/egressgateway#1547

Merged

mjanez added a commit to mjanez/ckan-docker that referenced this issue Nov 13, 2024

Add Trivy database repositories to vulnerability scanner workflows

9448619

More info at: ballerina-platform/ballerina-library#7357 that fixes: aquasecurity/trivy-action#389

uladzimirdev mentioned this issue Nov 13, 2024

ci: fix path to the jar file metabase/metabase#49965

Merged

This was referenced Nov 14, 2024

AppArmor: Add read permission for executables kubernetes-sigs/security-profiles-operator#2554

Merged

AppArmor recorder: add readdir support kubernetes-sigs/security-profiles-operator#2555

Merged

hairmare mentioned this issue Nov 14, 2024

fix(trivy): add ecr registry as a fallback option for trivy db radiorabe/actions#98

Merged

ArtieReus mentioned this issue Nov 15, 2024

[Task](ci): Fix Rate Limiting Error in GitHub Actions for Trivy DB Download cloudoperators/juno#623

Open

ben-harvey mentioned this issue Nov 15, 2024

Fix Trivy rate limit error by pulling vulnerability DB from ECR Enterprise-CMCS/mac-fc-github-actions-runner-aws#227

Merged

This was referenced Nov 20, 2024

Fix trivy in GitHub action ghga-de/file-services-backend#72

Merged

Specify Trivy DB repository ghga-de/gh-action-ci#9

Merged

eiffel-fl mentioned this issue Nov 21, 2024

ci: Bump trivy-action to 0.29.0. inspektor-gadget/inspektor-gadget#3716

Merged

bmunguli mentioned this issue Nov 29, 2024

fix: trivy action workflow fetching database from gh registry eclipse-tractusx/sldt-bpn-discovery#174

Closed

2 tasks

WhiteNoise0000 added a commit to WhiteNoise0000/jf-test that referenced this issue Dec 5, 2024

Update trivy.yml

7957dca

Trivy DBの取得に失敗する問題の対策 aquasecurity/trivy-action#389

Ratelimiting error when downloading vulnerability db from ghcr.io #389

Ratelimiting error when downloading vulnerability db from ghcr.io #389

Comments

HenrikDK commented Sep 18, 2024

simar7 commented Sep 18, 2024

benglewis commented Sep 19, 2024

billhammond-dev commented Sep 19, 2024 • edited Loading

billhammond-dev commented Sep 19, 2024

billhammond-dev commented Sep 20, 2024 • edited Loading

nelsonleblanc-rl commented Sep 20, 2024 • edited Loading

billhammond-dev commented Sep 20, 2024

9838183063 commented Sep 22, 2024

srenatus commented Sep 23, 2024

baksetercx commented Sep 23, 2024

nnellanspdl commented Sep 23, 2024 • edited Loading

ybelMekk commented Sep 23, 2024 • edited Loading

NicholasFiorentini commented Sep 23, 2024

NicholasFiorentini commented Sep 23, 2024 • edited Loading

BRONSOLO commented Sep 23, 2024

eugentius commented Sep 23, 2024 • edited Loading

billhammond-dev commented Sep 23, 2024

nbenmoody-tesouro commented Sep 23, 2024

nbenmoody-tesouro commented Sep 23, 2024

nnellanspdl commented Sep 23, 2024

Nava-JoshLong commented Sep 23, 2024

simar7 commented Sep 24, 2024

jpalomaki commented Sep 24, 2024 • edited Loading

nvuillam commented Nov 11, 2024 • edited Loading

Darwiner commented Nov 11, 2024

nvuillam commented Nov 11, 2024

nnellanspdl commented Nov 18, 2024 • edited Loading

askpatrickw commented Nov 19, 2024

mythi commented Nov 20, 2024

danjeffery commented Nov 25, 2024

ktzsolt commented Nov 26, 2024

nvuillam commented Nov 26, 2024

danjeffery commented Nov 27, 2024

simar7 commented Dec 2, 2024

adambirds commented Dec 8, 2024

billhammond-dev commented Sep 19, 2024 •

edited

Loading

billhammond-dev commented Sep 20, 2024 •

edited

Loading

nelsonleblanc-rl commented Sep 20, 2024 •

edited

Loading

nnellanspdl commented Sep 23, 2024 •

edited

Loading

ybelMekk commented Sep 23, 2024 •

edited

Loading

NicholasFiorentini commented Sep 23, 2024 •

edited

Loading

eugentius commented Sep 23, 2024 •

edited

Loading

jpalomaki commented Sep 24, 2024 •

edited

Loading

nvuillam commented Nov 11, 2024 •

edited

Loading

nnellanspdl commented Nov 18, 2024 •

edited

Loading