Can't push to ghcr 403: unexpected status from HEAD request

[pavolloffay]

Contributing guidelines

• I've read the contributing guidelines and wholeheartedly agree

I've found a bug, and:

• The documentation does not mention anything about my problem
• There are no open or closed issues that are related to my problem

Description

Failed to push to ghcr

Expected behaviour

push passed

Actual behaviour

Failed to push to ghcr

Repository URL

https://github.com/open-telemetry/opentelemetry-operator

Workflow run URL

https://github.com/open-telemetry/opentelemetry-operator/actions/runs/6251580120/job/17400025264

YAML workflow

name: "Publish operator bundle"

on:
  push:
    paths:
      - '.github/workflows/publish-operator-bundle.yaml'
      - 'bundle.Dockerfile'
      - 'bundle/**'
    branches:
      - main
    tags:
      - 'v*'
  workflow_dispatch:

permissions:
  contents: read
  packages: write

jobs:
  publish:
    runs-on: ubuntu-22.04

    steps:
      - uses: actions/checkout@v4

      - name: Docker meta
        id: docker_meta
        uses: docker/metadata-action@v5
        with:
          images: |
            otel/opentelemetry-operator-bundle
            ghcr.io/open-telemetry/opentelemetry-operator/opentelemetry-operator-bundle
          tags: |
            type=semver,pattern={{version}}
            type=semver,pattern={{major}}.{{minor}}
            type=semver,pattern={{raw}}
            type=ref,event=branch


      - name: Set up QEMU
        uses: docker/setup-qemu-action@v3

      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v3

      - name: Cache Docker layers
        uses: actions/cache@v3
        with:
          path: /tmp/.buildx-cache
          key: ${{ runner.os }}-buildx-${{ github.sha }}
          restore-keys: |
            ${{ runner.os }}-buildx-

      - name: Log into Docker.io
        uses: docker/login-action@v3
        if: ${{ github.event_name == 'push' }}
        with:
          username: ${{ secrets.DOCKER_USERNAME }}
          password: ${{ secrets.DOCKER_PASSWORD }}

      - name: Login to GitHub Package Registry
        uses: docker/login-action@v3
        if: ${{ github.event_name == 'push' }}
        with:
          registry: ghcr.io
          username: ${{ github.repository_owner }}
          password: ${{ secrets.GITHUB_TOKEN }}

      - name: Build and push Operator image
        uses: docker/build-push-action@v5
        with:
          context: .
          file: ./bundle.Dockerfile
          platforms: linux/amd64,linux/arm64
          push: ${{ github.event_name != 'pull_request' }}
          tags: ${{ steps.docker_meta.outputs.tags }}
          labels: ${{ steps.docker_meta.outputs.labels }}
          cache-from: type=local,src=/tmp/.buildx-cache
          cache-to: type=local,dest=/tmp/.buildx-cache

Workflow logs

#0 building with "builder-23c37d3b-e71a-435e-88cc-e7cf7ef5c039" instance using docker-container driver

#1 [internal] load build definition from bundle.Dockerfile
#1 transferring dockerfile: 969B done
#1 DONE 0.0s

#2 [internal] load .dockerignore
#2 transferring context: 2B done
#2 DONE 0.0s

#3 importing cache manifest from local:11630741587206904603
#3 inferred cache manifest type: application/vnd.oci.image.index.v1+json done
#3 DONE 0.0s

#4 [internal] load build context
#4 transferring context: 469.27kB 0.0s done
#4 DONE 0.0s

#5 [linux/arm64 1/3] COPY bundle/manifests /manifests/
#5 DONE 0.0s

#6 [linux/arm64 2/3] COPY bundle/metadata /metadata/
#6 DONE 2.5s

#7 [linux/arm64 3/3] COPY bundle/tests/scorecard /tests/scorecard/
#7 DONE 0.0s

#8 exporting to image
#8 exporting layers 0.0s done
#8 exporting manifest sha256:1de0577b2a4e4514d2e083d31167e97d703f11a3409a04d83efba0e73364cbef 0.0s done
#8 exporting config sha256:5d109a3d046b2f9797bb6158b6a28ca1c194f197cd7568d997d88885acfcec6e
#8 exporting config sha256:5d109a3d046b2f9797bb6158b6a28ca1c194f197cd7568d997d88885acfcec6e done
#8 exporting attestation manifest sha256:705d28f160ee9b6afb87c1dfe38c8454bfe498dd097375f94863ef2ce6e16fe4 done
#8 exporting manifest sha256:089acbf011764b01baf76707c2511f583e53b25cf0f6481e763b82d8f2c2faf3 done
#8 exporting config sha256:8d12e52c4dd7312ce223ccd266597ac0d2ffd584b19d79a3ca2c260483e0be0c done
#8 exporting attestation manifest sha256:43a8b8a574ad4ddc587d2ad732d5be9bdc49babaf563eb2a542bf04913c78e29 done
#8 exporting manifest list sha256:0acc0a05194cde88424a4ad24d5a8f0c9de43e64854ba4b1cc3a8beea9ac63a6 done
#8 pushing layers
#8 ...

#9 [auth] otel/opentelemetry-operator-bundle:pull,push token for registry-1.docker.io
#9 DONE 0.0s

#8 exporting to image
#8 pushing layers 1.7s done
#8 pushing manifest for docker.io/otel/opentelemetry-operator-bundle:main@sha256:0acc0a05194cde88424a4ad24d5a8f0c9de43e64854ba4b1cc3a8beea9ac63a6
#8 ...

#10 [auth] open-telemetry/opentelemetry-operator/opentelemetry-operator-bundle:pull,push token for ghcr.io
#10 DONE 0.0s

#8 exporting to image
#8 pushing manifest for docker.io/otel/opentelemetry-operator-bundle:main@sha256:0acc0a05194cde88424a4ad24d5a8f0c9de43e64854ba4b1cc3a8beea9ac63a6 1.3s done
#8 pushing layers 0.3s done
#8 ERROR: failed to push ghcr.io/open-telemetry/opentelemetry-operator/opentelemetry-operator-bundle:main: unexpected status from HEAD request to https://ghcr.io/v2/open-telemetry/opentelemetry-operator/opentelemetry-operator-bundle/blobs/sha256:39bfe6f3a782ef49b98bb41125ff5afa4a9fe5b9ea43edc6ab6edb53fb46d457: 403 Forbidden

exporting to image:

---

ERROR: failed to solve: failed to push ghcr.io/open-telemetry/opentelemetry-operator/opentelemetry-operator-bundle:main: unexpected status from HEAD request to https://ghcr.io/v2/open-telemetry/opentelemetry-operator/opentelemetry-operator-bundle/blobs/sha256:39bfe6f3a782ef49b98bb41125ff5afa4a9fe5b9ea43edc6ab6edb53fb46d457: 403 Forbidden
Error: buildx failed with: ERROR: failed to solve: failed to push ghcr.io/open-telemetry/opentelemetry-operator/opentelemetry-operator-bundle:main: unexpected status from HEAD request to https://ghcr.io/v2/open-telemetry/opentelemetry-operator/opentelemetry-operator-bundle/blobs/sha256:39bfe6f3a782ef49b98bb41125ff5afa4a9fe5b9ea43edc6ab6edb53fb46d457: 403 Forbidden

BuildKit logs

No response

Additional info

No response

[pavolloffay]

[suyash-811]

@pavolloffay How did you solve this issue?

[pavolloffay]

open-telemetry/opentelemetry-operator#2001 (comment)

[gsakkis]

Not sure if it's the same issue but I'm getting the same error consistently when I push two tags (main and latest); the first one succeeds the second one it doesn't:

#16 pushing manifest for ghcr.io/MY_ORG/MY_REPO:main@sha256:c5e0209a9a99e75c048f733f22318bf70b10eba5c300b21cb162d64f98d3d956
#16 pushing manifest for ghcr.io/MY_ORG/MY_REPO:main@sha256:c5e0209a9a99e75c048f733f22318bf70b10eba5c300b21cb162d64f98d3d956 0.4s done
#16 pushing layers 0.1s done
#16 ERROR: failed to push ghcr.io/MY_ORG/MY_REPO:latest: unexpected status from HEAD request to https://ghcr.io/v2/MY_ORG/MY_REPO/blobs/sha256:f698428706820e407a258fef7b91e82cecfa28636a487094468a8ee12ce15b7b: 403 Forbidden
------
 > exporting to image:
------
ERROR: failed to solve: failed to push ghcr.io/MY_ORG/MY_REPO:latest: unexpected status from HEAD request to https://ghcr.io/v2/MY_ORG/MY_REPO/blobs/sha256:f698428706820e407a258fef7b91e82cecfa28636a487094468a8ee12ce15b7b: 403 Forbidden
Error: buildx failed with: ERROR: failed to solve: failed to push ghcr.io/MY_ORG/MY_REPO:latest: unexpected status from HEAD request to https://ghcr.io/v2/MY_ORG/MY_REPO/blobs/sha256:f698428706820e407a258fef7b91e82cecfa28636a487094468a8ee12ce15b7b: 403 Forbidden

When I rerun the job, the remaining tag (latest) is pushed successfully:

#16 pushing manifest for ghcr.io/MY_ORG/MY_REPO:latest@sha256:a28fccda9e904b23bd3e0cb6184173e68bd5a542f776b3a1e016daa7757aa888
#16 pushing manifest for ghcr.io/MY_ORG/MY_REPO:latest@sha256:a28fccda9e904b23bd3e0cb6184173e68bd5a542f776b3a1e016daa7757aa888 0.7s done

[burgerdev]

For anyone arriving here via search engines: I had the same problem and it turned out that the package already existed, but was somehow orphaned within the org. An administrator needed to grant permissions again. So the linked quote

I would really like to know why changing the name fixed this problem. Was the previous name tainted in some way?

is correct: the package was tainted, somewhat.