Sign docker images using github attestations #8685

mjpieters · 2024-10-29T20:41:46Z

GitHub attestation signing uses the GitHub action ID token to retrieve an ephemeral code signing certificate from Fulcio, and store the signature in the Rekor transparency log.

Once an image has been successfully signed, you should be able to verify the signature with:

gh attestation verify --owner astral-uv oci://ghcr.io/astral-sh/uv:latest

Closes #8670

mjpieters · 2024-10-29T20:44:34Z

A word of warning: I am not 100% certain I fully understand everything that the build-docker.yml pipeline does. I'd appreciate some feedback on how the digest of the final docker-republish image manifest is shared with the signing step, for example.

zanieb · 2024-10-29T23:20:51Z

cc @samypr100 (as always, no obligation)

.github/workflows/build-docker.yml

samypr100 · 2024-10-30T12:51:05Z

Note, I'm running a release job on my fork with these changes (9b162d) as I'd like to see it working E2E

https://github.com/samypr100/uv/actions/runs/11593512252

samypr100 · 2024-10-30T13:09:24Z

@mjpieters

Note, I'm running a release job on my fork with these changes (9b162d) as I'd like to see it working E2E

https://github.com/samypr100/uv/actions/runs/11593512252

@mjpieters signed images are here https://github.com/samypr100/uv/pkgs/container/uv in case you'd like to verify the signatures are working as expected

mjpieters · 2024-10-30T13:55:54Z

@mjpieters

Note, I'm running a release job on my fork with these changes (9b162d) as I'd like to see it working E2E
https://github.com/samypr100/uv/actions/runs/11593512252

@mjpieters signed images are here https://github.com/samypr100/uv/pkgs/container/uv in case you'd like to verify the signatures are working as expected

Excellent! I was working my way through understanding cargo dist in my own fork to try this out but hadn't gotten as far as actually seeing the workflow complete.

The first thing I notice is that the the .sig files are now obscuring the actual docker images on that page, which is a bummer.

The signatures are valid however!

% cosign verify ghcr.io/samypr100/uv:latest --certificate-identity-regexp='.*' --certificate-oidc-issuer-regexp='.*' --output-file /tmp/samypr100-uv-verify.json

Verification for ghcr.io/samypr100/uv:latest --
The following checks were performed on each of these signatures:
  - The cosign claims were validated
  - Existence of the claims in the transparency log was verified offline
  - The code-signing certificate was verified using trusted certificate authority certificates

% jq -r '.[].optional.Subject' /tmp/samypr100-uv-verify.json
https://github.com/samypr100/uv/.github/workflows/build-docker.yml@refs/heads/docker-test
https://github.com/samypr100/uv/.github/workflows/build-docker.yml@refs/heads/docker-test
https://github.com/samypr100/uv/.github/workflows/build-docker.yml@refs/heads/docker-test

I'll have to see what can be done about those signature files in the registry.

mjpieters · 2024-10-30T15:42:25Z

So cosign pushes the signature as a new 'container' to the registry, tagged with sha256-$DIGEST.sig, with the signature contained in the manifest (see https://hackmd.io/@maelvls/how-cosign-uses-registries-to-store-signatures for an overview). It's these tags that are shown at the top of the page by ghcr.io, because it only ever shows the latest tags that have been pushed. The GH packages API and UI (which include the ghcr container registry) is very very sparse and doesn't offer any control over this.

I've looked at a number of other projects that use cosign to sign their containers (of which there are a very large number), and all the ones I looked at so far show these tags at the top. E.g. the Dependabot update bundler (a Github project!) has signature tags at the top. You see this on other registries too, e.g. the boxyhq jackson container on the docker hub.

I see but a few options here:

Accept that the signature tags are going to be pushed last and end up on top.
Experiment with the Github packages API to delete and then restore specific tags, to see if that puts them back at the top again.
Tag the containers after signing. Build and push the images, store the digests, sign the digests, then handle the version and 'latest' tags afterwards.
Tell cosign to push signatures to a separate registry. This would complicate verification of the images, I don't quite know how that'd work.
Decline this PR and forgo the security of having signed containers (which would be a huge pity, security is important, especially in supply-chain tools like uv).

zanieb · 2024-10-30T15:53:12Z

We're already pushing the latest tag separately for this reason, can we push the signature before that final image tag?

mjpieters · 2024-10-30T16:30:17Z

We're already pushing the latest tag separately for this reason, can we push the signature before that final image tag?

You push not only a tag, but a manifest as well. You'd have to separate the two steps; push the manifest without tags and then tag the digest of the manifest afterwards. It could be that you can retag the same digest; I'll try this out in my own fork registry.

samypr100 · 2024-10-30T17:25:19Z

We're already pushing the latest tag separately for this reason, can we push the signature before that final image tag?

You push not only a tag, but a manifest as well. You'd have to separate the two steps; push the manifest without tags and then tag the digest of the manifest afterwards. It could be that you can retag the same digest; I'll try this out in my own fork registry.

I think the main issue here is how GH ranks packages and gives no control over the ordering to orgs/users of what's displayed, so we have to rely on these ordering hacks related to publishing.

mjpieters · 2024-10-30T17:30:43Z

I think the main issue here is how GH ranks packages and gives no control over the ordering to orgs/users of what's displayed, so we have to rely on these ordering hacks related to publishing.

and from my experiments it is clear that it is order that the manifest were pushed to the registry that is used to list them. I tried deleting then restoring, and retagging (moving tags to another manifest then back again to the right manifest), and nothing shifts the signature from the top.

So the only thing that would work is to build and not push, sign (and let cosign push the signature manifest), and only then push the actual image manifest.

mjpieters · 2025-01-30T17:08:31Z

.github/workflows/build-docker.yml

+        # PRs from forks don't have access to secrets, disable this step in that case.
+        if: ${{ github.event.pull_request.head.repo.full_name == 'astral-sh/uv' }}


I added this in here so this workflow would at least build the docker image in this PR. The jobs otherwise would just fail with a Password required error.

Note, I think there's 3 other places in this file missing this conditional (if we're in favor of adding it)

I didn't think there was any point as those 3 all are part of jobs that only run on release.

zanieb

Thank you! I might wait for @samypr100 to give it a look too.

.github/workflows/build-docker.yml

samypr100 · 2025-01-31T04:09:20Z

.github/workflows/build-docker.yml

+        # PRs from forks don't have access to secrets, disable this step in that case.
+        if: ${{ github.event.pull_request.head.repo.full_name == 'astral-sh/uv' }}


Note, I think there's 3 other places in this file missing this conditional (if we're in favor of adding it)

samypr100 · 2025-01-31T04:11:02Z

Good news is that this approach does keep the tags working as-is.

I tried it on a test run here https://github.com/samypr100/uv/actions/runs/13066181068

Which did keep the package versions tags intact https://github.com/samypr100/uv/pkgs/container/uv

Verifications also work!

$ gh attestation verify --owner samypr100 oci://ghcr.io/samypr100/uv:latest                          [23:14:17]
Loaded digest sha256:ebd769519134c730eb1af61ddd06a8f5629f5082d0c2d8f02a6d577fe4b2a9dc for oci://ghcr.io/samypr100/uv:latest
Loaded 1 attestation from GitHub API
✓ Verification succeeded!

sha256:ebd769519134c730eb1af61ddd06a8f5629f5082d0c2d8f02a6d577fe4b2a9dc was attested by:
REPO          PREDICATE_TYPE                  WORKFLOW
samypr100/uv  https://slsa.dev/provenance/v1  .github/workflows/build-docker.yml@refs/heads/cosign-docker-images

$ gh attestation verify --owner samypr100 oci://ghcr.io/samypr100/uv:0.5.26                          [23:14:23]
Loaded digest sha256:ebd769519134c730eb1af61ddd06a8f5629f5082d0c2d8f02a6d577fe4b2a9dc for oci://ghcr.io/samypr100/uv:0.5.26
Loaded 1 attestation from GitHub API
✓ Verification succeeded!

sha256:ebd769519134c730eb1af61ddd06a8f5629f5082d0c2d8f02a6d577fe4b2a9dc was attested by:
REPO          PREDICATE_TYPE                  WORKFLOW
samypr100/uv  https://slsa.dev/provenance/v1  .github/workflows/build-docker.yml@refs/heads/cosign-docker-images

$ gh attestation verify --owner samypr100 oci://ghcr.io/samypr100/uv:0.5                             [23:14:27]
Loaded digest sha256:ebd769519134c730eb1af61ddd06a8f5629f5082d0c2d8f02a6d577fe4b2a9dc for oci://ghcr.io/samypr100/uv:0.5
Loaded 1 attestation from GitHub API
✓ Verification succeeded!

sha256:ebd769519134c730eb1af61ddd06a8f5629f5082d0c2d8f02a6d577fe4b2a9dc was attested by:
REPO          PREDICATE_TYPE                  WORKFLOW
samypr100/uv  https://slsa.dev/provenance/v1  .github/workflows/build-docker.yml@refs/heads/cosign-docker-images

$ gh attestation verify --owner samypr100 oci://ghcr.io/samypr100/uv:python3.13-bookworm             [23:14:41]
Loaded digest sha256:5537896c0c8bd2351350d32dea35c590997c418e954fb5b31d6fcbfd96126562 for oci://ghcr.io/samypr100/uv:python3.13-bookworm
Loaded 1 attestation from GitHub API
✓ Verification succeeded!

sha256:5537896c0c8bd2351350d32dea35c590997c418e954fb5b31d6fcbfd96126562 was attested by:
REPO          PREDICATE_TYPE                  WORKFLOW
samypr100/uv  https://slsa.dev/provenance/v1  .github/workflows/build-docker.yml@refs/heads/cosign-docker-images

...

cosign uses the GitHub action ID token to retrieve an ephemeral code signing certificate from Fulcio, and store the signature in the Rekor transparency log.

zanieb · 2025-01-31T15:01:01Z

Would you mind following with a brief guide on how to verify images in https://docs.astral.sh/uv/guides/integration/docker/ ?

mjpieters · 2025-01-31T15:54:45Z

Would you mind following with a brief guide on how to verify images in https://docs.astral.sh/uv/guides/integration/docker/ ?

I'll see what I can do!

mjpieters · 2025-01-31T19:16:01Z

@zanieb PR up at #11140

This MR contains the following updates: | Package | Update | Change | |---|---|---| | [astral-sh/uv](https://github.com/astral-sh/uv) | patch | `0.5.25` -> `0.5.27` | MR created with the help of [el-capitano/tools/renovate-bot](https://gitlab.com/el-capitano/tools/renovate-bot). **Proposed changes to behavior should be submitted there as MRs.** --- ### Release Notes <details> <summary>astral-sh/uv (astral-sh/uv)</summary> ### [`v0.5.27`](https://github.com/astral-sh/uv/blob/HEAD/CHANGELOG.md#0527) [Compare Source](astral-sh/uv@0.5.26...0.5.27) ##### Enhancements - Avoid setting permissions during tar extraction ([#11191](astral-sh/uv#11191)) - Remove warnings for missing lower bounds ([#11195](astral-sh/uv#11195)) - Update PubGrub to set-based outdated priority tracking ([#11169](astral-sh/uv#11169)) - Improve error messages for `uv pip install` with `--extra` or `--all-extras` and invalid sources ([#11193](astral-sh/uv#11193)) - Sign Docker images using GitHub attestations ([#8685](astral-sh/uv#8685)) ##### Preview features - Don't expand self-referential extras in the build backend ([#11142](astral-sh/uv#11142)) ##### Performance - Filter discovered Python executables by source before querying ([#11143](astral-sh/uv#11143)) - Optimize exclusion computation for markers ([#11158](astral-sh/uv#11158)) - Use Astral-maintained `tokio-tar` fork ([#11174](astral-sh/uv#11174)) - Remove unneeded `.clone()` ([#11127](astral-sh/uv#11127)) ##### Bug fixes - Fix relative paths in bytecode compilation ([#11177](astral-sh/uv#11177)) - Percent-decode URLs in canonical comparisons ([#11088](astral-sh/uv#11088)) - Respect concurrency limits in parallel index fetch ([#11182](astral-sh/uv#11182)) - Use wire JSON schema for conflict items ([#11196](astral-sh/uv#11196)) - Use explicit `_GLibCVersion` tuple in uv-python crate ([#11122](astral-sh/uv#11122)) ##### Documentation - Add Git SHA locking behavior to docs ([#11125](astral-sh/uv#11125)) - Add best-practice flags to `pip install` example in troubleshooting guide ([#11194](astral-sh/uv#11194)) - Set `VIRTUAL_ENV` in Jupyter kernels ([#11155](astral-sh/uv#11155)) - Add instructions for deactivating an environment ([#11200](astral-sh/uv#11200)) ### [`v0.5.26`](https://github.com/astral-sh/uv/blob/HEAD/CHANGELOG.md#0526) [Compare Source](astral-sh/uv@0.5.25...0.5.26) ##### Enhancements - Add support for `uvx python` ([#11076](astral-sh/uv#11076)) - Allow `--no-dev --invert` in `uv tree` ([#11068](astral-sh/uv#11068)) - Update `uv python install --reinstall` to reinstall all previous versions ([#11072](astral-sh/uv#11072)) - Consistently write log messages with capitalized first word ([#11111](astral-sh/uv#11111)) - Suggest `--build-backend` when `--backend` is passed to `uv init` ([#10958](astral-sh/uv#10958)) - Improve retry trace message ([#11108](astral-sh/uv#11108)) ##### Performance - Remove unnecessary UTF-8 conversion in hash parsing ([#11110](astral-sh/uv#11110)) ##### Bug fixes - Ignore non-hash fragments in HTML API responses ([#11107](astral-sh/uv#11107)) - Avoid resolving symbolic links when querying Python interpreters ([#11083](astral-sh/uv#11083)) - Avoid sharing state between universal and non-universal resolves ([#11051](astral-sh/uv#11051)) - Error when `--script` is passing a non-PEP 723 script ([#11118](astral-sh/uv#11118)) - Make metadata deserialization failures non-fatal in the cache ([#11105](astral-sh/uv#11105)) - Mark metadata as dynamic when reading from built wheel cache ([#11046](astral-sh/uv#11046)) - Propagate credentials for `<index>/simple` to `<index>/...` endpoints ([#11074](astral-sh/uv#11074)) - Fix conflicting extra bug during `uv sync` ([#11075](astral-sh/uv#11075)) ##### Documentation - Add PyTorch XPU instructions to the PyTorch guide ([#11109](astral-sh/uv#11109)) - Add docs for signal handling ([#11041](astral-sh/uv#11041)) - Explain build frontend vs. build backend ([#11094](astral-sh/uv#11094)) - Fix formatting of `RUST_LOG` documentation ([#10053](astral-sh/uv#10053)) - Fix typo in `--no-deps` description ([#11073](astral-sh/uv#11073)) - Reflow CLI documentation comments ([#11040](astral-sh/uv#11040)) - Shorten "Using existing Python versions" nav item so it fits on one line ([#11077](astral-sh/uv#11077)) - Some minor touch-ups to the Python install guide ([#11116](astral-sh/uv#11116)) - Update Dependabot tracking issue link ([#11054](astral-sh/uv#11054)) - Update documentation for running in a container ([#11052](astral-sh/uv#11052)) - Upgrade PyTorch version in documentation ([#11114](astral-sh/uv#11114)) - Use `sys_platform` in lieu of `platform_system` in PyTorch docs ([#11113](astral-sh/uv#11113)) - Use positive (rather than negative) markers in PyTorch examples ([#11112](astral-sh/uv#11112)) - Fix unnecessary backslashes in brackets ([#11059](astral-sh/uv#11059)) - Suggest setting copy link mode in GitLab integration guide ([#11067](astral-sh/uv#11067)) </details> --- ### Configuration 📅 **Schedule**: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever MR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this MR and you won't be reminded about this update again. --- - [ ] If you want to rebase/retry this MR, check this box --- This MR has been generated by [Renovate Bot](https://github.com/renovatebot/renovate).

@zanieb

As [requested by @zanieb](#8685 (comment)).

mjpieters temporarily deployed to release October 29, 2024 20:41 — with GitHub Actions Inactive

mjpieters force-pushed the cosign-docker-images branch from fd48d69 to 5abc4eb Compare October 29, 2024 20:43

mjpieters temporarily deployed to release October 29, 2024 20:43 — with GitHub Actions Inactive

zanieb reviewed Oct 29, 2024

View reviewed changes

.github/workflows/build-docker.yml Outdated Show resolved Hide resolved

zanieb reviewed Oct 29, 2024

View reviewed changes

.github/workflows/build-docker.yml Outdated Show resolved Hide resolved

samypr100 reviewed Oct 30, 2024

View reviewed changes

.github/workflows/build-docker.yml Show resolved Hide resolved

mjpieters force-pushed the cosign-docker-images branch from 5abc4eb to d1a3809 Compare October 30, 2024 10:19

mjpieters temporarily deployed to release October 30, 2024 10:19 — with GitHub Actions Inactive

mjpieters force-pushed the cosign-docker-images branch from d1a3809 to 2872b9a Compare October 30, 2024 12:29

mjpieters temporarily deployed to release October 30, 2024 12:29 — with GitHub Actions Inactive

mjpieters force-pushed the cosign-docker-images branch from 2872b9a to 9b162d3 Compare October 30, 2024 12:30

mjpieters temporarily deployed to release October 30, 2024 12:31 — with GitHub Actions Inactive

mjpieters force-pushed the cosign-docker-images branch from 9b162d3 to 4a278c1 Compare October 30, 2024 13:09

mjpieters temporarily deployed to release October 30, 2024 13:10 — with GitHub Actions Inactive

mjpieters temporarily deployed to release January 30, 2025 17:06 — with GitHub Actions Inactive

mjpieters commented Jan 30, 2025

View reviewed changes

mjpieters force-pushed the cosign-docker-images branch from 1d03cda to 59afb8e Compare January 30, 2025 17:09

mjpieters temporarily deployed to release January 30, 2025 17:10 — with GitHub Actions Inactive

mjpieters requested review from zanieb and samypr100 January 30, 2025 17:11

zanieb approved these changes Jan 30, 2025

View reviewed changes

mjpieters mentioned this pull request Jan 30, 2025

Use docker cache mounts for apt, pip and cargo #11106

Open

samypr100 reviewed Jan 31, 2025

View reviewed changes

Sign docker images using cosign

04398a4

cosign uses the GitHub action ID token to retrieve an ephemeral code signing certificate from Fulcio, and store the signature in the Rekor transparency log.

mjpieters force-pushed the cosign-docker-images branch from 59afb8e to 04398a4 Compare January 31, 2025 10:07

mjpieters temporarily deployed to release January 31, 2025 10:07 — with GitHub Actions Inactive

mjpieters requested a review from samypr100 January 31, 2025 12:47

zanieb approved these changes Jan 31, 2025

View reviewed changes

zanieb added releases Related to building and distributing release artifacts of uv security labels Jan 31, 2025

zanieb merged commit 47f80a6 into astral-sh:main Jan 31, 2025
96 checks passed

mjpieters changed the title ~~Sign docker images using cosign~~ Sign docker images using github attestations Jan 31, 2025

mjpieters deleted the cosign-docker-images branch January 31, 2025 15:54

mjpieters mentioned this pull request Jan 31, 2025

Docs on how to verify uv docker image attestations #11140

Merged

BrewTestBot mentioned this pull request Feb 4, 2025

uv 0.5.27 Homebrew/homebrew-core#206465

Merged

zanieb pushed a commit that referenced this pull request Feb 4, 2025

Docs on how to verify uv docker image attestations (#11140)

04374b0

As [requested by @zanieb](#8685 (comment)).

samypr100 mentioned this pull request Feb 9, 2025

Add uv release artifact attestations #11357

Draft

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Sign docker images using github attestations #8685

Sign docker images using github attestations #8685

mjpieters commented Oct 29, 2024 •

edited

Loading

mjpieters commented Oct 29, 2024

zanieb commented Oct 29, 2024

samypr100 commented Oct 30, 2024 •

edited

Loading

samypr100 commented Oct 30, 2024

mjpieters commented Oct 30, 2024

mjpieters commented Oct 30, 2024 •

edited

Loading

zanieb commented Oct 30, 2024 •

edited

Loading

mjpieters commented Oct 30, 2024

samypr100 commented Oct 30, 2024

mjpieters commented Oct 30, 2024

mjpieters Jan 30, 2025 •

edited

Loading

samypr100 Jan 31, 2025

mjpieters Jan 31, 2025

zanieb left a comment

samypr100 Jan 31, 2025

samypr100 commented Jan 31, 2025 •

edited

Loading

zanieb commented Jan 31, 2025

mjpieters commented Jan 31, 2025

mjpieters commented Jan 31, 2025

		# PRs from forks don't have access to secrets, disable this step in that case.
		if: ${{ github.event.pull_request.head.repo.full_name == 'astral-sh/uv' }}

Sign docker images using github attestations #8685

Sign docker images using github attestations #8685

Conversation

mjpieters commented Oct 29, 2024 • edited Loading

mjpieters commented Oct 29, 2024

zanieb commented Oct 29, 2024

samypr100 commented Oct 30, 2024 • edited Loading

samypr100 commented Oct 30, 2024

mjpieters commented Oct 30, 2024

mjpieters commented Oct 30, 2024 • edited Loading

zanieb commented Oct 30, 2024 • edited Loading

mjpieters commented Oct 30, 2024

samypr100 commented Oct 30, 2024

mjpieters commented Oct 30, 2024

mjpieters Jan 30, 2025 • edited Loading

Choose a reason for hiding this comment

samypr100 Jan 31, 2025

Choose a reason for hiding this comment

mjpieters Jan 31, 2025

Choose a reason for hiding this comment

zanieb left a comment

Choose a reason for hiding this comment

samypr100 Jan 31, 2025

Choose a reason for hiding this comment

samypr100 commented Jan 31, 2025 • edited Loading

zanieb commented Jan 31, 2025

mjpieters commented Jan 31, 2025

mjpieters commented Jan 31, 2025

mjpieters commented Oct 29, 2024 •

edited

Loading

samypr100 commented Oct 30, 2024 •

edited

Loading

mjpieters commented Oct 30, 2024 •

edited

Loading

zanieb commented Oct 30, 2024 •

edited

Loading

mjpieters Jan 30, 2025 •

edited

Loading

samypr100 commented Jan 31, 2025 •

edited

Loading