Verify local image with Cosign

[captainfalcon23]

Question
Hi,
I want to pull an image from a repo, and verify my locally pulled image in my local docker cache, NOT directly against the public repo.

Is that possible? I can't identify a CLI option to enable this?

[znewman01]

A workflow like the following should accomplish much the same:

$ IMAGE=$(docker image inspect cgr.dev/chainguard/static -f '{{index .RepoDigests 0}}')
$ cosign verify $IMAGE --certificate-identity 'https://github.com/chainguard-images/images/.github/workflows/release.yaml@refs/heads/main' --certificate-oidc-issuer 'https://token.actions.githubusercontent.com'

Part of the issue is that pulling the image causes the digest to change, so you can't really verify the signatures against the local image.

[captainfalcon23]

Reading through that link, it seems that only images which use a v1 manifest will cause the digest to change.

What if there is a scenario where someone is performing a man-in-the-middle attack? By verifying the locally downloaded image, we can be sure that the image we downloaded is exactly the same as the image in the repo, and the exact image we were intended to download.

It's not too dissimilar to having yum use a GPG key to verify a package after it has been downloaded.

[znewman01]

Reading through that link, it seems that only images which use a v1 manifest will cause the digest to change.

Lots of popular registries/images are still using v1 manifests:

$ docker pull alpine
$ docker image inspect alpine -f 'local={{.Id}} repo={{index .RepoDigests 0}}'
local=sha256:d74e625d91152966d38fe8a62c60daadb96d4b94c1a366de01fab5f334806239 repo=alpine@sha256:ff6bdca1701f3a8a67e328815ff2346b0e4067d32ec36b7992c1fdc001dc851

In general, your local Docker daemon functions quite different from a registry; most of the tricks Cosign uses to, e.g., store signatures require the full registry API. I'm personally optimistic that most of the standardization work in OCI will move us in the direction of enabling Cosign-like workflows directly on your local Docker (or Podman) daemon but that will take some time.

What if there is a scenario where someone is performing a man-in-the-middle attack? By verifying the locally downloaded image, we can be sure that the image we downloaded is exactly the same as the image in the repo, and the exact image we were intended to download.

Docker digests help with that: if you pull by digest, you get integrity checking during download. So the workflow:

1. resolve digest
2. check signature
3. pull by digest

will handle MITM attacks.

It's not too dissimilar to having yum use a GPG key to verify a package after it has been downloaded.

But isn't it better not to download the bad package in the first place? You can verify a signature over an artifact digest, and only then download the artifact. If the digest doesn't match (due to MITM or some other condition), you can reject the artifact during download.

This is how The Update Framework (TUF) handles this issue and it has a couple nice side-effects: one is that if an image pull completes successfully, we know that the signature was verified. So you don't have to figure out which local images are/are not trusted.

---

To be clear, we've left this issue open because we agree with you that there should be better workflows for local Docker images with Cosign. Don't interpret my arguing as trying to defend the status quo, just explaining why we haven't (1) prioritized this workflow (there are passable workarounds, and in a Kubernetes context, which has been a big focus thus far, a registry-centric verification flow works) and (2) why it's not a straightforward matter to implement.

Unfortunately we've got to work with the reality of how Docker is implemented, and if we recommend a workflow that doesn't work a large portion of the time that's worse than nothing IMO. We're hopeful that future work in OCI will make the situation a little better: #2516

CCing my resident OCI experts to see if they have any ideas for how to make this better: @sudo-bmitch @jdolitsky @imjasonh

[captainfalcon23]

Thanks for the detailed information @znewman01.

I will review and come back if I have any further questions :)

[github-actions]

This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 5 days.

[github-actions]

This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 5 days.

[captainfalcon23]

Commenting to ensure the stale label gets removed.

[github-actions]

This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 5 days.

[captainfalcon23]

Commenting to ensure the stale label gets removed.

[sudo-bmitch]

Lots of popular registries/images are still using v1 manifests:

$ docker pull alpine
$ docker image inspect alpine -f 'local={{.Id}} repo={{index .RepoDigests 0}}'
local=sha256:d74e625d91152966d38fe8a62c60daadb96d4b94c1a366de01fab5f334806239 repo=alpine@sha256:ff6bdca1701f3a8a67e328815ff2346b0e4067d32ec36b7992c1fdc001dc851

There might be some confusion here. Everything uploaded to Hub has been on v2 manifests for quite a long while now, a lot of registries are phasing out v1 support entirely. That's separate from the docker v2 to OCI media type transition that's been happening lately in tools like buildkit.

The image ID in docker is the digest of the config blob. It's unique since it contains the layer digests, but one level too low for cosign. The RepoDigest is the one you want. It's tracking the hash of the image on the registry, and it's a list since you may have pushed/pulled from multiple registries, and there may be a few digests for the manifest list and the separate platform specific manifest. Docker started tracking this a while back and it's reasonable for cosign to depend on it for any docker specific workflows.

$ docker pull alpine
...

$ docker image inspect alpine -f 'local={{.Id}} repo={{index .RepoDigests 0}}'
local=sha256:7e01a0d0a1dcd9e539f8e9bbd80106d59efbdf97293b3d38f5d7a34501526cdb repo=alpine@sha256:7144f7bab3d4c2648d7e59409f15ec52a18006a128c733fcff20d3a4a54ba44a

$ crane manifest alpine@sha256:7144f7bab3d4c2648d7e59409f15ec52a18006a128c733fcff20d3a4a54ba44a | jq .mediaType
"application/vnd.docker.distribution.manifest.list.v2+json"

In general, your local Docker daemon functions quite different from a registry; most of the tricks Cosign uses to, e.g., store signatures require the full registry API. I'm personally optimistic that most of the standardization work in OCI will move us in the direction of enabling Cosign-like workflows directly on your local Docker (or Podman) daemon but that will take some time.

You wouldn't be able to sign an image locally (though I'd really like to get there). But you can verify that a local image is the same as the one on the registry and verify or sign that image on the registry using the RepoDigest value in docker.