Signing built images

[imjasonh]

Users can sign images produced with ko publish using tools like cosign.

For example:

$ cosign sign -key cosign.key $(ko publish ./)

ko resolve produces potentially many images, which makes this a bit harder. You could ko resolve then scan the resulting YAML for ko-built image references and sign all of those with some bash magic, but 🤮 .

Would it be useful to have a ko resolve --sign cosign.key flag that used the provided key to sign all images built during ko resolve?

ko publish --sign cosign.key could also be a convenience alias for, effectively, cosign sign $(ko publish), which wouldn't require users to have cosign installed.

@dlorenc good idea? bad idea?

[dlorenc]

@dlorenc good idea? bad idea?

Great idea!

[jonjohnsonjr]

This is where I really want a sidechannel for build outputs (my stupid "layoutput" thing).

[mattmoor]

+1 to native integration here.

Generally my preference would be to bias towards the ephemeral key route and use the same ephemeral key to sign all of the images produced (we should probably also start to think about incorporating a variety of opinionated attributes based on things we know about the build).

re: many images

something like mink produced ~25 images at its peak, for ~5 architectures, so 1 key vs ~150 keys (25*5 + 25) is probably a better way to go 🤣

[mattmoor]

This PR includes a technique I have been using to incorporate keyless signing into some ko-like tools: sigstore/cosign#647

Basically it only kicks in when COSIGN_EXPERIMENTAL=true for now, which is similar to how we've gated certain features with GGCR_EXPERIMENTAL_*, and this feels like an extremely lightweight way to integrate things.

We sacrifice some amount of configurability, but my inclination here would be to probably adjust how cosign itself works to use envconfig as a way to support a standard scheme for env-var fallback for configuration (no cobra, so no viper).

[imjasonh]

+1 to KO_EXPERIMENTAL_SOMETHING to iterate on this. How do we imagine this looking when it graduates from an experiment? We don't have a good example of doing that in GGCR as far as I know.

Ideally we'd end in a state where things are signed by default using ambient tokens if available, which can be disabled with a flag or config. Do we want to support signing with explicit keys? Or just leave that for cosign?

[mattmoor]

I think the first order of business will be to get the distroless images signed against Fulcio.

For configuring things, I think trying to standardize on env vars in cosign upstream would be my preference (for portability across tools).

For opting out (beyond experiment), I think some .ko.yaml syntax is called for, we can introduce this to ease folks into it too.

Another step between “experiment guarded” and “opt-out” to encourage folks to start signing will be to always verify, but make verification non-fatal (print warnings).

I also think we probably want some sort of refresh token flow for users before we move beyond experimental.

sorry this ended up sort of stream of consciousness.

[mattmoor]

Ok, full keyboard so here's sort of what I'm thinking in terms of phasing this in.

1. No change unless TBD *EXPERIMENT* env var is set, at which point base images must be signed, and published images are signed.

(once the default base images are signed against Fulcio)

2. Start verifying base images all the time, but verification is only fatal when TBD *EXPERIMENT* flag is set. When it isn't set we emit a warning. We still only sign things when it is set. We will need to sort out a way to opt-out of verification for each base image (likely in .ko.yaml?).

(once keyless signing is no longer "experimental" in Sigstore)

3. Start signing output images whenever ambient OIDC auth is available. Provide a way to require this (e.g. release envs), or opt-out entirely (e.g. dev environments avoiding 3LOs).

(once we have a clear opt-out path for base images)

4. Start to make verification of base images fatal by default.

---

As with --platform=all, we should never require signing by default (e.g. this would slow dev loops with 3LOs, and break using ko in PRs which won't have OIDC).

If we find out that being more aggressive signing things in dev is important (e.g. back-pressure from cosigned rejecting things), then we can always revisit this, but hopefully by then we have some sort of OIDC refresh token flow.

[mattmoor]

Starting to poke through code, I think the most natural integration point for signing is likely to implement a new publisher.Interface that composes with another publisher similar to the caching version:

	// Wrap publisher in a memoizing publisher implementation.
	return publish.NewCaching(innerPublisher)
}

If we have a signing publisher, we can accumulate / pass through images to an inner publisher, and then sign all of the accumulated images in Close() before closing the inner publisher. I think this would mean we use a single key to sign the entire set of images we publish 🤞

---

As I'm writing this I realize that cosign likely doesn't have functions that take a v1.Image (really build.Result) and return the v1.Image containing the signature, so we can't leverage the inner publisher to write the signature, which means the only publisher this should compose with right now is the default publisher. It would be cool to write such a function in cosign because it would allow us to emit signed images in tarballs and OCI layouts as well.

[mattmoor]

So it looks like currently:

return cli.Sign().Exec(context.Background(), n.digests.List())

... generates a new cert for every image, and send the user through a separate 3LO for each one, so on big projects (e.g. knative.dev/serving) this is sort of prohibitive.

cc @dlorenc

[imjasonh]

Have we thought through what new CLI flags we'd need?

Regardless of how cosign-as-an-sdk evolves, cosign sign has quite a lot of flags -- are we going to need all of these in every ko surface?

ko resolve -f config/ \
    --fulcio-url https://staging.fulcio.not-sigstore.dev \
    -a foo=bar \
    --oidc-issuer=some.other-issuer.example \
    ...

Maybe we should wait for the [EXPERIMENTAL] stuff to become un-experimental -- sad as it would be to have to wait for the nice things -- before adding all those cosign flags to ko?

[dlorenc]

Maybe we should wait for the [EXPERIMENTAL] stuff to become un-experimental -- sad as it would be to have to wait for the nice things -- before adding all those cosign flags to ko?

I don't really think experimental will change much in the flag surface either way.

[imjasonh]

Picking this back up, since I think we're a lot closer to having this be possible than we were six months ago.

The concerns I think we all have are roughly threefold:

1. scary [EXPERIMENTAL] warnings around keyless and Rekor -- this is going to go away soonish, when Sigstore goes GA.
2. dependency graph explosion: cosign depends on a bunch of stuff we don't need. This is a known issue we're now trying to tackle better upstream.
3. CLI surface explosion: cosign sign takes a bunch of flags, which we don't necessarily want to just blindly copy into ko build et al.

To solve (3) I'd like to propose we start by only having ko sign things if it detects it can do so keylessly, using ambient OIDC credentials -- e.g., running in a GitHub Actions workflow with idtoken:write pushing to GHCR, or on GCB pushing to GCR/AR. Since this assumes an unattended flow, it will also only upload to Rekor if it detects the repo is public, and not ask for confirmation.

This eliminates the need for flags to configure a key, including references to keys in KMS systems, and security key flows. Instead of having flags for --fulcio-url, --oidc-issuer, etc. flags, these will come from standard environment variables, which we expect only to be used when building against your own Sigstore infrastructure.

Signing will be disabled by default, behind a new flag, --sign/-s.

If we're happy with the added dependencies required to get this working, we're pretty close to having the user-signing OIDC flow too, using sigstore/sigstore's pkg/oauthflow. This will will pop up a browser to go through OAuth before keylessly signing and uploading to Rekor -- if the image repo is private, we'll ask for confirmation before uploading to Rekor.

btw, after shelving #433 for adding ~400kloc of dependencies, we had about half that many lines sneak in through #718 anyway, so I think the dependency ship may have sailed ⛵. Not that we shouldn't keep working to get cosign's dependencies down, but maybe it's worth just biting the bullet and getting signing in before cutting out those deps.

[imjasonh]

Once we have image signing in place we should also have ko sign the SBOMs it produces, or attach them as signed attestations instead of unsigned SBOMs.

[imjasonh]

Some more considerations I think we'll need to...consider:

• We probably don't want to pop up an OAuth flow for each image we build and sign, despite wanting to keylessly sign each image. We can cache the certificate, but it's only valid for 10 minutes, and the time between the first image and the last image in a ko resolve may be longer than 10 minutes.
• Lots of times, a ko resolve on N images won't actually produce N new images; should we sign them all each time they're a part of a ko resolve, even if there's no new data being signed? That seems potentially spammy. We can try deduping? Maybe? Latest signature wins and overwrites the oldest?
• We shouldn't publish to Rekor until after the image is successfully pushed (in case we don't have permission to push to that registry), so we'll need to keylessly sign, hold on to that cert(s), then push the image and signatures, then publish to Rekor for all the images we built+signed+pushed, using the correct cert(s) for each image we pushed. Should we publish in a batch at the end of ko resolve? Probably not, since Fulcio's certs may be expired by the time we finish resolveing. If we publish to Rekor ASAP during ko resolve, we'll probably want to have some summary of what we did printed at the end, or else all the useful Rekor verification info is interleaved in the (already pretty noisy) build logs.

[mattmoor]

I figured we'd have something like git commit -s to trigger signing. We could also use .ko.yaml to default -s on like I do with gitsign for example.

[imjasonh]

I figured we'd have something like git commit -s to trigger signing. We could also use .ko.yaml to default -s on like I do with gitsign for example.

Is that effectively "ko shells out to cosign" then? If someone configures a cosign sign that doesn't do anything, ko would be oblivious. And we'd probably want to make sure the ko image contains cosign. Bleh.

gitsign has to deal with a lot of these issues too, fwiw, to prevent OAuth popup spam while always providing a valid unexpired cert (sigstore/gitsign#75). I'd kind of rather write that in Go, even if it's painful.

gitsign has to live in Git's constraints, ko is much more free to do better.

[mattmoor]

I think that with -s we'd also want to encode the SBOMs as attestations, so I suspect that'd be really cumbersome via shelling out (certainly no way to avoid multiple 3LOs).

[imjasonh]

I think that with -s we'd also want to encode the SBOMs as attestations, so I suspect that'd be really cumbersome via shelling out (certainly no way to avoid multiple 3LOs).

Yeah. I think we're talking about the same thing. -s should sign all the images it builds, and attach SBOMs as signed attestations in addition to (instead of?) unsigned SBOMs.

The problems in #357 (comment) roughly concern how to avoid multiple unnecessary 3LOs, since that will really ruin the experience of using ko anywhere outside of CI.

gitsign currently avoids it by running a daemon that it connects to over a socket to get cached certs, but that's because how gitsign is invoked is up to Git and not gitsign -- it's invoked once for every commit that needs to be signed and Rekor-published, and we're invoked once to potentially build N images, some of which may not be new and worthy of signing or Rekording.

[imjasonh]

Some more considerations I think we'll need to...consider:

• We probably don't want to pop up an OAuth flow for each image we build and sign, despite wanting to keylessly sign each image. We can cache the certificate, but it's only valid for 10 minutes, and the time between the first image and the last image in a ko resolve may be longer than 10 minutes.

Nope, this was dumb. We should just build all the images we're going to build (1 for ko build, N for ko resolve) and do the 3LO once and sign all those images with the resulting Fulcio certificate. If any build fails, don't sign anything.

• Lots of times, a ko resolve on N images won't actually produce N new images; should we sign them all each time they're a part of a ko resolve, even if there's no new data being signed? That seems potentially spammy. We can try deduping? Maybe? Latest signature wins and overwrites the oldest?

I still don't have a good answer to this.

• We shouldn't publish to Rekor until after the image is successfully pushed (in case we don't have permission to push to that registry), so we'll need to keylessly sign, hold on to that cert(s), then push the image and signatures, then publish to Rekor for all the images we built+signed+pushed, using the correct cert(s) for each image we pushed. Should we publish in a batch at the end of ko resolve? Probably not, since Fulcio's certs may be expired by the time we finish resolveing. If we publish to Rekor ASAP during ko resolve, we'll probably want to have some summary of what we did printed at the end, or else all the useful Rekor verification info is interleaved in the (already pretty noisy) build logs.

I think doing one signing pass after all the builds complete means this isn't an issue. It shouldn't take 10m just to push the signatures.

[imjasonh]

I still don't have a good answer to this.

I might have an okay answer to this.

ko build and ko resolve should have a --sign/-s flag, but ko apply|create|run shouldn't. The latter are intended for hot-inner-loop development where signing doesn't make sense; the former are generally for publishing stuff to other folks, where you should sign it.

[kameshsampath]

@imjasonh - may be -sign/s with --sign-key wouldn't be better to start with? We can then start adding other OIDC signing methods once the experimental flags on them are removed?

[imjasonh]

OIDC signing for cosign is no longer experimental, after the recent Sigstore GA.

I still think I'd like to focus on keyless signing in ko, since that's going to end up being the lowest-friction way to generate signed release artifacts, especially using GitHub Actions OIDC. We'll undoubtedly need to support keyful signing too, but my hope is that's rarer.

[mattmoor]

We've also been talking about plumbing sufficient environment variables into cosign to configure custom sigstore instances. The idea is to enable a pattern like minikube's env setup:

eval $(configure-sigstore-env)
cosign sign foo

Where the former emits a set of:

FOO=bar
BAZ=blah

I think we should aim to align with that model for configuring ko so that it can compose with tooling the same way cosign does.

Perhaps we should have a way to configure keyful via the env too?

[kameshsampath]

OIDC signing for cosign is no longer experimental, after the recent Sigstore GA.
I still need to use COSIGN_EXPERIMENTAL for doing OIDC

I love the keyless signing but then wondering how we can configure keyless signing w/o browser, typically to have integration with CI/automation. I see lots of --odic* options to cosign but wondering will we be able to configure github/google oAuth via those options ( if so excuse my ignorance here). I see this https://docs.sigstore.dev/cosign/openid_signing#oauth-flows but not sure that helps CI/automation cases as there will be manual intervention

Perhaps we should have a way to configure keyful via the env too?
IMHO that should be, as may CI tools set values using env vars

[kameshsampath]

I also faced an dependencies issue as detailed here sigstore/cosign#2477

[imjasonh]

I love the keyless signing but then wondering how we can configure keyless signing w/o browser, typically to have integration with CI/automation. I see lots of --odic* options to cosign but wondering will we be able to configure github/google oAuth via those options ( if so excuse my ignorance here). I see this https://docs.sigstore.dev/cosign/openid_signing#oauth-flows but not sure that helps CI/automation cases as there will be manual intervention

GitHub OIDC and Google and Amazon Workload Identity OIDC are all pretty straightforward to detect and use automatically, without flags.

The oidc flags for cosign sign for example mostly duplicate those that would be configured by the env vars Matt described.

[kameshsampath]

The oidc flags for cosign sign` for example mostly duplicate those that would be configured by the env vars Matt described.

That should be better, we should wait for it IMHO

[kameshsampath]

@imjasonh when trying the https://docs.sigstore.dev/cosign/openid_signing#oauth-flows I still see we get prompted to open browser or other kind of manual intervention. Not sure how they play with automation/CI workflow , I mean just by providing --oidc* parameters.

[imjasonh]

When you run cosign sign inside GitHub Actions with OIDC enabled (id-token: write), cosign will automatically pick up the credentials from the environment and not require a browser.

For example:

jobs:
  sign:
    runs-on: ubuntu-latest

    permissions:
      packages: write
      id-token: write
      
    steps:
      - uses: sigstore/cosign-installer@main
      - run: cosign sign ghcr.io/${{github.repository}}

This will sign the image as the GitHub Actions workload identity. You can see what that looks like in Rekor here: https://rekor.tlog.dev/?logIndex=7576978

To do this, cosign checks for the presence of a ACTIONS_ID_TOKEN_REQUEST_URL env var, which is set when the workflow is run with the id-token:write permission. If set, requests that URL with another GitHub-provided token. The response is an OIDC token that cosign can pass to Fulcio to get a short-lived signing cert, which it uses to sign the image and log to Rekor.

All of this should already be abstracted by sign.SignCmd, and ko would ideally do a similar thing using a future better-factored Go API.

[kameshsampath]

@imjasonh have you tried using GitHub OIDC outside of GH Actions similar to how we can do GCP --identity-token signing ? I checked the GH Docs but not finding any clue on how to get the id-token from GH PAT

[imjasonh]

@imjasonh have you tried using GitHub OIDC outside of GH Actions similar to how we can do GCP --identity-token signing ? I checked the GH Docs but not finding any clue on how to get the id-token from GH PAT

I don't believe this is supported. GitHub OIDC is only a workload identity, and not a user identity based on a PAT.

[RealHarshThakur]

We'll undoubtedly need to support keyful signing too, but my hope is that's rarer.

I might be in this situation. We have an internal build system that runs on Kubernetes and connects to Buildkit directly where we can "reliably" inject a private key for a build Job. I think supporting key-based signing would be a good start for Ko too and I'd like to help contribute if it's on the roadmap as this might be the fastest way for us to start signing images.

[evankanderson]

It looks like this issue has gone somewhat fallow... the last comments I see are in #603 (comment), which mention the need to reduce sigstore's dependency tree. Is that still the blocker, or is it implementation of Matt's environment setup in cosign in #357 (comment) ?

[mattmoor]

Yeah, I think that cosign needs to go on a major diet.

We've been achieving signing of ko (and apko) build images and attesting SBOMs and more with a set of terraform providers, which skirt this (direct) dependency.

[imjasonh]

@jonjohnsonjr has done a fair bit of hacking and cutting on cosign's code to use it in tf-cosign, with an internal fork called secant: https://github.com/chainguard-dev/terraform-provider-cosign/tree/main/internal/secant

I could see this being a good avenue for further exploration and integration into ko. If it works, I think we should at least un-internalize it, and maybe even split it out into its own repo, possibly upstream into sigstore. I know we've wanted a slimmer cosign-as-a-library for (checks calendar) about two and a half years now. Maybe this is it?

[evankanderson]

See my most recent comments in #603 (comment) -- it looks like the sigstore-go library (new as of April of this year) should be able to provide a lower-dependency avenue for implementing image signing.