Verify base image signatures

[imjasonh]

If ko is instructed to build an image based on another image, it would be nice to verify signatures on those base images.

It just so happens the default base image, distroless, is signed. 🎉

• What should the CLI surface look like for enabling/disabling signing base images?
• Should we start by only signing (edit) verifying the default distroless base image?
• How should we get the keys to verify the signatures?
• How should users configure the keys to verify arbitrary images' signatures?

cc @dlorenc

[mattmoor]

I'd propose a similar two-release strategy to things we've done previously (e.g. requiring ko://):

1. Start to verify things by default, be noisy when they are NOT (link to how), and have an option to fail,
2. Require verification by default, fail closed, and have an option to allow unsigned images.

[imjasonh]

I don't think there are enough signed base images in the wild yet to justify failing closed. A big scary warning sgtm though.

Since we control the default and the default is signed, I do think it's safe to fail closed if we detect distroless isn't signed.

[mattmoor]

@imjasonh Yeah 2 doesn't have to be the next release, but we should indicate in our scary warning that this is the direction we intend to head. We can always wake up smarter and walk that back, but the more people we can scare into taking action the better IMO

[mattmoor]

To pontificate a bit. At least for me, ko and other "last mile" build tools exist to try to realize the best practices (e.g. nonroot, multi-arch, minimal base).

While the breaking nature of this sort of against the idea of back-compatibility, I think the north star we want to remain compatible with is that we leave folks with containers that are as good as we can possibly produce given the evolution of best practices over time.

I wonder if it would be worth including a bit about that as a sort of "mission statement" or "philosophy" for the tool in the main README.md? Sort of like we added the principles section to GGCR.

</pontification>

[mattmoor]

This applies similarly to: #357

[imjasonh]

Taking a stab at the latter two points:

.ko.yaml:

verifySignatures:
  "alpine:1.23":
  - publicKey: https://some.url/path/to/key.pub
    annotations:
      key: value
      another: value2

If any image is based on alpine:1.23 then ko will verify that it has a signature signed using the private key it can find at the URL, and has those annotations.

The default distroless base image has an assumed - publicKey: https://github.com/GoogleContainerTools/distroless/blob/main/cosign.pub as documented here.

Users can configure other signatures that need to be verified for distroless, or any base image they use.

Q: how should ephemeral OIDC keys be configured to be verified?

If all base images involved in an invocation of ko are signed and verified, no warning. If any are missing a signature, warn. If any have signatures that can't be verified as configured, fail loudly.

wdyt?

[mattmoor]

Revisiting this and adding a few thoughts after mulling this over a bit this morning.

I think that rather than inventing our own ~API for verification, we should (similar to the Go releaser precedent) align around the sigstore/policy-controller API for verification, which closes a lot of open questions (including rollout).

I think there are a few key elements of configuration:

1. Global configuration (e.g. what to do for uncovered images?)
2. Policy configuration (e.g. what to check for which images?)

For 1. I think we want a little bit of configuration that exposes the knobs that policy-controller defines within its configmap here. For example, the no-match-policy setting can warn on uncovered images (probably what we should start with as the default!), but folks can configure this to allow (match current behavior) or deny (to block things uncovered by policy).

For 2. I think it makes sense to align with the ClusterImagePolicy API shape (and TrustRoot coming soon!) rather than inventing our own thing. This allows us access to the following authorities properties:

1. Statically trust certain images with static: {action: pass},
2. Statically distrust certain images with static: {action: fail},
3. Verify signatures with fixed keys using key: {data: ... }
4. Verify signatures with KMS keys using key: {kms: gcpkms://... }
5. Verify signatures with CAs using keyless: { ... }
6. Verify signature with TSAs using rfc3161timestamp: { ... }
7. Verify transparency log inclusion with ctlog: { ... }
8. Verify attestation signatures in all of the above ways
9. Verify individual attestation predicates using Cue or Rego
10. Verify N of M-like properties with top-level Cue or Rego policies.

This is significantly more capable that what you can easily do via cosign verify directly today (benefits of API over CLI!), will only grow in capability over time, and will always exceed the capabilities of anything homegrown we try to define ourselves. 😅

Before foraying into the next section, let me ACK the dependency tree nightmare we have inherited from sigstore/cosign, which @imjasonh and others are working to fix with sigstore/sigstore-go.

sigstore/policy-controller is also intended to be used as a library, and exposes a ValidatePolicy method to allow folks to validate policies against image digests directly. It currently has something we call the policy-tester which is a toy CLI for playing with policies, and demonstrates some of how you can call this method directly with a policy today.

---

As a total strawperson proposal for this:

verification:
  no-match-policy: ...
  policies:
  - data: # inline policy
  - url: # URL for a policy
  - path: # Path to policy (support directories as well, like kubectl/ko -f)

I think we should default this to something like:

verification:
  no-match-policy: warn
  policies:
  # a default policy covering our default base image (cgr.dev/chainguard/static*)
  - data: ...
  # Users can put CIPs/Secrets/TrustRoots under .ko/ (alongside .ko.yaml)
  - path: .ko/

Folks can get the current behavior with:

verification:
  # Allow uncovered images
  no-match-policy: allow
  # Specify no policies
  policies: []