[cosign sign] UNAUTHORIZED: You don't have the needed permissions

[rawlingsj]

Hi, just been looking at signing all Jenkins X images and would also like to make it super easy for Jenkins X users to get automated signing in their own pipelines. We're using a Tekton task + secret that contains the private key + cosign password like this...

apiVersion: v1
stringData:
  cosign.key: |-
    -----BEGIN ENCRYPTED COSIGN PRIVATE KEY-----
    abc123....
    -----END ENCRYPTED COSIGN PRIVATE KEY-----
  password: foo
kind: Secret
metadata:
  name: cosign
type: Opaque

When the cosign sign step runs we're hitting UNAUTHORIZED: You don't have the needed permissions when pushing the signature. We tend to have a mix of container registries that we push to and we use a ~/.docker/config.json for auth to these registries.

I managed to work around the error above by adding docker-credential-gcr to the cosign step image like this https://github.com/rawlingsj/jx3-pipeline-catalog/blob/de0d1aff086f90f22da15835b7e0c1498ee59b3e/tasks/cosign/cosign.yaml#L19-L21 - using this workaround we do get signing of the images, it seems to work great.

AFAIK Kaniko uses a similar approach here https://github.com/GoogleContainerTools/kaniko/blob/7b6495426d9a4713b997a1afcf197d87eecb33a3/deploy/Dockerfile_debug#L32-L45 but I wasn't sure if this is how you'd expect folks to auth with registries?

Should these credential helper CLIs be added to the cosign images or is there a better / preferred way?

[dlorenc]

Ugh, yeah there's no great way here.

cc @jonjohnsonjr. The kaniko way "works" but requires us to add every helper to the image. The other way that we've been using in Tekton is the "k8schain" approach, where you have to setup credentials in an ImagePullSecret. Have you tried that one here?

[rawlingsj]

Ok cool, I'll take a look at the "k8schain" approach. Yeah it would be good to keep all those helper CLIs out of the image.

[dlorenc]

This is something I'd love to fix "somewhere" by making a "workload identity" cred helper that just magically works with whatever standard workload identity mechanisms exist across all the cloud providers, but I have no idea how the other ones work :(

[jonjohnsonjr]

Somewhat relevant thread here: knative/serving#10259 (comment)

If it's possible to build a general solution on top of KEP 541, that would be ideal. Otherwise, I think the best we can do is something like k8schain without any of the magic -- just imagePullSecrets.

[dlorenc]

Any luck here @rawlingsj?

[rawlingsj]

I had a quick look but wasn't 100% sure on the ImagePullSecrets approach.

So I'd need to use a secret containing the docker config json and then add that via a ImagePullSecret on the build pod? We already have a kubernetes.io/dockerconfigjson secret attached to the Tekton kubernetes service account which gets automatically mounted into the build pod (which didn't work unless using the credential helper binaries) so wasn't sure if the ImagePullSecret is a different approach?

As a complete hack for now so I could see things working end to end I put this together to test things out https://github.com/jenkins-x/cosign-image - this includes those credential helpers.

I should have time this afternoon to take a proper look at the image pull secrets approach, I'll jump on your slack later today as that might be easier to chat on too.

[dlorenc]

Did you manage to figure this out @rawlingsj ?

[dlorenc]

Closing this one out - reopen if you can repro!