Setting .Values.nameOverride makes the pod not have rights to update secret cert-manager-approver-policy-tls

[smuda]

Description

When setting helm parameter .Values.nameOverride to anything else than it's default value cert-manager-approver-policy the approver fails to generate it's tls certificate during startup.

The role allows access to one secret with a specific name which (when .Values.nameOverride is set to smuda) would be smuda-tls. However, in pkg/internal/webhook/tls/tls.go the name of the secret seems hard coded to cert-manager-approver-policy-tls.

To reproduce:

helm repo add jetstack https://charts.jetstack.io 
helm install cert-manager-approver jetstack/cert-manager-approver-policy --set nameOverride=smuda

Expected result

That the approver pod would startup and respond happily to the readiness-probe.

Result

The approver pod looks for and tries to update secret cert-manager-approver-policy-tls while the role allows smuda-tls. The pod is unhappy.

I0303 17:47:18.371313       1 webhook.go:67] webhook "msg"="running tls bootstrap process..." 
W0303 17:47:18.373066       1 reflector.go:424] pkg/mod/k8s.io/client-go@v0.26.1/tools/cache/reflector.go:169: failed to list *v1.Secret: secrets "cert-manager-approver-policy-tls" is forbidden: User "system:serviceaccount:addon-cert-manager:smuda" cannot list resource "secrets" in API group "" in the namespace "addon-cert-manager"
E0303 17:47:18.373122       1 reflector.go:140] pkg/mod/k8s.io/client-go@v0.26.1/tools/cache/reflector.go:169: Failed to watch *v1.Secret: failed to list *v1.Secret: secrets "cert-manager-approver-policy-tls" is forbidden: User "system:serviceaccount:addon-cert-manager:smuda" cannot list resource "secrets" in API group "" in the namespace "addon-cert-manager"
E0303 17:47:19.378513       1 tls.go:130] webhook/tls "msg"="failed to generate initial serving certificate, retrying..." "error"="failed verifying CA keypair: tls: failed to find any PEM data in certificate input" "interval"="1s"
W0303 17:47:19.595334       1 reflector.go:424] pkg/mod/k8s.io/client-go@v0.26.1/tools/cache/reflector.go:169: failed to list *v1.Secret: secrets "cert-manager-approver-policy-tls" is forbidden: User "system:serviceaccount:addon-cert-manager:smuda" cannot list resource "secrets" in API group "" in the namespace "addon-cert-manager"
E0303 17:47:19.595408       1 reflector.go:140] pkg/mod/k8s.io/client-go@v0.26.1/tools/cache/reflector.go:169: Failed to watch *v1.Secret: failed to list *v1.Secret: secrets "cert-manager-approver-policy-tls" is forbidden: User "system:serviceaccount:addon-cert-manager:smuda" cannot list resource "secrets" in API group "" in the namespace "addon-cert-manager"
E0303 17:47:20.372423       1 tls.go:130] webhook/tls "msg"="failed to generate initial serving certificate, retrying..." "error"="failed verifying CA keypair: tls: failed to find any PEM data in certificate input" "interval"="1s"
E0303 17:47:21.373552       1 tls.go:130] webhook/tls "msg"="failed to generate initial serving certificate, retrying..." "error"="failed verifying CA keypair: tls: failed to find any PEM data in certificate input" "interval"="1s"
E0303 17:47:22.372740       1 tls.go:130] webhook/tls "msg"="failed to generate initial serving certificate, retrying..." "error"="failed verifying CA keypair: tls: failed to find any PEM data in certificate input" "interval"="1s"
W0303 17:47:22.726563       1 reflector.go:424] pkg/mod/k8s.io/client-go@v0.26.1/tools/cache/reflector.go:169: failed to list *v1.Secret: secrets "cert-manager-approver-policy-tls" is forbidden: User "system:serviceaccount:addon-cert-manager:smuda" cannot list resource "secrets" in API group "" in the namespace "addon-cert-manager"
E0303 17:47:22.726622       1 reflector.go:140] pkg/mod/k8s.io/client-go@v0.26.1/tools/cache/reflector.go:169: Failed to watch *v1.Secret: failed to list *v1.Secret: secrets "cert-manager-approver-policy-tls" is forbidden: User "system:serviceaccount:addon-cert-manager:smuda" cannot list resource "secrets" in API group "" in the namespace "addon-cert-manager"
E0303 17:47:23.373272       1 tls.go:130] webhook/tls "msg"="failed to generate initial serving certificate, retrying..." "error"="failed verifying CA keypair: tls: failed to find any PEM data in certificate input" "interval"="1s"
E0303 17:47:24.372112       1 tls.go:130] webhook/tls "msg"="failed to generate initial serving certificate, retrying..." "error"="failed verifying CA keypair: tls: failed to find any PEM data in certificate input" "interval"="1s"
E0303 17:47:25.373125       1 tls.go:130] webhook/tls "msg"="failed to generate initial serving certificate, retrying..." "error"="failed verifying CA keypair: tls: failed to find any PEM data in certificate input" "interval"="1s"
E0303 17:47:26.372917       1 tls.go:130] webhook/tls "msg"="failed to generate initial serving certificate, retrying..." "error"="failed verifying CA keypair: tls: failed to find any PEM data in certificate input" "interval"="1s"
W0303 17:47:26.407488       1 reflector.go:424] pkg/mod/k8s.io/client-go@v0.26.1/tools/cache/reflector.go:169: failed to list *v1.Secret: secrets "cert-manager-approver-policy-tls" is forbidden: User "system:serviceaccount:addon-cert-manager:smuda" cannot list resource "secrets" in API group "" in the namespace "addon-cert-manager"
E0303 17:47:26.407557       1 reflector.go:140] pkg/mod/k8s.io/client-go@v0.26.1/tools/cache/reflector.go:169: Failed to watch *v1.Secret: failed to list *v1.Secret: secrets "cert-manager-approver-policy-tls" is forbidden: User "system:serviceaccount:addon-cert-manager:smuda" cannot list resource "secrets" in API group "" in the namespace "addon-cert-manager"
E0303 17:47:27.372600       1 tls.go:130] webhook/tls "msg"="failed to generate initial serving certificate, retrying..." "error"="failed verifying CA keypair: tls: failed to find any PEM data in certificate input" "interval"="1s"
E0303 17:47:28.372665       1 tls.go:130] webhook/tls "msg"="failed to generate initial serving certificate, retrying..." "error"="failed verifying CA keypair: tls: failed to find any PEM data in certificate input" "interval"="1s"
E0303 17:47:29.372708       1 tls.go:130] webhook/tls "msg"="failed to generate initial serving certificate, retrying..." "error"="failed verifying CA keypair: tls: failed to find any PEM data in certificate input" "interval"="1s"
E0303 17:47:30.373261       1 tls.go:130] webhook/tls "msg"="failed to generate initial serving certificate, retrying..." "error"="failed verifying CA keypair: tls: failed to find any PEM data in certificate input" "interval"="1s"
E0303 17:47:31.372485       1 tls.go:130] webhook/tls "msg"="failed to generate initial serving certificate, retrying..." "error"="failed verifying CA keypair: tls: failed to find any PEM data in certificate input" "interval"="1s"
E0303 17:47:32.372828       1 tls.go:130] webhook/tls "msg"="failed to generate initial serving certificate, retrying..." "error"="failed verifying CA keypair: tls: failed to find any PEM data in certificate input" "interval"="1s"
E0303 17:47:33.372578       1 tls.go:130] webhook/tls "msg"="failed to generate initial serving certificate, retrying..." "error"="failed verifying CA keypair: tls: failed to find any PEM data in certificate input" "interval"="1s"
E0303 17:47:34.372749       1 tls.go:130] webhook/tls "msg"="failed to generate initial serving certificate, retrying..." "error"="failed verifying CA keypair: tls: failed to find any PEM data in certificate input" "interval"="1s"
E0303 17:47:35.372694       1 tls.go:130] webhook/tls "msg"="failed to generate initial serving certificate, retrying..." "error"="failed verifying CA keypair: tls: failed to find any PEM data in certificate input" "interval"="1s"
E0303 17:47:36.372690       1 tls.go:130] webhook/tls "msg"="failed to generate initial serving certificate, retrying..." "error"="failed verifying CA keypair: tls: failed to find any PEM data in certificate input" "interval"="1s"

The created role smuda:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  annotations:
    meta.helm.sh/release-name: cert-manager-approver
    meta.helm.sh/release-namespace: addon-cert-manager
  creationTimestamp: "2023-03-03T17:47:14Z"
  labels:
    app.kubernetes.io/instance: cert-manager-approver
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: smuda
    app.kubernetes.io/version: v0.6.2
    helm.sh/chart: cert-manager-approver-policy-v0.6.2
  name: smuda
  namespace: addon-cert-manager
  resourceVersion: "1654"
  uid: 4e8f5114-4353-4c53-aa0d-cc174c58fe71
rules:
- apiGroups:
  - coordination.k8s.io
  resources:
  - leases
  verbs:
  - create
- apiGroups:
  - coordination.k8s.io
  resourceNames:
  - policy.cert-manager.io
  resources:
  - leases
  verbs:
  - get
  - update
- apiGroups:
  - ""
  resourceNames:
  - smuda-tls
  resources:
  - secrets
  verbs:
  - get
  - list
  - watch
  - create
  - update

[smuda]

There is something more happening than just the resourceName in the role, because even when I add both smuda-tls and cert-manager-approver-policy-tls to the role, it won't start but seems get longer. But if I remove resourceNames totally (giving access to all secrets) it starts.

[jonathanio]

In playing around, I've just run into this issue, too. The issue appears to be here:

approver-policy/pkg/internal/cmd/cmd.go

Lines 63 to 72 in a953908

	certificateSource := &servertls.DynamicSource{
	DNSNames: []string{fmt.Sprintf("%s.%s.svc", opts.Webhook.ServiceName, opts.Webhook.CASecretNamespace)},
	Authority: &authority.DynamicAuthority{
	SecretNamespace: opts.Webhook.CASecretNamespace,
	SecretName: "cert-manager-approver-policy-tls",
	RESTConfig: opts.RestConfig,
	CADuration: opts.Webhook.CADuration,
	LeafDuration: opts.Webhook.LeafDuration,
	},
	}

The application hard-codes the name of the Secret. Adding permission to access doesn't help, as the Helm Chart didn't create the resource to access. Either the name needs to be dynamically generated based on the deployment name, or the Helm Chart needs to fix the name of the Secret being created.

I don't know which would be preferred here.

[erikgb]

Thanks for looking into this @jonathanio! This needs to be fixed! Which fix would you personally prefer?

[jonathanio]

Setting nameOverride or fullnameOverride typically means that you're running concurrent resources inside the same namespace (maybe with different sets of permissions). However, that doesn't make much sense as the policies control what can and cannot be done inside a namespace with read-only access.

Also, defining it based on the deployment name feels hidden and implicit. I think it should be a configuration option within the application, which can then be overridden, as needed, by the deployment if nameOverride is set through the Helm Chart?

[erikgb]

Thanks! I will bring this issue into our stand-up on Tuesday next week - to discuss the options here. We might also take a look at cert-manager? I would assume we should choose the same approach, at least if it makes sense.

[jonathanio]

Yeah, agreed.

Thank you for looking into this.

[erikgb]

I had a closer look at this, and I believe the solution is quite obvious. So I opened #534 to fix this. PTAL @jonathanio @smuda!

[erikgb]

https://github.com/cert-manager/approver-policy/releases/tag/v0.17.0 has just been released, containing a fix for this issue. 🎉