[EKS] [BUG]: CSR is not returning the certificate after approval in 1.21

[ahachete]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Description

This is a potential bug, as a) it doesn't match the service expectations; b) it doesn't happen with any previous EKS version; c) it works as expected on other, non-EKS 1.21s.

Essentially, upon submitting a Certificate Signing Request, EKS 1.21 is returning success but not including the certificate in the CSR's .status.certificate field. To reproduce please try to run:

cd /tmp
cat <<EOF | cfssl genkey - | cfssljson -bare server
{
  "hosts": [
    "label-demo",
    "label-demo.default",
    "label-demo.default.svc",
    "label-demo.default.svc.cluster.local"
  ],
  "CN": "system:node:label-demo.default.svc",
  "key": {
    "algo": "ecdsa",
    "size": 256
  },
  "names": [
    {
      "O": "system:nodes"
    }
  ]
}
EOF

cat <<EOF | kubectl apply -f -
apiVersion: certificates.k8s.io/v1
kind: CertificateSigningRequest
metadata:
  name: label-demo
spec:
  request: $(cat /tmp/server.csr | base64 | tr -d '\n')
  signerName: kubernetes.io/kubelet-serving
  usages:
  - digital signature
  - key encipherment
  - server auth
EOF

The issue was discovered and documented in detail as part of the StackGres project. Please refer to the issue there for completeness.

[cameojokes]

Same issue here. Any traction on a resolution?

[scbunn]

any updates here?

[heartTorres]

I also encounter this issue..

[Lupunita]

After a few hours of trying to figure out what's going on, I found this open issue. Could you please share any updates?

[mikestef9]

By design EKS does not issue certificates for CSRs with signerName "kubernetes.io/kubelet-serving" unless the CSR was actually requested by a kubelet. EKS's custom signer validates this by checking that the requested SANs for CSRs with signerName kubernetes.io/kubelet-serving match an actual EC2 instance's IPs/DNS names. In other words, EKS does not issue certificates for CSRs with signerName kubernetes.io/kubelet-serving posing as kubelets, it only issues certificates for CSRs with signerName kubernetes.io/kubelet-serving for actual kubelets.

If you need a certificate for a non-kubelet application, for 1.21 and below use CSR v1beta1 API and signerName legacy-unknown. For 1.22 #1397 and above, we will be documenting an alternate approach because legacy-unknown is not supported.

[raesene]

@mikestef9 does EKS support a signer name of kubernetes.io/kube-apiserver-client for standard Kubernetes API server client certs? Looking through docs / forums/GH I can't see anything definitive, but I've noticed that in 1.21 and 1.19 if I use that I get a cert Approved but not issued?

[cdenneen]

@mikestef9 with 1.22 out, I didn’t see that procedure in the blog post by @chris-short

[mikestef9]

This is now covered in our documentation https://docs.aws.amazon.com/eks/latest/userguide/cert-signing.html. I will leave this issue open for a little while to make sure what we included in v1.22 launch meets the needs outlined in this issue.

[raesene]

@mikestef9 is there a docs page which mentions which signers are supported? I checked kubernetes.io/kube-apiserver-client on a 1.22 cluster but I'm still seeing approved but non-issued certs. What I'm not sure of is if that's by design or not.

[mikestef9]

Can you clarify what you mean by which signers. From the doc, this is signer name we support

signerName: beta.eks.amazonaws.com/app-serving

[raesene]

@mikestef9 so in upstream Kubernetes there are a number of in-built signers as detailed here I'm looking at one of these kubernetes.io/kube-apiserver-client and what I'm seeing is that certificate requests can be approved using that signer but the certificates are never issued.

Looking at the EKS docs, I can't see a definitive list of supported signers, so I'm not sure if the behaviour I'm seeing is a bug or working as intended.

This is a quick screenshot of what I'm seeing on EKS the cert is approved ok, but never issued.

[cobu2b]

I found the same issue on k8s 1.21 using kubernetes.io/kube-apiserver-client as a signer name. What should be the right approach? If I move to k8s 1.22, am I expected to use only custom signer as the document suggested? IMHO, the built-in signer names should work. Thank you

The output from kubectl version command:

Server Version: version.Info{Major:"1", Minor:"21+", GitVersion:"v1.21.5-eks-bc4871b", GitCommit:"5236faf39f1b7a7dabea8df12726f25608131aa9", GitTreeState:"clean", BuildDate:"2021-10-29T23:32:16Z", GoVersion:"go1.16.8", Compiler:"gc", Platform:"linux/amd64"}

[adrienjt]

By design EKS does not issue certificates for CSRs with signerName "kubernetes.io/kubelet-serving" unless the CSR was actually requested by a kubelet. EKS's custom signer validates this by checking that the requested SANs for CSRs with signerName kubernetes.io/kubelet-serving match an actual EC2 instance's IPs/DNS names. In other words, EKS does not issue certificates for CSRs with signerName kubernetes.io/kubelet-serving posing as kubelets, it only issues certificates for CSRs with signerName kubernetes.io/kubelet-serving for actual kubelets.

@mikestef9, this breaks virtual-kubelet implementations that want to support kubectl logs/exec via the vk controller, where the CSR must be signed by kubelet-serving for the vk controller pod IP. Is there any way we could relax this design constraint? Or is there another way I could get a certificate trusted by the API server for kubelet serving?

[huangjiasingle]

I found the same issue on eks v1.19.15-eks-9c63c4 , v1.20.11-eks-f17b81 using kubernetes.io/kube-apiserver-client as a signer name.

[Alevsk]

For CSRs with server auth usage the docs https://docs.aws.amazon.com/eks/latest/userguide/cert-signing.html says to use beta.eks.amazonaws.com/app-serving signer.

But what signer to use for CSRs with client auth usage?

[muchtall]

I'm the same boat with signerName: kubernetes.io/kube-apiserver-client. Can't figure out a way of getting the client CSR approved.

[chuyee]

I'm having the same question: does eks support client auth anyway?

[marcos-pinheiro-ml]

I'm having a similar problem with Kubelet CSR's in EKS... I have some EKS clusters in 1.21 and 1.22 versions and some Kubelets started to request by CSR but many CSRs stood in pending or approved state and few CSRs with approved and issued state.

The hostname-type is the same for LaunchTemplate and Subnets with IP Name type matching with SAN DNS and IP. All these CSRs are requested by real Kubelets and fails when ControlPlane or other tools try to comunicate with Kubelet in default TLS port.

Even with hostname type ip-name doesn't work :( Some idea why in a few worker nodes works?

[dclayton-godaddy]

We are running into this as well. We need client auth support for OPA admission controller webhooks. Do we have an actual issue to track client auth? We need kubernetes.io/kube-apiserver-client to issue the cert.

[mikeshng]

We are also running into this issue because we need client auth support.
Using the beta.eks.amazonaws.com/app-serving as signer, it errors with

    message: 'invalid usages for eks app serving certificate: [digital signature key encipherment client auth]'
    reason: SignerValidationFailure
    status: "True"
    type: Failed

Any workaround available?

[stevenzamborsky]

I'm hitting this issue as well when trying to setup Vault on EKS via the HashiCorp learn guide here.

Following the AWS Certificate Signing Instructions does indeed work when following the Standalone Vault TLS guide with the following caveats.

• You must be using EKS version 1.22 or higher (I was using version 1.21)
• You must update the signerName of the csr.yaml from kubernetes.io/kubelet-serving to beta.eks.amazonaws.com/app-serving. This results in the CSR being both Approved and Issued. I'll include the updated CSR for the HashiCorp guide below.

cat <<EOF >${TMPDIR}/csr.yaml
apiVersion: certificates.k8s.io/v1
kind: CertificateSigningRequest
metadata:
  name: ${CSR_NAME}
spec:
  groups:
  - system:authenticated
  request: $(cat ${TMPDIR}/server.csr | base64 | tr -d '\r\n')
  signerName: beta.eks.amazonaws.com/app-serving
  usages:
  - digital signature
  - key encipherment
  - server auth
EOF

And here's some sample output demonstrating that it worked:

$ kubectl create -f ${TMPDIR}/csr.yaml
certificatesigningrequest.certificates.k8s.io/vault-csr created

$ kubectl certificate approve ${CSR_NAME}
certificatesigningrequest.certificates.k8s.io/vault-csr approved

$ kubectl get csr ${CSR_NAME}
NAME        AGE   SIGNERNAME                           REQUESTOR          REQUESTEDDURATION   CONDITION
vault-csr   0s    beta.eks.amazonaws.com/app-serving   kubernetes-admin   <none>              Approved,Issued

$ kubectl get secret vault-server-tls -o jsonpath='{.data.vault\.crt}' --namespace=vault-namespace | base64 -d
-----BEGIN CERTIFICATE-----
MIIEEDCCAvigAwIBAgIUBMayz5vzpUL0LHhGyv/0LzdzEKEwDQYJKoZIhvcNAQEL
...
WpEPdQITdq6DY7ZumGBWWOoRhKNr75o8YRNiFjLopDvdh/Nu
-----END CERTIFICATE-----

[grraffe]

If you're using EKS on Fargate, check your cluster role if its trust relationship allows AssumeRole for "eks.amazonaws.com".

[gvinayakk]

The documentation provided https://docs.aws.amazon.com/eks/latest/userguide/cert-signing.html works to have an approved, issued CSR. But existing deployed mutatingwebhook configuration has started throwing following errors:

CSR was applied via local user having admin access.

Kubectl Version:
Client Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.0", GitCommit:"cb303e613a121a29364f75cc67d3d580833a7479", GitTreeState:"clean", BuildDate:"2021-04-08T16:31:21Z", GoVersion:"go1.16.1", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"22+", GitVersion:"v1.22.10-eks-84b4fe6", GitCommit:"cc6a1b4915a99f49f5510ef0667f94b9ca832a8a", GitTreeState:"clean", BuildDate:"2022-06-09T18:24:04Z", GoVersion:"go1.16.15", Compiler:"gc", Platform:"linux/amd64"}

Error message:
x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kubernetes"

Anybody else encountering similar issues? The current CA bundle points to eks cluster certificate authority. what should be the right config here?

#Update 1:
To get rid of the error, I update the caBundle config with contents of generated cert i.e cert.pem.

[mclayer]

And still, what about client auth? Is there any way to auth as a client in EKS cluster without IAM(and aws-cli on a local host)? Currently I can't auth using just plain client certificates, because I was unable to sign them.

[vipulsabhaya]

EKS currently does not support Kubelet client auth, hence we do not currently sign client certificates.

[mrf]

"EKS currently does not support Kubelet client auth, hence we do not currently sign client certificates." Would save a lot of people a lot of trouble if that line was right at the top of the docs.

[garo]

By design EKS does not issue certificates for CSRs with signerName "kubernetes.io/kubelet-serving" unless the CSR was actually requested by a kubelet.

May I ask @mikestef9 why this design choise? Parting from Kubernetes standard just causes issues to users and in our case I did spent quite a few hours to figure out why a previously working subsystem suddenly broke.

Also having to use beta.eks.amazonaws.com/app-serving as the signerName just screams of something which will eventually break again. Best practise would to use only stable APIs and in this case EKS going backwards is not what I want from a stable product.