Support configuration of Service Account Issuer Discovery

[JanKoppe]

Description of the feature you would like to add / User story

As a Kubernetes cluster administrator using KubeOne
I would like to be able to configure the service account issuer discovery parameters of kube-apiserver
in order to allow workloads to make use of AWS IRSA to authenticate against AWS APIs.

Solution details

• Implement ServiceAccountSettings in the KubeOneCluster spec, similar to https://docs.kubermatic.com/kubermatic/v2.25/references/crds/#serviceaccountsettings
• Kubeone will hand over the ServiceAccountSettings to kubeadm with the proper arguments
• kubeadm will configure the kube-apiserver with the proper arguments (e.g. at least --service-account-issuer)

In total, users need to be able to adjust the iss value in service account tokens to a user-specified value.

Alternative approaches

I've considered modifying the kube-apiserver manifest after the cluster is up and running. This won't work though as it is a persistent manifest on the disk of the controllers. It will also be overwritten by KubeOne/kubeadm.

I've considered providing a method to pass on free-form arguments to kubeadm, but it was decided against offering something like that in the past (See #3074 (comment))

Use cases

AWS provides IAM roles for service accounts as a defined mechanic to allow workloads in EKS to fetch temporary AWS API credentials with the specific permissions of an IAM role using their service account tokens.

While AWS markets this feature for their own EKS of course, this is actually not specific to EKS. The feature allows in general any verifiable web (openid) identity to assume an IAM role by providing a valid token.

A verifiably web identity in this case can be any OpenID connect identity provider that provides at least the .well-known/openid-configuration endpoint and a JWKS (JSON Web Key Set).

AWS describes the method to setup such an OpenID connect identity provider in this documentation: https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_providers_create_oidc.html

The complication that we're running into when using KubeOne is that AWS requires that the value of the issuer iss claim in the provided tokens must be of the same hostname that provides the .well-known/openid-configuration endpoint.

Kubernetes allows the configuration of that iss value through the --service-account-issuer flag for kube-apiserver (https://kubernetes.io/docs/reference/command-line-tools-reference/kube-apiserver/).

But KubeOne currently does not allow the user to configure this value to be passed on to kubeadm so that the kube-apiserver is configured correctly.

To compare: The Cluster spec of KKP does allow this to be configured, and we've confirmed that this will make clusters in KKP be able to use AWS IRSA. https://docs.kubermatic.com/kubermatic/v2.25/references/crds/#serviceaccountsettings

Additional information

Another complication that should be considered as context, but not part of the problem: The endpoint that is referenced in the iss value needs to be reachable via HTTPS with a valid TLS certificate (i.e. by a CA of AWS' trust store), but KubeOne sets up the kube-apiserver with a self-signed CA. This makes sense as the majority of KubeOne users probably do not have their own intermediate CA that is in AWS' trust store, so the easiest workaround here is to reverse-proxy that part of the kube-apiserver with a valid TLS certificate (e.g. through an application load balancer in AWS, or something like nginx + Let's Encrypt elswhere) which also allows to only expose those two specific URLs (openid-configuration and JWKS) to AWS in case users do not wish to publicly expose the kube-apiserver (which is recommended for production deployments). Having this external and independent from the kube-apiserver generates the need for this to be a user-definable value because KubeOne can't/shouldn't imply this URL from the provided apiserver hosts.

[JanKoppe]

Quick update: This feature makes most sense if you're also able to configure --service-account-jwks-uri. This one can be hacked around on the outside (just replace the .well-known/openid-configuration file with the correct value ;) ) but it's certainly much cleaner to announce a correct JWKS URI from the apiserver itself.