Kubernetes 1.24: Token-handling code assumes auto-created service account token secrets

[liggitt]

Sweeping token-scraping of auto-generated Kubernetes token secrets in preparation for Kubernetes 1.24 showed the following code locations assume auto-generated tokens will exist:

argo-workflows/workflow/controller/operator.go

Line 3605 in 62e0a8c

return account.Secrets[0].Name, nil

argo-workflows/server/auth/webhook/interceptor.go

Line 90 in a3c326f

tokenSecret, err := secretsInterface.Get(ctx, serviceAccount.Secrets[0].Name, metav1.GetOptions{})

argo-workflows/server/auth/gatekeeper.go

Line 321 in a3c326f

secret, err := s.cache.SecretLister.Secrets(serviceAccount.GetNamespace()).Get(serviceAccount.Secrets[0].Name)

That assumption is not universally correct.

In 1.21+, secret-based tokens are no longer used for mounting into pods (ephemeral time-limited tokens are), and the token controller can be turned off.

In 1.24+, secret-based tokens are no longer auto-created by default for new service accounts.

Using ephemeral time-bound tokens is preferred in 1.21+ (see the TokenRequest API) if possible.

If a secret-based token is still desired, one can be created manually, but will not be referenced from the service account's .secrets list.

---

Message from the maintainers:

Impacted by this bug? Give it a 👍. We prioritise the issues with the most 👍.

[alexec]

@liggitt do you have some thoughts on how to adapt the code to address this?

[liggitt]

if you need tokens to use outside the context of a pod (where they continue to be mounted in automatically for use), either request one using the TokenRequest API, or create a secret to hold one.

[alexec]

None of these SAs are useful if they do not have a token. The user could presumably create a token?

[liggitt]

Service account tokens are not automatically ambient in secrets in 1.24+ (and possibly in 1.21+ if the cluster owner turned off the token controller since it is no longer required for getting tokens into pods).

Tokens are created automatically for mounting into pods (but are not persisted in Secret API objects).

If a secret-based token is desired/needed, it can be created manually, but would not be referenced from the ServiceAccount's secrets list field.

[liggitt]

it looks like at least some of the uses of getServiceAccountTokenName are to get the name of a Secret object in order to create a volume mount mounting in the token?

If you can't resolve a secret name, I'd recommend using projected token volumes instead, which let you request a token be mounted into a pod without needing to look up a Secret name.

See https://kubernetes.io/docs/reference/access-authn-authz/service-accounts-admin/#bound-service-account-token-volume for details on how to replicate the content of an old secret-based token volume with the new projected token volumes

[alexec]

Do project volumes work on GKE auto-pilot?

[liggitt]

yes

[alexec]

@jessesuen @jannfis FYI

[alexec]

Project volumes will not work for our case. We don't know the name of the service account token ahead of time. A single pod read many different tokens.

I think users will have to create the tokens themselve. We should update the code to returrn an error (rather than panic) so that users know what to do.

@liggitt thoughts?

[liggitt]

We don't know the name of the service account token ahead of time.

The name of the token, or the name of the service account?

A single pod read many different tokens.

Projected token volumes can be used multiple times, but only for a single service account (the one referenced by the pod spec)

I think users will have to create the tokens themselves

If so, they'll either need to give you the names of the secrets they created (since they won't be referenced from the serviceaccount's .secrets field), or you'll have to query for the secrets (filtering by type, then matching up the kubernetes.io/service-account.name annotation to the name of the service account you're looking for).

Use of secret-based tokens (even manually created ones) is discouraged, since it exposes token content via the API.

[alexec]

We don't know the name of the service account token ahead of time.

The name of the token, or the name of the service account?

Sorry. We don't know the service account name ahead of time (i.e. when pod starts up).

If so, they'll either need to give you the names of the secrets they created (since they won't be referenced from the serviceaccount's .secrets field), or you'll have to query for the secrets (filtering by type, then matching up the kubernetes.io/service-account.name annotation to the name of the service account you're looking for).

Use of secret-based tokens (even manually created ones) is discouraged, since it exposes token content via the API.

I understand.

[lknite]

This looks related "argocd cluster add ...", timing out waiting for a secret.

[jrhoward]

looking at the code you could change it so that it retrieves the secret by annotation
kubernetes.io/service-account.name: "service account name " in the method getServiceAccountTokenName using the operator.go as an example but that requires users to explicitly create the secret. This means you would be missing out on the security enhancements provided in 1.24.

It seems to me that the long term solution is to create a token at run time from the argoexe container using something like

...CoreV1().ServiceAccounts(Namespace).CreateToken(ctx, name, &tokenRequest, metav1.CreateOptions{})

based on the name of the service account and inject it into the the container as a tmpfs volume as there will be no way to mount a pre-existing service account secret

[urbantz-technical-support]

I found another place where we assume tokens to be auto generated:

argo-workflows/server/auth/gatekeeper.go

Line 319 in 30bd96b

if len(serviceAccount.Secrets) == 0 {

This essentially means that you can't run argo-workflows on k8s 1.24 (like we do) with sso enabled.

Maybe until it gets addressed we should add something to the documentation, or maybe pinned this issue explaining it is not compatible with k8s 1.24

[alexec]

@juliev0 @sarabala1979 this issue needs to scheduled and fixed.

[sarabala1979]

@alexec I will add it in RTB we will look at this after 3.4.0. we can target and include it in 3.4.1

[alexec]

v3.5 would be fine I think.

[jrhoward]

Don't think I am qualified to take over the PR but I'll help out where I can as I have the branch running locally with 1.24.

I have installed the sample executor plugin and successfully run a sample workflow based on the revised instructions with a service account token suffixed with .service-account-token.

[jrhoward]

interceptor.go is missing updates. I modified and built locally.

argo-workflows/server/auth/webhook/interceptor.go

Lines 87 to 90 in a3c326f

	if len(serviceAccount.Secrets) == 0 {
	return fmt.Errorf("failed to get secret for service account \"%s\": no secrets", serviceAccountName)
	}
	tokenSecret, err := secretsInterface.Get(ctx, serviceAccount.Secrets[0].Name, metav1.GetOptions{})

Remove lines 87 to 89 and change line 90 from serviceAccount.Secrets[0].Name to leverage the new util:

tokenSecret, err := secretsInterface.Get(ctx, secrets.SecretName(serviceAccount), metav1.GetOptions{})

[gcsfred2]

I followed the steps of https://argoproj.github.io/argo-workflows/running-locally/ cloning @terrytangyuan 's https://github.com/terrytangyuan/argo-workflows/tree/support-k8s-124 and I got an error with make start:
https://pastebin.com/NCn3s7Wr

[terrytangyuan]

@gcsfred2 That is unrelated. Try creating a new local cluster and start again.

[gcsfred2]

Retrying https://github.com/terrytangyuan/argo-workflows/tree/support-k8s-124 and running locally. My recent error: https://pastebin.com/hBySXzWV . There was no error running "make start UI=true" and the Hello World example.

[jrhoward]

ok I just discovered that there is a work around which does not require a code change on argoworkflow.

Not sure if it is documented so I can't vouch for how long it will last but you can manually insert the secret name into the service account manifest which means you can deploy workflow without a fix into 1.24, as an example this works on server version:
{Major:"1", Minor:"24", GitVersion:"v1.24.4+k3s1}, yes tested with k3d > k3s

note the trick is to define the secret name in the service account manifest

I applied the following manifests:

apiVersion: v1
kind: Secret
type: kubernetes.io/service-account-token
metadata:
  name: uber-argo
  annotations:
    kubernetes.io/service-account.name: "uber-argo"
---

apiVersion: v1
kind: ServiceAccount
metadata:
  name: uber-argo
secrets:
- name: uber-argo

functionality has been removed but the spec is still valid

[terrytangyuan]

I won't have time to finish #9680 given other priorities, so I am closing that PR and freeing up this issue if anyone else wants to continue the work. The TokenRequest API suggested by @jessesuen seems like a better approach. ArgoCD has done this so feel free to use the following resources as reference:

• fix: create serviceaccount token for v1.24 clusters argo-cd#9546
• Unable to add 1.24.0 Kubernetes cluster argo-cd#9422
• https://docs.google.com/document/d/1MmYIfM8tbEp2irCaLtgrv9jJL-coYG3u-wa3xTVkEOU

If you are blocked by the error failed to get token volumes: service account argo/default does not have any secrets, please use the above workaround for now.

[alexec]

@jrhoward awesome work finding the work-around. It does appear to be documented:

https://kubernetes.io/docs/reference/access-authn-authz/service-accounts-admin/

The short term fix myself and @sarabala1979 were considering was exactly the same as this.

[liggitt]

Manually adding token secrets to the list in the service account object is not expected. That list is specifically for secrets which are mounted into pods, and secret based tokens no longer are in 1.21+

[sarabala1979]

@terrytangyuan can you update the PR with @jrhoward examples for Short term solutions? ServiceAccount will have an annotation with a secret name. So the controller will look at the annotation if there is no Secret on ServiceAccount.

[terrytangyuan]

@terrytangyuan can you update the PR with @jrhoward examples for Short term solutions? ServiceAccount will have an annotation with a secret name. So the controller will look at the annotation if there is no Secret on ServiceAccount.

I couldn't get some tests passing and didn't have much time to investigate further. The remaining issues are documented in #9680 (comment) in case anyone else wants to pick up.

[alexec]

Hmmm.kubernetes.io/service-account.name is an annotation on the secret, not the service account. We actually need the service account to point to the secret.

[rubenpetrosyan1]

Hi team,
Is the issue assumed as fixed?
I'm asking because this comes up pretty frequently on Slack.

[zhaorizhao]

set secrets to make this field accessible:

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: read-only
  namespace: argo
  annotations:
    workflows.argoproj.io/rbac-rule: "true"
    workflows.argoproj.io/rbac-rule-precedence: "0"
secrets: 
- name: read-only-token-manual
---
apiVersion: v1
kind: Secret
metadata:
  name: read-only-token-manual
  namespace: argo
  annotations:
    kubernetes.io/service-account.name: "read-only"
type: kubernetes.io/service-account-token