It is not possible to configure Agones HELM with Stackdriver in GCloud when the cluster has Workload Identity.

[sisso]

What happened:

The "Workload Identity is the recommended way to access Google Cloud" [0]. When you enable it, your pods will be unable to use the VM/Node IAM anymore but will require that you allow a Google Service Account to be used as a workloadIdentity, and annotate a Kubernetes service account to assume, "Kubernetes workload know which service account to use to access Google Cloud services" [1]:

Agones HELM create its own k8n ServiceAccount for service-controller and there are no variables to add the annotations "iam.gke.io/gcp-service-account=[GSA_NAME]@[PROJECT_NAME].iam.gserviceaccount.com"

[0] - https://cloud.google.com/kubernetes-engine/docs/how-to/workload-identity
[1] - https://cloud.google.com/blog/products/containers-kubernetes/introducing-workload-identity-better-authentication-for-your-gke-applications

What you expected to happen:

Deploy with HELM with arguments that flag my cluster uses Workload Identity and what IAM the ServiceAccount should be annotated to be assumed.

How to reproduce it (as minimally and precisely as possible):

1. Create new gcloud cluster with workload identity: https://cloud.google.com/kubernetes-engine/docs/how-to/workload-identity#enable_on_cluster
2. Deploy agones by helm using slackdrive: https://agones.dev/site/docs/guides/metrics/#stackdriver-installation
3. Find and fail to find a way to give agones permissions to send slackdrive metrics without manually annotate the Agones ServiceAccounts.

Anything else we need to know?:

Workarounds are to change the helm locally or annotate by command line after the deployment.

Environment:

• Agones version: 1.13

• Kubernetes version (use kubectl version):

    Client Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.0", GitCommit:"cb303e613a121a29364f75cc67d3d580833a7479", GitTreeState:"clean", BuildDate:"2021-04-16T18:16:59Z", GoVersion:"go1.16.3", Compiler:"gc", Platform:"linux/amd64"}
    Server Version: version.Info{Major:"1", Minor:"18+", GitVersion:"v1.18.17-gke.100", GitCommit:"c702fe983d18291df698149425102b51a8258b71", GitTreeState:"clean", BuildDate:"2021-03-19T09:19:23Z", GoVersion:"go1.13.15b4", Compiler:"gc", Platform:"linux/amd64"}
  

• Cloud provider or hardware configuration:

GCloud + linux

• Install method (yaml/helm): HELM
• Troubleshooting guide log(s):

2021/05/12 11:04:03 Failed to export to Stackdriver: rpc error: code = PermissionDenied desc = Permission monitoring.metricDescriptors.create denied (or the resource may not exist).

• Others:

[markmandel]

Thanks for the bug! Looks like some things have changed since it was written.

We have examples of allowing arbitrary annotations in places in the helm charts. We could do something similar and also document that area too. I wonder if we should do something similar here, and document it.

@roberthbailey you got any thoughts?

(Also noting that stackdriver is no longer named stackdriver!)

[roberthbailey]

I was able to reproduce the error (and the fix by manually applying the required annotations and restarting the agones system pods).

https://cloud.google.com/kubernetes-engine/docs/how-to/workload-identity#gcloud_1 describes what the command is to manually add the annotation:

kubectl annotate serviceaccount \
  --namespace K8S_NAMESPACE \
  KSA_NAME \
  iam.gke.io/gcp-service-account=GSA_NAME@PROJECT_ID.iam.gserviceaccount.com

Because this annotation includes both the project name (which helm doesn't know) and also the customized name of the google service account (GSA_NAME) it's impossible for us to properly set this annotation automatically in the helm charts. But as Mark mentioned we can make it so that you can pass it in as a setting so that you can set it during installation.

One other thing I noticed is that after creating the GSA, the IAM binding, and adding the annotation, I had to restart the agones pods for them to successfully connect to stackdriver. I still need to test to see if running helm upgrade --install --wait with the new parameters will restart pods or just patch them with the new annotation.