Autoscale of pubsub subscriber is failing with error

[apurvc]

Followed the article to scale the subscriber : https://cloud.google.com/kubernetes-engine/docs/tutorials/autoscaling-metrics#pubsub_8 however the HPA is failing with mentioned error. I am using a private cluster (1.15.12-gke.2) which should not be an issue as per my understanding.

ScalingActive False FailedGetExternalMetric the HPA was unable to compute the replica count: unable to get external metric default/pubsub.googleapis.com|subscription|num_undelivered_messages/&LabelSelector{MatchLabels:map[string]string{resource.labels.subscription_id: echo-read,},MatchExpressions:[],}: unable to fetch metrics from external metrics API: pubsub.googleapis.com|subscription|num_undelivered_messages.external.metrics.k8s.io is forbidden: User "system:serviceaccount:kube-system:horizontal-pod-autoscaler" cannot list resource "pubsub.googleapis.com|subscription|num_undelivered_messages" in API group "external.metrics.k8s.io" in the namespace "default"

• I could see the metrics in stackdriver however hpa seems to be failing.
• tried adding cluserrole binding for the given user however no success :(

[apurvc]

As updated in GoogleCloudPlatform/k8s-stackdriver#315 by @aubm i was able to get this to work

I managed to make it work with WI using the following approach :

gcloud iam service-accounts create custom-metrics-sd-adapter --project "$GCP_PROJECT_ID"

gcloud projects add-iam-policy-binding "$GCP_PROJECT_ID" \
  --member "serviceAccount:custom-metrics-sd-adapter@$GCP_PROJECT_ID.iam.gserviceaccount.com" \
  --role "roles/monitoring.editor"

gcloud iam service-accounts add-iam-policy-binding \
  --role roles/iam.workloadIdentityUser \
  --member "serviceAccount:$GCP_PROJECT_ID.svc.id.goog[custom-metrics/custom-metrics-stackdriver-adapter]" \
  "custom-metrics-sd-adapter@$GCP_PROJECT_ID.iam.gserviceaccount.com"

kubectl create -f https://raw.githubusercontent.com/GoogleCloudPlatform/k8s-stackdriver/master/custom-metrics-stackdriver-adapter/deploy/production/adapter.yaml

kubectl annotate serviceaccount custom-metrics-stackdriver-adapter \
  "iam.gke.io/gcp-service-account=custom-metrics-sd-adapter@$GCP_PROJECT_ID.iam.gserviceaccount.com" \
  --namespace custom-metrics

[Shabirmean]

@apurvc

Thank you very much for reporting this issue. The issue seems to be present only for clusters that have Workload Identity enabled. All our docs are based on the default cluster setup enabled for clusters created via the gcloud tool.

I have updated the documentation to inform the readers about it and have pointed to the Github k8s-stackdriver setup example. The second step "Start Custom Metrics - Stackdriver Adapter" explains the steps required which are similar to the ones you have shared above.

For reference, we can get the setup from the tutorial working (for WI enabled clusters) by executing the following steps before the step to "Download the service account key file".

gcloud projects add-iam-policy-binding "$PROJECT_ID" \
    --member "serviceAccount:autoscaling-pubsub-sa@$PROJECT_ID.iam.gserviceaccount.com" \
    --role "roles/monitoring.editor"

gcloud iam service-accounts add-iam-policy-binding \
    --role roles/iam.workloadIdentityUser \
    --member "serviceAccount:$PROJECT_ID.svc.id.goog[custom-metrics/custom-metrics-stackdriver-adapter]" \
    autoscaling-pubsub-sa@$PROJECT_ID.iam.gserviceaccount.com

kubectl annotate serviceaccount \
    --namespace custom-metrics \
    custom-metrics-stackdriver-adapter \
    iam.gke.io/gcp-service-account=autoscaling-pubsub-sa@$PROJECT_ID.iam.gserviceaccount.com

Please note that it might take a few minutes for the service account changes to be propagated. Thus, initially the HPA might fail for a bit.

Further you can verify if the WI enabled service account was properly linked by following step 7 of this tutorial.