401 Unauthorized for GCP Cloud Monitoring sink after an hour

[ansel1]

`### A note for the community

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Problem

Using the GCP Cloud Monitoring sync, with vector running in a Pod in GKE. Using GCP workload identity (GCP service account linked to GKE service account) to authenticate to GCP.

This works fine for an hour, then we start seeing 401 errors. 100% reproducible.

Configuration

vector config for sink:

    [sinks.gcp]
    type = "gcp_stackdriver_metrics"
    inputs = [ "enrich_metrics" ]
    project_id = "${GCP_PROJECT_ID:?gcp_project_id must be supplied}"
    
    [sinks.gcp.resource]
    type = "gce_instance"
    project_id = "${GCP_PROJECT_ID:?gcp_project_id must be supplied}"
    instance_id = "cloud_containers"
    zone = "us-central1-a"

Source can be any metrics source. Vector is configured as a StatefulSet. The pod spec includes:

      serviceAccountName: vector
      nodeSelector:
        iam.gke.io/gke-metadata-server-enabled: "true"

The k8s service account is declared like this:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: vector
  labels:
    app.kubernetes.io/name: vector
    app.kubernetes.io/instance: vector
    app.kubernetes.io/component: Aggregator
    app.kubernetes.io/version: "latest-alpine"
  annotations:
    iam.gke.io/gcp-service-account: vector@our-gcp-project.iam.gserviceaccount.com
automountServiceAccountToken: true

Version

v0.29.1

Debug Output

2023-04-27T12:03:47.174065Z ERROR sink{component_kind="sink" component_id=gcp component_type=gcp_stackdriver_metrics component_name=gcp}:request{request_id=6983}: vector::sinks::util::sink: Response failed. response=Response { status: 401, version: HTTP/1.1, headers: {"www-authenticate": "Bearer realm="https://accounts.google.com/\", error="invalid_token"", "vary": "Origin", "vary": "X-Origin", "vary": "Referer", "content-type": "application/json; charset=UTF-8", "transfer-encoding": "chunked", "date": "Thu, 27 Apr 2023 12:03:47 GMT", "server": "ESF", "cache-control": "private", "x-xss-protection": "0", "x-frame-options": "SAMEORIGIN", "x-content-type-options": "nosniff"}, body: b"{\n "error": {\n "code": 401,\n "message": "Request had invalid authentication credentials. Expected OAuth 2 access token, login cookie or other valid authentication credential. See https://developers.google.com/identity/sign-in/web/devconsole-project.\",\n "status": "UNAUTHENTICATED",\n "details": [\n {\n "@type": "type.googleapis.com/google.rpc.ErrorInfo",\n "reason": "ACCESS_TOKEN_EXPIRED",\n "domain": "googleapis.com",\n "metadata": {\n "method": "google.monitoring.v3.MetricService.CreateTimeSeries",\n "service": "monitoring.googleapis.com"\n }\n }\n ]\n }\n}\n" }

Example Data

No response

Additional Context

No response

References

#10828

[spencergilbert]

Taking it look it doesn't look like the metrics sink calls the function that refreshes the authentication 🤔 digging into that, might be a super easy fix.