infra/gcp: k8s-metrics move cleanup

[spiffxp]

Related:

• Part of: Migrate test-infra metrics to wg-k8s-infra #1306
• Followup to: config/jobs: promote metrics-bigquery-canary test-infra#23115

Specific things done here:

• add the roles/bigquery.user role to k8s-infra-prow-build-trusted's terraform that I had to manually add to get the metrics-bigquery-canary job working
• switch the special-case in ensure-main-project.sh to create gs://k8s-metrics, and flip the iam binding calls for the google.com prow instance to removes

While I was here I also tried to reorganize some of the service account creation resources in k8s-infra-prow-build-trusted terraform into a workload_identity_service_account pseudo-module

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: spiffxp

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [spiffxp]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[spiffxp]

/cc @ameukam @dims

[spiffxp]

Updated PTAL

[ameukam]

/lgtm
/hold
Remove hold when ready to apply.

[spiffxp]

/hold cancel

[spiffxp]

Ran ./infra/gcp/ensure-main-project.sh

Ensuring prow special cases for: kubernetes-public
  Special case: ensuring k8s-infra-ci-robot-github-token accessible by k8s-infra-prow-build-trusted
  Special case: ensuring gs://k8s-metrics exists
    @@ -20,5 +20,3 @@
       role: roles/storage.objectAdmin
     - member: serviceAccount:triage@k8s-gubernator.iam.gserviceaccount.com
       role: roles/storage.legacyBucketWriter
    -- member: serviceAccount:triage@k8s-gubernator.iam.gserviceaccount.com
    -  role: roles/storage.objectAdmin
    @@ -18,5 +18,3 @@
       role: roles/storage.legacyBucketWriter
     - member: serviceAccount:k8s-metrics@k8s-infra-prow-build-trusted.iam.gserviceaccount.com
       role: roles/storage.objectAdmin
    -- member: serviceAccount:triage@k8s-gubernator.iam.gserviceaccount.com
    -  role: roles/storage.legacyBucketWriter
    Updated IAM policy for project [kubernetes-public].
    @@ -54,6 +54,8 @@
       role: roles/viewer
     - member: serviceAccount:k8s-infra-monitoring-viewer@kubernetes-public.iam.gserviceaccount.com
       role: roles/monitoring.viewer
    +- member: serviceAccount:k8s-metrics@k8s-infra-prow-build-trusted.iam.gserviceaccount.com
    +  role: roles/bigquery.user
     - member: serviceAccount:kubernetes-external-secrets@kubernetes-public.iam.gserviceaccount.com
       role: roles/secretmanager.secretAccessor
     - member: serviceAccount:kubernetes-public@appspot.gserviceaccount.com

[spiffxp]

Ran terraform apply

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # google_project_iam_member.k8s_metrics_sa_bigquery_user will be created
  + resource "google_project_iam_member" "k8s_metrics_sa_bigquery_user" {
      + etag    = (known after apply)
      + id      = (known after apply)
      + member  = "serviceAccount:k8s-metrics@k8s-infra-prow-build-trusted.iam.gserviceaccount.com"
      + project = "k8s-infra-prow-build-trusted"
      + role    = "roles/bigquery.user"
    }

Plan: 1 to add, 0 to change, 0 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

google_project_iam_member.k8s_metrics_sa_bigquery_user: Creating...
google_project_iam_member.k8s_metrics_sa_bigquery_user: Creation complete after 7s [id=k8s-infra-prow-build-trusted/roles/bigquery.user/serviceAccount:k8s-metrics@k8s-infra-prow-build-trusted.iam.gserviceaccount.com]

Apply complete! Resources: 1 added, 0 changed, 0 destroyed.