google_container_cluster.cluster_autoscaling.auto_provisioning_defaults does not allow oauth_scopes and service_account

[dancmeyers]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request.
• Please do not leave +1 or me too comments, they generate extra noise for issue followers and do not help prioritize the request.
• If you are interested in working on this issue or have submitted a pull request, please leave a comment.
• If an issue is assigned to the modular-magician user, it is either in the process of being autogenerated, or is planned to be autogenerated soon. If an issue is assigned to a user, that user is claiming responsibility for the issue. If an issue is assigned to hashibot, a community member has claimed the issue already.

Terraform Version

Terraform v0.12.20
+ provider.google v3.8.0
+ provider.google-beta v3.8.0
+ provider.random v2.2.1

Affected Resource(s)

• google_container_cluster

Terraform Configuration Files

resource "google_service_account" "account" {
  account_id   = "gke-service"
}

resource "google_project_iam_member" "roles" {
  for_each = toset(["roles/container.clusterViewer", "roles/artifactregistry.reader", "roles/storage.objectViewer"])
  role     = each.value
  member   = "serviceAccount:${google_service_account.account.email}"
}

resource "google_container_cluster" "kubernetes" {
  provider   = google-beta
  name       = "test-gke"
  location   = "europe-west1"

  logging_service    = "logging.googleapis.com/kubernetes"
  monitoring_service = "monitoring.googleapis.com/kubernetes"

  remove_default_node_pool = false
  initial_node_count       = 1

  release_channel {
      channel = "STABLE"
  }

  master_auth {
    # This disables HTTP basic auth
    username = ""
    password = ""

    client_certificate_config {
      issue_client_certificate = false
    }
  }

  node_config {
    preemptible     = false
    machine_type    = "n1-standard-1"
    service_account = google_service_account.account.email

    tags = ["gke-cluster"]

    metadata = {
      disable-legacy-endpoints = "true"
    }

    oauth_scopes = [
      "https://www.googleapis.com/auth/cloud-platform"
    ]
  }

  addons_config {
    horizontal_pod_autoscaling {
      disabled = false
    }

    http_load_balancing {
      disabled = false
    }
  }

  ## cluster_autoscaling configuration
  cluster_autoscaling {
    enabled = var.enable_autoscaling
    resource_limits {
      resource_type = "memory"
      minimum       = 10
      maximum       = 100

    }
    resource_limits {
      resource_type = "cpu"
      minimum       = 1
      maximum       = 10
    }
    # This structure contains defaults for a node pool created by NAP (Node Auto-provisioning)
    auto_provisioning_defaults {
      service_account = google_service_account.account.email
      oauth_scopes = [
        "https://www.googleapis.com/auth/cloud-platform"
      ]
    }
  }
}

Debug Output

https://gist.github.com/dancmeyers/ed4fc39a005357279ccc42ead6ff898d

Panic Output

N/A

Expected Behavior

A cluster should be created with node autoprovisioning enabled, where any new nodes spun up use our custom service account, and also have the full cloud-platform oauth scope so that they can access all GCP APIs and restrictions are done using fine-grained IAM on the service account itself.

Actual Behavior

An error is generated (from the Terraform provider, rather than the GCP API), saying that we are not allowed to specify both oauth_scopes and service_account:

Error: "cluster_autoscaling.0.auto_provisioning_defaults.0.oauth_scopes": only one of `cluster_autoscaling.0.auto_provisioning_defaults.0.oauth_scopes,cluster_autoscaling.0.auto_provisioning_defaults.0.service_account` can be specified, but `cluster_autoscaling.0.auto_provisioning_defaults.0.oauth_scopes,cluster_autoscaling.0.auto_provisioning_defaults.0.service_account` were specified.
  on .terraform/modules/picozoa.kubernetes/main.tf line 73, in resource "google_container_cluster" "kubernetes":
  73: resource "google_container_cluster" "kubernetes" {

If we remove oauth_scopes from the definition then when new nodes are autoprovisioned, if we do not use the default service account, the nodes do not even have enough access to pull container images from the container registry in the same project. This is because the default enabled scopes do not include one with that level of access. The default pool, using the same service account and the cloud-platform scope, works fine.

Either we should be able to specify both these options (which the gcloud CLI tool allows), or the cloud-platform 'all access' scope should be defaulted when a service account is provided.

Steps to Reproduce

1. terraform apply

Important Factoids

N/A

References

N/A

[rileykarson]

Duplicate of #5545, a fix for this should have landed in 3.9.0 yesterday. If you find that it doesn't work, do you mind filing a new issue?

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. If you feel I made an error 🤖 🙉 , please reach out to my human friends 👉 hashibot-feedback@hashicorp.com. Thanks!