beta-autopilot-private-cluster uses default service account

[TrieBr]

TL;DR

Using the beta-autopilot-private-cluster uses a the default service account for the cluster created in GCP.

gcloud container clusters describe --region northamerica-northeast2 cluster-main-test-29420926 | grep serviceAccount
    serviceAccount: default
  serviceAccount: default
    serviceAccount: default

Looking in cluster.tf I don't even see any reference to the service_account variable.

Expected behavior

The cluster should use the service account created by the module.

Observed behavior

Cluster uses default service account which is against GKE hardening guidelines.

Terraform Configuration

resource "random_id" "cluster" {
  byte_length = 4
}

// Subnet for Cluster.
resource "google_compute_subnetwork" "cluster-subnet" {
  project       = var.project_id
  name          = "cluster-${var.name}-subnet-${var.environment_realm}-${random_id.cluster.hex}"
  ip_cidr_range = "10.3.0.0/16"
  region        = var.region
  network       = var.network
  private_ip_google_access = true
  secondary_ip_range {
    range_name    = "cluster-${var.name}-subnet-${var.environment_realm}-range-pods"
    ip_cidr_range = "192.168.10.0/24"
  }
  secondary_ip_range {
    range_name    = "cluster-${var.name}-subnet-${var.environment_realm}-range-services"
    ip_cidr_range = "192.168.11.0/24"
  }
  depends_on = [
    var.network
  ]
}

module "main-cluster" {
  source                     = "terraform-google-modules/kubernetes-engine/google//modules/beta-autopilot-private-cluster"
  project_id                 = var.project_id
  name                       = "cluster-${var.name}-${var.environment_realm}-${random_id.cluster.hex}"
  region                     = var.region
  zones                      = ["${var.region}-a", "${var.region}-b"]
  network                    = var.network
  subnetwork                 = google_compute_subnetwork.cluster-subnet.name
  ip_range_pods              = "cluster-${var.name}-subnet-${var.environment_realm}-range-pods"
  ip_range_services          = "cluster-${var.name}-subnet-${var.environment_realm}-range-services"
  horizontal_pod_autoscaling = true
  enable_private_endpoint    = false
  enable_private_nodes       = true
  enable_vertical_pod_autoscaling = true
  release_channel = "RAPID"
  
  master_ipv4_cidr_block     = "10.0.0.0/28"
  depends_on = [
    google_compute_subnetwork.cluster-subnet
  ]
}

Terraform Version

+ provider registry.terraform.io/hashicorp/google v4.36.0
+ provider registry.terraform.io/hashicorp/google-beta v4.44.1
+ provider registry.terraform.io/hashicorp/kubernetes v2.16.0
+ provider registry.terraform.io/hashicorp/null v3.2.1
+ provider registry.terraform.io/hashicorp/random v3.4.3
+ provider registry.terraform.io/hashicorp/time v0.9.1

Additional information

No response

[bharathkkb]

@TrieBr Thanks for the report! I believe we initially did not add this due to the conflictsWith on the provider. Happy to review a PR fixing this if the provider bug is resolved.

https://github.com/hashicorp/terraform-provider-google/blob/d2ea6580ef848edd94862d13ab7cddcb162df904/google/resource_container_cluster.go#L386

[ferrarimarco]

Hi @bharathkkb ! I tried fixing this in #1495 :)

[rojomisin]

So does changing an Autopilot cluster's service account work for anyone yet? I've read a lot of threads now and I've put in a support ticket, but I keep seeing resolved and merged PRs, but with the latest 4.54 provider using google_container_cluster and adding

  cluster_autoscaling {
    auto_provisioning_defaults {
      ### https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/container_cluster#service_account
      service_account = var.service_account
    }
  }

I get a successful plan, but the apply gives:

│ Error: googleapi: Error 400: Overriding Autopilot autoscaling settings is not allowed., badRequest

[rojomisin]

per google support :

We would like to inform you that in the Autopilot GKE cluster the default SA can be changed only at the time of an Autopilot cluster creation and the SA for an existing Autopilot cluster cannot be changed.

I'm going to ask about why they have an update command which wouldn't support this, and if this is specified in their documentation, however just wanted to relay the information. Was this already known and assumed by the terraform gke module authors?

[rojomisin]

per google support :

We would like to inform you that in the Autopilot GKE cluster the default SA can be changed only at the time of an Autopilot cluster creation and the SA for an existing Autopilot cluster cannot be changed.

I'm going to ask about why they have an update command which wouldn't support this, and if this is specified in their documentation, however just wanted to relay the information. Was this already known and assumed by the terraform gke module authors?