Provisioning of ASM and GKE in the same workspace fails

[jwtracy]

TL;DR

When provisioning a new instance of my module with GKE (beta-private-cluster-update-variant, 20.0.0) and ASM (21.0.0), Terraform fails to plan with strange error messages.

I'll note that I have an explicit dependency on the GKE module for ASM so I would expect it to provision both sequentially.

Expected behavior

Provision a GKE cluster, then provision ASM in that GKE cluster.

Observed behavior

https://gist.github.com/jwtracy/774261188c54fdefee09649674ccabbf

The first issue seems to come from the GKE module regarding a plugin, I am entirely unfamiliar with this kind of error.

│ Error: Plugin did not respond
│ 
│   with module.platform_eng_environments.module.plato_admin_instance.module.gke.module.gcloud_delete_default_kube_dns_configmap.module.gcloud_kubectl.null_resource.module_depends_on[0],
│   on .terraform/modules/platform_eng_environments.plato_admin_instance.gke.gcloud_delete_default_kube_dns_configmap/main.tf line 62, in resource "null_resource" "module_depends_on":
│   62: resource "null_resource" "module_depends_on" {
│ 
│ The plugin encountered an error, and failed to respond to the
│ plugin.(*GRPCProvider).ValidateResourceConfig call. The plugin logs may
│ contain more details.

But there are also cases where the ASM module is trying to get the data source of a cluster that doesn't exist yet and has null references within interpolated templates,

│ Error: Invalid template interpolation value
│ 
│   on .terraform/modules/platform_eng_environments.plato_dev_instance.asm/modules/asm/hub.tf line 24, in resource "google_gke_hub_membership" "membership":
│   24:       resource_link = "//container.googleapis.com/${data.google_container_cluster.asm.id}"
│     ├────────────────
│     │ data.google_container_cluster.asm.id is null
│ 
│ The expression result is null. Cannot include a null value in a string
│ template.

Any invalid for_each errors are my own doing and should be ignored. They are note likely the source of this issue.

Terraform Configuration

data "google_client_config" "provider" {}
                                                                         
data "google_project" "project" {            
  project_id = var.project_id                    
}                                                                        
                                                                         
# Core GKE regional cluster       
module "gke" {                                                                                                                                     
  source  = "terraform-google-modules/kubernetes-engine/google//modules/beta-private-cluster-update-variant"
  version = "20.0.0"                                                     
  project_id = var.project_id                                                                                                                                                                                                                                                                
  name       = local.resource_name                                                                                                                                                                                                                                                                     
  region     = var.region                                                                                                                                                                                                                                                                              
                                                                                                                                                                                                                                                                                                       
  cluster_resource_labels = { "mesh_id" : "proj-${data.google_project.project.number}" }                                                                                                                                                                                                               
                                                                                                                                                                                                                                                                                                       
  network           = module.vpc.network_name                                                                                                                                                                                                                                                          
  subnetwork        = module.vpc.subnets_names[0]                                                                                                                                                                                                                                                      
  ip_range_pods     = local.pod_subnet_range                                                                                                                                                                                                                                                           
  ip_range_services = local.svc_subnet_range                                                                                                       
                                                                         
  monitoring_service = "monitoring.googleapis.com/kubernetes"                                                                                      
  logging_service    = "logging.googleapis.com/kubernetes"
  release_channel    = "STABLE"
                                                                         
  network_policy             = true                           
  horizontal_pod_autoscaling = true   
  remove_default_node_pool   = true
                                                                         
  enable_shielded_nodes        = true
  enable_private_nodes         = true         
  enable_private_endpoint      = false             
  master_ipv4_cidr_block       = var.apiserver_cidr                                                                                                
  master_global_access_enabled = true                                                                                                           
  create_service_account = false      
  node_pools = [
    {                                                                                                                                              
      name               = "default-node-pool"                           
      machine_type       = "e2-standard-4"                    
      min_count          = 1                                  
      max_count          = 4          
      disk_size_gb       = 100                                           
      disk_type          = "pd-standard"                                                                                                           
      image_type         = "COS_CONTAINERD"
      auto_upgrade       = true                                      
      service_account    = module.gke_node_service_account.email
      enable_secure_boot = true       
      preemptible        = false
    },                                                                                                                                             
  ]
                                                                       
}   
                                                                                                                                                   
provider "kubernetes" {                                    
  host                   = "https://${module.gke.endpoint}"                                                                                        
  token                  = data.google_client_config.provider.access_token
  cluster_ca_certificate = base64decode(module.gke.ca_certificate)                                                                                 
}      
                                                                                                                                                   
module "asm" {                                                                                                                                     
  source = "terraform-google-modules/kubernetes-engine/google//modules/asm"
  version = "21.0.0"
                                                                         
  project_id                = var.project_id 
  cluster_name              = module.gke.name    
  cluster_location          = module.gke.location                        
  enable_fleet_registration = true
  enable_mesh_feature       = true
                                                                                                                                                   
  # This explicit dependency was added after seeing errors. I experience similar issues with or without it.                                            
  depends_on = [                                                         
    module.gke,              
  ]                               
}                        

### Terraform Version

```sh
> terraform version
Terraform v1.1.2
on linux_amd64

Your version of Terraform is out of date! The latest version
is 1.2.1. You can update by downloading from https://www.terraform.io/downloads.html

Additional information

No response

[bharathkkb]

Hi @jwtracy
Can you try with v21 of the GKE module which has a fix for this #1189

[jwtracy]

Hi @bharathkkb, I'll give this a try and report back. Should I leave the explicit dependency?

[jwtracy]

Upgrading GKE to 21.0.0 seemed to get me through it though I'm in a predicament where I'm on gke 20.0.0 in one environment and asm 20.0.0 stil has this issue. gke 20.0.0 to 21.0.0 upgrade seems to delete and recreate the node pool.

That's my problem though since these are major version releases. Thanks for the quick recommendation!