Enable auto-upgrade in beta clusters with a release channel

[dpetersen]

I have been testing some changes with zonal clusters, but receive this error:

Error: error creating NodePool: googleapi: Error 400: Auto_upgrade cannot be false when release_channel REGULAR is set., badRequest

The node pools I define do have autoupgrade enabled, but I think the default node pool (which I am deleting with remove_default_node_pool anyway) is the problem node pool. Regional clusters already have auto-upgrade enabled by default, so I'm guessing nobody has used the specific combination of a zonal cluster and release channels with this module yet.

This is noted in the documentation as a disallowed configuration:

Note: Node auto-upgrade and auto-repair are enabled and cannot be disabled when using release channels.

It's complicated slightly because release channels still only exist in the beta Google Terraform provider, as far as I can tell. I have left the non-beta configuration untouched and added an if beta_cluster block for my changes.

[comment-bot-dev]

Thanks for the PR! 🚀
✅ Lint checks have passed.

[morgante]

Thanks for the contribution, I'm not quite sure what's happening with tests. @bharathkkb Any chance this error is connected to your bundling work?

terraform_validate ./modules/safer-cluster 
Error verifying checksum for provider "kubernetes"
The checksum for provider distribution from the Terraform Registry
did not match the source. This may mean that the distributed files
were changed after this version was released to the Registry.
Error: 
unable to verify checksum

[bharathkkb]

@morgante this is actually what bundling aims to resolve. I will finish that up this week. This happens once in a while because we download all providers once per example, modules and fixtures.

[bharathkkb]

/gcbrun

[bharathkkb]

Hi @dpetersen thanks for the PR!
You will need to run make build to propagate these changes from /autogen to the modules.

The node pools I define do have autoupgrade enabled

At what point of cluster provisioning does this error occur, i.e during creation of google_container_cluster.primary or google_container_node_pool.pools. This can help us narrow down.

[dpetersen]

Yes, I have a separate branch where I am checking in the result of make build. I have a couple of PRs that I am combining at the moment. Should I be running make build and committing that in my PR, or do you folks do that when you build the module releases?

Here's a minimal example of what fails, and how:

module "prtest" {                                                                            
  source  = "terraform-google-modules/kubernetes-engine/google//modules/beta-private-cluster"
  version = "11.1.0"                                                                         
                                                                                             
  project_id             = module.gcp-project.project_id                                     
  name                   = "prtest"                                                          
  network                = module.gke-network.vpc_network.name                               
  subnetwork             = module.gke-network.vpc_subnetworks["prtest"].subnetwork.name      
  master_ipv4_cidr_block = module.gke-network.vpc_subnetworks["prtest"].master_cidr_block    
  ip_range_pods          = module.gke-network.vpc_subnetworks["prtest"].pods_range_name      
  ip_range_services      = module.gke-network.vpc_subnetworks["prtest"].services_range_name  
                                                                                             
  regional = false                                                                           
  region   = "us-central1"                                                                   
  zones    = ["us-central1-a", "us-central1-b", "us-central1-c"]                             
                                                                                             
  release_channel = "REGULAR"                                                                
}                                                                                            `

module.prtest.google_container_cluster.primary: Creating...
module.prtest.google_container_cluster.primary: Still creating... [10s elapsed]
module.prtest.google_container_cluster.primary: Still creating... [20s elapsed]
module.prtest.google_container_cluster.primary: Still creating... [30s elapsed]
module.prtest.google_container_cluster.primary: Still creating... [40s elapsed]
module.prtest.google_container_cluster.primary: Still creating... [50s elapsed]
module.prtest.google_container_cluster.primary: Still creating... [1m0s elapsed]
module.prtest.google_container_cluster.primary: Still creating... [1m10s elapsed]
module.prtest.google_container_cluster.primary: Still creating... [1m20s elapsed]
module.prtest.google_container_cluster.primary: Still creating... [1m30s elapsed]
module.prtest.google_container_cluster.primary: Still creating... [1m40s elapsed]
module.prtest.google_container_cluster.primary: Still creating... [1m50s elapsed]
module.prtest.google_container_cluster.primary: Still creating... [2m0s elapsed]
module.prtest.google_container_cluster.primary: Still creating... [2m10s elapsed]
module.prtest.google_container_cluster.primary: Still creating... [2m20s elapsed]
module.prtest.google_container_cluster.primary: Creation complete after 2m26s [id=projects/content-don-5ea5/locations/us-central1-a/clusters/prtest]
module.prtest.google_container_node_pool.pools["default-node-pool"]: Creating...

Error: error creating NodePool: googleapi: Error 400: Auto_upgrade cannot be false when release_channel REGULAR is set., badRequest

  on .terraform/modules/prtest/modules/beta-private-cluster/cluster.tf line 278, in resource "google_container_node_pool" "pools":
 278: resource "google_container_node_pool" "pools" {

The key here is that it's a zonal cluster. I'm fairly sure that a regional cluster will succeed, because of the logic around the default behavior for auto-upgrading. If I point that original code at my branch instead of the 11.1.0 version of this module, it succeeds.

[bharathkkb]

Should I be running make build and committing that in my PR

yeah, those should be committed

module.prtest.google_container_cluster.primary: Still creating... [2m20s elapsed]
module.prtest.google_container_cluster.primary: Creation complete after 2m26s [id=projects/content-don-5ea5/locations/us-central1-a/clusters/prtest]
module.prtest.google_container_node_pool.pools["default-node-pool"]: Creating...

This is interesting, this means that the google_container_cluster has actually successfully spun up (including deleting default pool via remove_default_node_pool) and then moves on to creating your nodepool. This nodepool is the "default" one that is created by this module, although it is managed via a google_container_node_pool. The default_auto_upgrade actually controls the auto_upgrade var in the google_container_node_pool.

Overall this change makes sense to me.

[dpetersen]

Got it, I guess I was confused earlier about which node pool was causing the problem. I have no committed the built files, as well. Thanks for looking at this!

[bharathkkb]

LGTM

[bharathkkb]

/gcbrun