Enabling private nodes in GKE does not recreate the node pools

[mbrancato]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request.
• Please do not leave +1 or me too comments, they generate extra noise for issue followers and do not help prioritize the request.
• If you are interested in working on this issue or have submitted a pull request, please leave a comment.
• If an issue is assigned to the modular-magician user, it is either in the process of being autogenerated, or is planned to be autogenerated soon. If an issue is assigned to a user, that user is claiming responsibility for the issue. If an issue is assigned to hashibot, a community member has claimed the issue already.

Terraform Version

% terraform -v                                                              
Terraform v0.13.5
+ provider registry.terraform.io/hashicorp/google v3.52.0

Affected Resource(s)

• google_container_cluster
• google_container_node_pool

Terraform Configuration Files

Initial config looks similar to this:

resource "google_container_cluster" "default" {
  name     = "${local.prefix}-cluster"
  location = "us-central1-a"

  remove_default_node_pool    = true
  initial_node_count          = 1
  network                     = data.google_compute_network.default.name
  subnetwork                  = data.google_compute_subnetwork.default.name
  default_max_pods_per_node   = 110
  enable_binary_authorization = false
  enable_kubernetes_alpha     = false
  enable_legacy_abac          = false
  enable_shielded_nodes       = true
  logging_service             = "logging.googleapis.com/kubernetes"
  monitoring_service          = "monitoring.googleapis.com/kubernetes"

}

resource "google_container_node_pool" "default" {
  name               = "default-pool"
  location           = "us-central1-a"
  cluster            = google_container_cluster.default.name
  initial_node_count = 2
  max_pods_per_node  = 110

...

}

Modified config looks like this:

resource "google_container_cluster" "default" {
  name     = "${local.prefix}-cluster"
  location = "us-central1-a"

  remove_default_node_pool    = true
  initial_node_count          = 1
  network                     = data.google_compute_network.default.name
  subnetwork                  = data.google_compute_subnetwork.default.name
  default_max_pods_per_node   = 110
  enable_binary_authorization = false
  enable_kubernetes_alpha     = false
  enable_legacy_abac          = false
  enable_shielded_nodes       = true
  logging_service             = "logging.googleapis.com/kubernetes"
  monitoring_service          = "monitoring.googleapis.com/kubernetes"

  private_cluster_config {
    enable_private_endpoint = false
    enable_private_nodes    = true
    master_ipv4_cidr_block  = "172.16.0.0/28"
  }

}

resource "google_container_node_pool" "default" {
  name               = "default-pool"
  location           = "us-central1-a"
  cluster            = google_container_cluster.default.name
  initial_node_count = 2
  max_pods_per_node  = 110

...

}

Debug Output

Panic Output

Expected Behavior

Terraform plan should warn about deleting the cluster and node pool.
Terraform apply should delete and recreate both the cluster and node pool.

Actual Behavior

Terraform does not delete the node pool, but does delete the cluster (which, in effect, deletes the node pool).
Terrraform creates the replacement cluster, but it has no node pools.

Steps to Reproduce

1. Starting with the initial config - terraform apply
2. Modify config to include private nodes - terraform apply
3. Run terraform apply one more time and node pools will finally be created

Important Factoids

When performing terraform apply, my plan output showed this: (snippet)

...

      + private_cluster_config {
          + enable_private_nodes   = true # forces replacement
          + master_ipv4_cidr_block = "172.16.0.0/28" # forces replacement
          + peering_name           = (known after apply)
          + private_endpoint       = (known after apply)
          + public_endpoint        = (known after apply)
        }

        release_channel {
            channel = "REGULAR"
        }

        workload_identity_config {
            identity_namespace = "<my id>.svc.id.goog"
        }
    }

Plan: 1 to add, 0 to change, 1 to destroy.

...

The replacement cluster now has no node pools, but a node pool is defined in Terraform:

Running a terraform plan or terraform apply will perform a refresh and show that the node pool is missing.

References

[edwardmedia]

@mbrancato don't you need to add ip_allocation_policy block in your code when you try to add private_cluster_config?

[mbrancato]

@mbrancato don't you need to add ip_allocation_policy block in your code when you try to add private_cluster_config?

@edwardmedia I provided a very slim example. In reality, I do have an ip_allocation_policy block in the cluster resource.

[edwardmedia]

@mbrancato this is the same behavior as this due to the limitation of Terraform Core. It is not related only to private_cluster_config. As long as you trigger recreation of the container_cluster (ie update network), you will see the same behavior. We can't resolve this at the provider level. Does this make sense to you?

[edwardmedia]

@mbrancato closing this then

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. If you feel I made an error 🤖 🙉 , please reach out to my human friends 👉 hashibot-feedback@hashicorp.com. Thanks!