Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Upgrading cluster master after cluster creation completes #3385

Closed
yellowmegaman opened this issue Apr 5, 2019 · 21 comments
Closed

Upgrading cluster master after cluster creation completes #3385

yellowmegaman opened this issue Apr 5, 2019 · 21 comments
Labels
bug forward/review In review; remove label to forward service/container

Comments

@yellowmegaman
Copy link

Community Note

  • Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
  • Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
  • If you are interested in working on this issue or have submitted a pull request, please leave a comment. If the issue is assigned to the "modular-magician" user, it is either in the process of being autogenerated, or is planned to be autogenerated soon. If the issue is assigned to a user, that user is claiming responsibility for the issue. If the issue is assigned to "hashibot", a community member has claimed the issue already.

Description

Currently when creating google_container_cluster the recommended way (separate node_pool), terraform signals that everything is OK, but after only a few moments, cluster is doing upgrade that is not configurable by gke maint window.

Upgrading cluster master
The values shown below are going to change soon.

So if I'm using terraform to create GKE cluster I can't be sure that the moment terraform is done, everything is OK and proceed with some automation.

Idea: add additional (configurable) timeout, after which terraform checks if cluster is available.
Currently even endpoint is unavail when performing upgrades.

New or Affected Resource(s)

  • google_container_cluster

Potential Terraform Configuration

Any recommended configuration from https://www.terraform.io/docs/providers/google/r/container_cluster.html

References

@ghost ghost added the enhancement label Apr 5, 2019
@rileykarson
Copy link
Collaborator

Are you able to consistently reproduce this with a config you can share? We run a pretty extensive # of clusters in our CI environment an this has never come up.

@yellowmegaman
Copy link
Author

@rileykarson
Here you go, almost default config, but without certificates/legacy endpoints, embracing RBAC.

resource "google_container_cluster" "dev" {
  name                     = "dev"
  location                 = "europe-west4-b"
  remove_default_node_pool = true
  initial_node_count       = 1
  min_master_version       = "1.12.6-gke.7"
  enable_legacy_abac       = false
  maintenance_policy {
    daily_maintenance_window {
      start_time = "01:00"
    }
  }
  master_auth {
    username = ""
    password = ""
    client_certificate_config {
      issue_client_certificate = false
    }
  }
}
resource "google_container_node_pool" "dev-n1s2-pool" {
  name               = "dev"
  location           = "europe-west4-b"
  cluster            = "${google_container_cluster.dev.name}"
  initial_node_count = "7"
  autoscaling {
    min_node_count   = "4"
    max_node_count   = "14"
  }
  management {
    auto_repair  = "true"
  }
  version        = "1.12.6-gke.7"
  node_config {
    disk_type    = "pd-ssd"
    disk_size_gb = "30"
    metadata {
      disable-legacy-endpoints = "true"
    }
    preemptible  = "false"
    machine_type = "n1-standard-2"
    oauth_scopes = [ "https://www.googleapis.com/auth/compute", "https://www.googleapis.com/auth/devstorage.read_only", "https://www.googleapis.com/auth/logging.write", "https://www.googleapis.com/auth/monitoring" ]
    tags         = ["ssh-wan"]
  }
}

After cluster creation complete i always get master node update.

P.S. version 1.12.5 - same, and 1.11 too.

@ghost ghost removed the waiting-response label Apr 7, 2019
@thefirstofthe300
Copy link

@yellowmegaman I suspect the issue here is that the master is resizing itself. On cluster creation, the master is generally size to only accomodate ~5 nodes. Since you're cluster has 7, the master is being upsized to accomodate those extra nodes. This autoscaling will happen regardless of what the maintenance window is set to since the autoscaling is required to maintain the health of the cluster and isn't really routine.

Note that the master currently only scales up so once the master reaches a certain size, it isn't scaled back down.

If you need an HA control plane, you should probably look into using a regional cluster. Otherwise, I suspect everything here is WAI.

@yellowmegaman
Copy link
Author

@thefirstofthe300 thanks for shedding the light! But still, I can't just wait for terraform to bring up the cluster to continue with further automation here, need to implement some kind of timeout and check, that's what troubles me.

Next best thing i can come up with - docs probably should tell about possible resizing, since if you use recommended example with, for instance, batch job with timeout, batch job will fail during cluster resize.

@ctrox
Copy link

ctrox commented Apr 9, 2019

@thefirstofthe300 I actually ran into the same thing here: #3249

If you need an HA control plane, you should probably look into using a regional cluster. Otherwise, I suspect everything here is WAI.

This is exactly what I did but the Terraform provider does not handle this gracefully. The master is working fine on the side of kubernetes but Terraform fails because it is hardcoded to not continue if the cluster is in RECONCILING state.

@danisla
Copy link
Contributor

danisla commented Apr 30, 2019

One of the workarounds I found is to set the initial_node_count to match the expected number of nodes in the managed node pools. This way, the master is already right-sized after the default node pool is deleted and the managed pools are created so an upgrade operation is not triggered after completion.

Keep in mind when calculating initial_node_count with regional clusters that this value is per zone, so setting it to a value like 2 in a region with 3 zones creates 6 nodes.

@binamov
Copy link
Contributor

binamov commented Apr 30, 2019 via email

@ctrox
Copy link

ctrox commented May 2, 2019

Thanks @danisla for that workaround. I got it to work by just setting the initial_node_count to whatever I want the final node count to be and set remove_default_node_pool to true. With that the masters are sized correctly from the beginning and deleting the default node pool won't cause the masters to scale down. I'm creating the custom node pools just after that is done (same tf apply) so I'm not sure if a longer wait in between would cause the masters to scale down again.

@singh-ajeet
Copy link

I am also facing this issue. I crated a GKE cluster with below command, and after increasing the load (using JMeter) on my deployed service cluster is doing upgrade of master.
gcloud container clusters create ajeet-gke --zone us-east4-b --node-locations us-east4-b --machine-type n1-standard-8 --num-nodes 1 --enable-autoscaling --min-nodes 4 --max-nodes 16

@naihsi
Copy link

naihsi commented Dec 18, 2019

Hi, I also encountered this issue. my case is also to add two additional nodepools, and it upgrade automatically to block the followed deployment.

@venkykuberan venkykuberan self-assigned this Jan 17, 2020
@venkykuberan
Copy link
Contributor

I am also facing this issue. I crated a GKE cluster with below command, and after increasing the load (using JMeter) on my deployed service cluster is doing upgrade of master.
gcloud container clusters create ajeet-gke --zone us-east4-b --node-locations us-east4-b --machine-type n1-standard-8 --num-nodes 1 --enable-autoscaling --min-nodes 4 --max-nodes 16

@singh-ajeet looks like you trying in gcloud which is not in scope for this channel. Please raise the issue against glcoud team if you still have the issue

@venkykuberan
Copy link
Contributor

@yellowmegaman Please let us know if you still face this issue or you want me to close it.
Also I am able to successfully create the GKE cluster with your config. Cluster completes within the timeout period after that don't see master getting upgraded.

Terraform v0.12.18
+ provider.google v3.4.0
+ provider.google-beta v3.3.0

@yellowmegaman
Copy link
Author

@venkykuberan just tried code above again. Had only to edit metadata field, add '=' to make it work with 0.12.X.

All is just the same. After cluster and nodepool are created, wait for few mins. In my case it was 2 minutes when cluster started to upgrade:

image

Kubectl returning this:

The connection to the server 34.90.187.152 was refused - did you specify the right host or port?

So if I don't use null resource to wait some time and check connectivity to cluster endpoint, terraform will fail to apply resources to cluster.

For anyone interested, currently I'm using this code to bypass this issue, not ideal one, may break on larger nodepools due longer upgrade times:

#%# wait/check
resource "null_resource" "placeholder-wait-for-k8s-sleep" {
  depends_on = [module.placeholder-node-pool]
  provisioner "local-exec" {
    command = "echo 'sleep started'; sleep 180"
  }
}
resource "null_resource" "placeholder-wait-for-k8s" {
  depends_on = [null_resource.placeholder-wait-for-k8s-sleep]
  provisioner "local-exec" {
    command = "until nc -w1 -z ${module.placeholder.gke-cluster.endpoint} 443; do sleep 1; done && echo 'cluster seems to be ready'"
  }
}
#%# wait/check

And so you can add other k8s-related modules here with depends_on = [null_resource.placeholder-wait-for-k8s]

@ghost ghost removed the waiting-response label Jan 19, 2020
@yellowmegaman
Copy link
Author

@venkykuberan just tried code above again. Had only to edit metadata field, add '=' to make it work with 0.12.X.

All is just the same. After cluster and nodepool are created, wait for few mins. In my case it was 2 minutes when cluster started to upgrade:

image

Kubectl returning this:

The connection to the server 34.90.187.152 was refused - did you specify the right host or port?

So if I don't use null resource to wait some time and check connectivity to cluster endpoint, terraform will fail to apply resources to cluster.

For anyone interested, currently I'm using this code to bypass this issue, not ideal one, may break on larger nodepools due longer upgrade times:

#%# wait/check
resource "null_resource" "placeholder-wait-for-k8s-sleep" {
  depends_on = [module.placeholder-node-pool]
  provisioner "local-exec" {
    command = "echo 'sleep started'; sleep 180"
  }
}
resource "null_resource" "placeholder-wait-for-k8s" {
  depends_on = [null_resource.placeholder-wait-for-k8s-sleep]
  provisioner "local-exec" {
    command = "until nc -w1 -z ${module.placeholder.gke-cluster.endpoint} 443; do sleep 1; done && echo 'cluster seems to be ready'"
  }
}
#%# wait/check

And so you can add other k8s-related modules here with depends_on = [null_resource.placeholder-wait-for-k8s]

@venkykuberan
Copy link
Contributor

@yellowmegaman could you try initial_node_count = 7 on your cluster config and see you can avoid the upgrade as with that config, state was Running for me all the time (tried for about 5 mins) when i hit cluster from gcloud after the cluster creation is complete from terraform.

However coming to the core issue, Once Terraform gets the status as cluster complete it will close the loop and it will not have any visibility about what GKE is doing on the cluster in background until next refresh call finds any difference. If GKE modifies the cluster outside of the maintenance window and if it affects your automation flow an issue can be raised directly against GKE here.

Since you already have work-around in-place. Shall i go head and close this issue ?

@thefirstofthe300
Copy link

thefirstofthe300 commented Jan 22, 2020 via email

@ghost ghost removed the waiting-response label Jan 22, 2020
@yellowmegaman
Copy link
Author

@venkykuberan Don't see any reason to try with initial node count = 7, since all users have different scenarios.
Actually same for workaround, it's working, but only for this size/count, for larger pools it may not work, at least the sleep limit.

I totally understand that this is happening outside terraform interaction loop, but was hoping to bring this issue to attention of someone developing GCP provider.

We can close it, since it isn't terraform fault, but then we'll have to admit that:
"Terraform can't consistently deploy both k8s cluster and workload on top of it."

And is hugely sad.

@venkykuberan
Copy link
Contributor

@yellowmegaman i didn't mean to use constant value 7 for the node count. I wanted to try the same node_count for default node pool as well, so that master is sized correctly in the initial creation time (doesn't have to resize it later).

Also please let us know
is terraform successfully creating the cluster and exiting or timing out ?

steved added a commit to dominodatalab/terraform-gcp-gke that referenced this issue Feb 11, 2020
After node pools are added, the cluster begins to scale-up and can cause
inaccessibility to the k8s master URL.

Workaround from
hashicorp/terraform-provider-google#3385
steved added a commit to dominodatalab/terraform-gcp-gke that referenced this issue Feb 11, 2020
After node pools are added, the cluster begins to scale-up and can cause
inaccessibility to the k8s master URL.

Workaround from
hashicorp/terraform-provider-google#3385
@venkykuberan
Copy link
Contributor

@yellowmegaman do you still want to keep the issue open ? or shall i close if you aren't looking anything from us here?

@roaks3 roaks3 added the forward/review In review; remove label to forward label Aug 17, 2023
@edwardmedia
Copy link
Contributor

No response. Assuming it is no longer an issue

@github-actions
Copy link

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

@github-actions github-actions bot locked as resolved and limited conversation to collaborators Oct 21, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
bug forward/review In review; remove label to forward service/container
Projects
None yet
Development

No branches or pull requests