Can't Add GPU Accelerator to Node Pool part of Regional Cluster

[DanielWozniak]

I currently have a regional cluster. For this cluster I would like to create a new node pool which contains a GPU accelerator. The problem with this is that in the google documentation it states:

For regional clusters that run GPUs, there is currently no region which has any GPU type in three zones. If you want to run GPUs in a regional cluster, you need to specify zones using the --node-locations flag

When I specify the location in the module like so

node_pools = [
    {
      name              = "node-pool-name"
      machine_type      = "n1-highmem-32"
      min_count         = 0
      max_count         = 1
      disk_size_gb      = 100
      disk_type         = "pd-standard"
      image_type        = "COS"
      accelerator_count = 2
      accelerator_type  = "nvidia-tesla-p100"
      auto_repair       = true
      auto_upgrade      = true
      preemptible       = false
      location          = "us-east1-b"
    },
  ]

For some reason the location is not getting picked up and in the plan we get:

      + location            = "us-east1"
      + max_pods_per_node   = (known after apply)
      + name                = "node-pool-name"
      + name_prefix         = (known after apply)
      + node_count          = (known after apply)
      + project             = "project-name"
      + region              = (known after apply)
      + version             = (known after apply)
      + zone                = (known after apply)

Even after running the apply it still doesn't get picked up and the following error gets thrown:

module.gke.google_container_node_pool.pools[1]: Creating...

Error: error creating NodePool: googleapi: Error 400: Accelerator type "nvidia-tesla-p100" does not exist in zone us-east1-d., badRequest

Would this be a bug in the module? Or is there something that I might be missing?

[DanielWozniak]

Went over the code and looks like the problem is that node pool doesn't accept a location. What ever location is specified in the cluster is used directly in the node pool. I've created a branch with some changes but while testing I can't seem to even get the cluster to create using the master branch. For some reason the cluster gets created with an error code 13 with the message INTERNAL(not very helpful), then proceeds to deleting the cluster.

[morgante]

Can you share the docs you're referring to? If you're able to get this working even via gcloud that would help.

[rileykarson]

FYI, node pool node locations (that are different than the cluster's default node locations) will be supported as part of the next provider release: GoogleCloudPlatform/magic-modules#2320

[morgante]

Interesting, I didn't know we supported the ability to run different node pools in different locations. Looks like we should be able to add this with the next provider release, thanks for the heads up!

[DanielWozniak]

Ah, makes sense. For some reason when I read the description of location in the docs, I assumed you could specify something other than the clusters location. Even though it clearly says in which the cluster resides.
Thanks, for clearing things up guys. Any idea when this might be part of this module?

[morgante]

@DanielWozniak Likely 2-3 weeks, depending on when the provider release lands.

[DanielWozniak]

Sounds good. In the meantime I suppose we can close this issue.

[DanielWozniak]

Hi All,

Since the feature has been released in version 2.16.0 could I request this be added to the module.
More specifically, being able to specify which zone is used in the node pool.

I've tried working on this myself however for some reason im not even able to create a simple cluster before even adding any changes I get a simple

Error: Error waiting for creating GKE cluster: Failed to create cluster

  on .terraform/modules/gke/cluster.tf line 22, in resource "google_container_cluster" "primary":
  22: resource "google_container_cluster" "primary" {

For some reason after 20 seconds of trying to create the cluster it just starts deleting itself, without giving any helpful message of why this is happening.

[morgante]

@DanielWozniak Is this no longer needed?

[DanielWozniak]

It's still needed, just thought that since this issue is tagged with question it might not get noticed. I was thinking of just creating a new one with all the required info to get rid of any confusion. Or if you are able to change the tag to Enhancement then we can keep this one.

[morgante]

Ok I've marked it as an Enhancement. We'll put it in our backlog.

[morgante]

I've actually moved this into #290.