Error creating backend service

[yosssi]

I got the following error when I try to create a backend service:

google_compute_backend_service.my-backend-service: Creating...
  backend.#:                               "" => "1"
  backend.499478644.balancing_mode:        "" => "UTILIZATION"
  backend.499478644.capacity_scaler:       "" => "1"
  backend.499478644.description:           "" => ""
  backend.499478644.group:                 "" => "https://www.googleapis.com/replicapool/v1beta2/projects/myproject-dev/zones/asia-east1-a/instanceGroupManagers/gke-myproject-api-27da66d4-group"
  backend.499478644.max_rate:              "" => ""
  backend.499478644.max_rate_per_instance: "" => ""
  backend.499478644.max_utilization:       "" => "0.8"
  description:                             "" => "xxxx"
  fingerprint:                             "" => "<computed>"
  health_checks.#:                         "" => "1"
  health_checks.2686024285:                "" => "https://www.googleapis.com/compute/v1/projects/myproject-dev/global/httpHealthChecks/myi-http-basic-check"
  name:                                    "" => "my-backend-service"
  port_name:                               "" => "http"
  protocol:                                "" => "HTTP"
  self_link:                               "" => "<computed>"
  timeout_sec:                             "" => "<computed>"

Error applying plan:

1 error(s) occurred:

* google_compute_backend_service.my-backend-service: Error creating backend service: googleapi: Error 400: Invalid value for field 'resource.backends[0]': ''.  Invalid group url, invalid

Terraform does not automatically rollback in the face of errors.
Instead, your Terraform state file has been partially updated with
any resources that successfully completed. Please address the error
above and apply again to incrementally change your infrastructure.

Following is my template:

resource "google_compute_backend_service" "my-backend-service" {
    name = "my-backend-service"
    description = "xxxx"
    port_name = "http"
    protocol = "HTTP"
    backend {
      group = "${element(google_container_cluster.my_cluster.instance_group_urls, 1)}"
    }
    health_checks = ["${google_compute_http_health_check.my-http-basic-check.self_link}"]
}

I don't know why this kind of error occurs and how to fix it.

What is the reason of this error and how can I fix it?

Thanks.

[phinze]

I'm not familiar offhand with this API, but perhaps @lwander can shed some light on this?

[lwander]

@yosssi, this looks like a bug with the Google API. The instance_group_url field should be exporting the underlying instance groups' URLs, instead it seems to providing the URLs of the instance group managers, which is why this call is failing. I will file a bug internally to address this.

[yosssi]

@lwander Thanks for your investigation. I understand. I hope this bug will be fixed.

[dcgoss]

@lwander any updates on this?

[lwander]

No, nothing yet. I'm hesitant to fix this on the Terraform end, since it will only break again once the API is fixed.

[dcgoss]

@lwander Makes sense, thanks for the quick response!
@yosssi The solution/hack below may help you, it worked for me.
For the time being, I just used the instance group API docs to hardcode in the value.
If you inspect the .tfstate file, you will see in the container cluster section attributes that look like this:

"instance_group_urls.#": "1",
"instance_group_urls.0": "https://www.googleapis.com/replicapool/v1beta2/projects/your_project_name-here/zones/your_zone_here/instanceGroupManagers/gke-your_cluster_name_here-hash_key_here-group",

If you take that last part of the url: gke-your_cluster_name_here-hash_key_here-group, you can put it into the correct API url form specified in the instance group API docs:

I had to put this in parentheses so the link wouldn't highlight -
(https://www.) googleapis.com/compute/v1/projects/project/zones/your_zone_here/instanceGroups/gke-your_cluster_name_here-your_hash_key_here-group

Then you can hardcode in the group value:

backend {
        group = "https://www.googleapis.com/compute/v1/projects/exampleproject/zones/us-central1-b/instanceGroups/gke-examplecluster-hdja738s-group"
        balancing_mode = "UTILIZATION"
        max_utilization = 0.8
    }

[mihirat]

@dcgoss
Although its just a workaround, but I've got a simple solution.
The instance_group_url returns
https://container.googleapis.com/v1/projects/myproject-dev/zones/asia-east1-a/instanceGroupManagers/gke-myproject-blahblah-group
and what we want is
https://container.googleapis.com/v1/projects/myproject-dev/zones/asia-east1-a/instanceGroups/gke-myproject-blahblah-group

Thus, just replacing "instanceGroupManagers" with "instanceGroups", it works.
In this workaround,

    backend {
      group = "${replace(element(google_container_cluster.my_cluster.instance_group_urls, 1), "Manager", "")}"
    }

That's it.

[paddycarver]

@lwander would a possible way forward on this issue be to examine the URL we're getting back from the API, attempt to parse it according to the known format (https://container.googleapis.com/v1/projects/myproject-dev/zones/asia-east1-a/instanceGroupManagers/gke-myproject-blahblah-group) and see if it uses instanceGroupManagers? If it does, then we change it to use instanceGroups. If it fails to parse, or doesn't use instanceGroupManagers, we don't do any special processing. That way it should fix things for now, so people don't have to implement workarounds in their config files, and when the issue is fixed properly, the workaround won't break anything. What do you think?

[lwander]

Hey @paddyforan, rather than do string substitution on matching URLs it would be neater to lookup the returned instance group manager to read which instance group it's managing, and then supply that. Depending on naming conventions feels brittle.

[paddycarver]

Hey @lwander, you're right, that does feel more robust, though it comes at the cost of an extra network request. In this case, I feel like the network request is worthwhile, even though we still need to rely on naming conventions to detect that the API has not been fixed, unless I'm missing something.

I'll try to get a test case for this that fails, then get it fixed. Can't promise an ETA on the PR (I'm still in catch-up mode after the holiday) but hopefully I'll get to this soon. If anyone beats me to the punch, I'm happy to review.

[lwander]

Sounds great, thanks @paddyforan

[paddycarver]

I have, at long last, opened a PR with the change. Apologies for the delay. I tagged you for a review, @lwander, as you seemed the most knowledgeable about it, but if you don't have time, there are others that can review. :)

[paddycarver]

PR just landed in master. This should be fixed in our next release. Thanks for the patience, and for reporting!

[dinvlad]

@paddycarver have you by any chance asked Google folks to solve this issue upstream? would be curious to join in on the conversation

[paddycarver]

I, unfortunately, don't quite recall--it's not showing up in my https://issuetracker.google.com history, which means if I did, I did so through an informal back-channel and don't have a link, anyways. :( If you're interested in it, I'd definitely suggest opening an issue on https://issuetracker.google.com!

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.