Adding instance group to existing region_backend_service creates permadiff

[mlauter]

Terraform Version

Terraform v0.11.7

• provider.google v1.14.0

Affected Resource(s)

• google_compute_region_backend_service

Terraform Configuration Files

webserver module outputs

output "instance_group" {
  value = "${google_compute_instance_group.webserver.*.self_link}"
}

main

module "webserver-blue" {
  source   = "./modules/webserver"
  ...
}

module "webserver-green" {
  source   = "./modules/webserver"
  ...
}

resource "google_compute_region_backend_service" "webserver" {
  name        = "webserver"
  description = "Backend service for a webserver cluster"
  region      = "${var.region}"

  protocol                        = "TCP"
  session_affinity                = "NONE"
  timeout_sec                     = 30
  connection_draining_timeout_sec = 0

  health_checks = ["${google_compute_health_check.webserver.self_link}"]

  backend = [
    {
      group = "${module.webserver-blue.instance_group[0]}"
    },
    # Adding this group
    {
      group = "${module.webserver-green.instance_group[0]}"
    },
  ]
}

resource "google_compute_forwarding_rule" "webserver" {
  name                  = "webserver"
  load_balancing_scheme = "INTERNAL"
  ip_protocol           = "TCP"
  backend_service       = "${google_compute_region_backend_service.webserver.self_link}"
  ports                 = [80]
  subnetwork            = "${data.google_compute_subnetwork.primary.self_link}"
}

Expected Behavior

-green instance_group will be added to webserver region backend service, -blue instance_group will remain as a backend for the webserver backend service.

Actual Behavior

-green instance_group is added as a backend to the region backend service, and -blue instance_group is removed.

Furthermore, if I run terraform apply again without making any changes to the config, the reverse will happen - -green will be removed and -blue will be added back. This appears to be a permanent diff state.

Steps to Reproduce

1. Setup a region backend service with a single instance group as the backend for that service.
2. Create a new instance group.
3. Add the new instance group as a backend in the region backend service.
4. terraform apply

[ericnorris]

For what it's worth, this is an excerpt from the debug logs when this issue happens:

2018-07-08T14:46:11.499-0400 [DEBUG] plugin.terraform-provider-google_v1.15.0_x4: 2018/07/08 14:46:11 [DEBUG] Updating existing Backend Service "api-internal-us-central1-0": compute.BackendService{AffinityCookieTtlSec:0, Backends:[]*compute.Backend{(*compute.Backend)(0xc4208fe240)}, CdnPolicy:(*compute.BackendServiceCdnPolicy)(nil), ConnectionDraining:(*compute.ConnectionDraining)(nil), CreationTimestamp:"", CustomRequestHeaders:[]string(nil), Description:"redacted", EnableCDN:false, Fingerprint:"S0f3kMMXzes=", HealthChecks:[]string{"redacted"}, Iap:(*compute.BackendServiceIAP)(nil), Id:0x0, Kind:"", LoadBalancingScheme:"INTERNAL", Name:"api-internal-us-central1-0", Port:0, PortName:"", Protocol:"TCP", Region:"", SecurityPolicy:"", SelfLink:"", SessionAffinity:"NONE", TimeoutSec:30, ServerResponse:googleapi.ServerResponse{HTTPStatusCode:0, Header:http.Header(nil)}, ForceSendFields:[]string(nil), NullFields:[]string(nil)}

The Backends array appears to contain a single backend, and so every terraform apply adds the missing backend but drops the existing one.

I was able to work around this by using the gcloud compute backend-services add-backend command, but this is not ideal for us using this in a production environment.

[paddycarver]

I've been looking into this, and don't have a complete answer just yet--it appears that the underlying Terraform framework is not behaving as it should, but I can't say that with much certainty yet--but I have discovered a workaround: if the backends inside the backend_service have a description, this bug doesn't seem to trigger. It's not perfect, but it may be a better solution than shelling out to another tool. I'll keep working on this.

[UPDATE] It appears even setting the description to an empty string (instead of just omitting it) in the config fixes the issue. I have a hunch this is related to the description: "" => "" in the diff, but I'm not 100% sure on why that would happen. Tracing it a bit.

[mlauter]

thanks for digging into this @paddycarver! we'll use the description workaround for now.

[mlauter]

@paddycarver one update, it looks like you have to use empty string for the description, otherwise we see a different permadiff state of continually trying to add the description to those backends.

[paddycarver]

@mlauter do you happen to have a log for that happening? In my tests, setting the backend descriptions to "test 1" and "test 2" resolves the issue :/

I've definitely narrowed this down to how the sub block interacts with self_links, because we need it to accept self_links for both beta and v1 versions of the API. Eventually, I think we have a plan for that to go away, but I'd rather not have this weird bug lingering around until then. Unfortunately, all my attempts to resolve it have been stymied so far. I think it may require a fix in core, and given that 0.12 is on the horizon, which will change some diffing behaviour, I'm loathe to spend a bunch of time diving into the diffing behaviour trying to figure out how to get a permanent solution here. How inconvenient is the empty string description workaround? Can that band-aid hold up as we wait for 0.12 to land?

[rileykarson]

I believe this is an occurrence of hashicorp/terraform#19658, which is fixed in Terraform 0.12. There isn't anything we can do on the provider side until the new SDK is released, but our release including the SDK should fix this issue. Feel free to reopen / file a new issue if you're still encountering this after that release!

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. If you feel I made an error 🤖 🙉 , please reach out to my human friends 👉 hashibot-feedback@hashicorp.com. Thanks!