Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

terraform plan always finds a disk device_name diff #574

Closed
dghubble opened this issue Oct 11, 2017 · 11 comments · Fixed by poseidon/typhoon#49
Closed

terraform plan always finds a disk device_name diff #574

dghubble opened this issue Oct 11, 2017 · 11 comments · Fixed by poseidon/typhoon#49
Assignees
Labels
forward/review In review; remove label to forward service/compute-instances

Comments

@dghubble
Copy link

dghubble commented Oct 11, 2017

Existing google_compute_instance_template resources recently started being recreated on every run of terraform apply. This is quite dangerous in its own right. Even performing the apply, plan still shows the same desired diff.

Terraform Version

v0.10.4

Affected Resource(s)

  • google_compute_instance_template

Terraform Configuration Files

resource "google_compute_instance_template" "controller" {
  ...
  disk {
    auto_delete  = true
    boot         = true
    source_image = "${var.os_image}"
    disk_size_gb = "${var.disk_size}"
  }
...

Example resource affected: https://github.com/poseidon/typhoon/blob/a5290dac3284ee4f1101e55a925dfd90c1e09809/google-cloud/container-linux/kubernetes/controllers/controllers.tf#L39

Actual Behavior

$ terraform plan
...
disk.0.device_name:                         "persistent-disk-0" => "" (forces new resource)

Running terraform apply (just for debugging) and then re-running terraform plan shows the same suggested diff.

Expected Behavior

terraform plan should not show a diff.

References

@dgolja
Copy link

dgolja commented Oct 11, 2017

We have the same issue. Basically rerunning the plan tries to recreate all the IT.

@dghubble
Copy link
Author

dghubble commented Oct 11, 2017

I see this behavior even using old plugins, which is surprising at first. But looking through old versions, the device_name field is not new, its been there all along, likely defaulting to "". I suspect the actual API has changed recently to start populating that value (seems innocent right, going from "" to a default value) and terraform is diligently trying to "fix" all existing instance groups.

I managed to wipe gce clusters because of this (we happen to prepare for this and failed over to another cloud), not noticing that suddenly a new change was proposed that would delete instance groups.

The change in #566 looks like it may be a solution, to address the API change, but its not available in a release that gets picked up by terraform init. Maybe a local build can be used.

@dgolja
Copy link

dgolja commented Oct 12, 2017

Hmmm in our case only the IT get recreated and IG gets only updates, so this shouldn't be a destructive action. Despite that it's annoying because of how GCP deals with old IT and the whole rolling update dance.

@dghubble
Copy link
Author

Yeah, only the instance_template resource itself is destroyed and recreated. The rest depends on usage, in my case the instance group is updated and each instance is replaced, which wasn't expected.

Workaround to avoid recreating the template is to explicitly set the device_name to "persistent-disk-0".

This seems to be an actual API change and this plugin has already made the change to start computing the name, which is about all that can be done in this source code.

dghubble added a commit to poseidon/typhoon that referenced this issue Oct 12, 2017
* Terraform provider "google" plugin releases leave the disk
device_name as "" by default. Recently the API has started to
set a default name "persistent-disk-0". Plan and apply show
all instance groups need to be recreated to "fix" the name
* Impact: Controller and worker instance groups are deleted
and recreated, deleting data on controllers and bringing
down clusters
* Fix: Explicitly set the disk_name to persistent-disk-0 so
that terraform finds no diff needs to be applied.
* #34
* hashicorp/terraform-provider-google#574
dghubble-robot pushed a commit to poseidon/terraform-google-kubernetes that referenced this issue Oct 12, 2017
* Terraform provider "google" plugin releases leave the disk
device_name as "" by default. Recently the API has started to
set a default name "persistent-disk-0". Plan and apply show
all instance groups need to be recreated to "fix" the name
* Impact: Controller and worker instance groups are deleted
and recreated, deleting data on controllers and bringing
down clusters
* Fix: Explicitly set the disk_name to persistent-disk-0 so
that terraform finds no diff needs to be applied.
* poseidon/typhoon#34
* hashicorp/terraform-provider-google#574
@dghubble
Copy link
Author

I'm content to close this. Its unfortunate. But the plugin already seems to have made the change that will be needed.

@dgolja
Copy link

dgolja commented Oct 12, 2017

What if you don't use terraform-google-kubernetes ? Could you leave it open until at least they release the #566 ?

@dghubble
Copy link
Author

I suppose it'd be nice if someone could clarify the "breaking API change" mentioned in #566 and the above analysis or maybe nudge along a release.

I don't think a code change or fix to this project is needed so feel free to close again.

@paddycarver
Copy link
Contributor

Essentially what happened (as far as I can tell) is that the API updated to return a default device_name when one is not set. Previously, we were expecting whatever we set to get returned, including if we set nothing, because that was the old behavior. #566 fixed this issue within hours of the API change occurring, but we needed to prepare the release, which unfortunately takes some time.

We've just released 1.1.0, which should solve this problem. I'm going to close this issue, as the fix is released. If you've updated to 1.1.0 and are still having trouble, please don't hesitate to open a new issue or reply here.

@dghubble
Copy link
Author

Thanks!

@dgolja
Copy link

dgolja commented Oct 15, 2017

thank you for the quick resolution.

dghubble added a commit to poseidon/typhoon that referenced this issue Nov 4, 2017
* Require google provider plugin 1.1 or higher which includes fix:
hashicorp/terraform-provider-google#574
* Remove workaround which statically set the persistent disk name
* Original reasons for workaround in a97df83 or GH #34
dghubble-robot pushed a commit to poseidon/terraform-google-kubernetes that referenced this issue Nov 4, 2017
* Require google provider plugin 1.1 or higher which includes fix:
hashicorp/terraform-provider-google#574
* Remove workaround which statically set the persistent disk name
* Original reasons for workaround in a97df839 or GH #34
justaugustus pushed a commit to justaugustus/typhoon-azure that referenced this issue Feb 12, 2018
* Terraform provider "google" plugin releases leave the disk
device_name as "" by default. Recently the API has started to
set a default name "persistent-disk-0". Plan and apply show
all instance groups need to be recreated to "fix" the name
* Impact: Controller and worker instance groups are deleted
and recreated, deleting data on controllers and bringing
down clusters
* Fix: Explicitly set the disk_name to persistent-disk-0 so
that terraform finds no diff needs to be applied.
* poseidon#34
* hashicorp/terraform-provider-google#574
justaugustus pushed a commit to justaugustus/typhoon-azure that referenced this issue Feb 12, 2018
* Require google provider plugin 1.1 or higher which includes fix:
hashicorp/terraform-provider-google#574
* Remove workaround which statically set the persistent disk name
* Original reasons for workaround in a97df83 or GH poseidon#34
@ghost
Copy link

ghost commented Mar 30, 2020

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. If you feel I made an error 🤖 🙉 , please reach out to my human friends 👉 hashibot-feedback@hashicorp.com. Thanks!

@ghost ghost locked and limited conversation to collaborators Mar 30, 2020
@github-actions github-actions bot added forward/review In review; remove label to forward service/compute-instances labels Jan 15, 2025
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
forward/review In review; remove label to forward service/compute-instances
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants