Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Unable to destroy unstable managed instance group #6126

Closed
sorenhansendk opened this issue Apr 16, 2020 · 4 comments
Closed

Unable to destroy unstable managed instance group #6126

sorenhansendk opened this issue Apr 16, 2020 · 4 comments
Assignees
@venkykuberan
Labels
bug forward/review In review; remove label to forward service/compute-managed

Comments

@sorenhansendk
Copy link
Contributor

Community Note

  • Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request.
  • Please do not leave +1 or me too comments, they generate extra noise for issue followers and do not help prioritize the request.
  • If you are interested in working on this issue or have submitted a pull request, please leave a comment.
  • If an issue is assigned to the modular-magician user, it is either in the process of being autogenerated, or is planned to be autogenerated soon. If an issue is assigned to a user, that user is claiming responsibility for the issue. If an issue is assigned to hashibot, a community member has claimed the issue already.

Terraform Version

Terraform v0.12.21

Affected Resource(s)

  • google_compute_region_instance_group_manager

Terraform Configuration Files

resource "google_compute_region_instance_group_manager" "group" {
  name               = var.group
  region             = "europe-west4"
  target_size        = var.target
  base_instance_name = var.group
  wait_for_instances = true

  version {
    name              = var.template
    instance_template = google_compute_instance_template.template.self_link
  }

  named_port {
    name = "http"
    port = 80
  }

  update_policy {
    type                         = "PROACTIVE"
    minimal_action               = "REPLACE"
    max_surge_fixed              = 3
    max_unavailable_fixed        = 0
    instance_redistribution_type = "PROACTIVE"
  }

  auto_healing_policies {
    health_check      = google_compute_health_check.healthcheck.self_link
    initial_delay_sec = 60
  }

  timeouts {
    create = "15m"
    update = "15m"
    delete = "15m"
  }
}

Debug Output

2020-04-16T08:09:40.765Z [DEBUG] plugin.terraform-provider-google_v3.13.0_x5: 2020/04/16 08:09:40 [DEBUG] Google API Response Details:
2020-04-16T08:09:40.765Z [DEBUG] plugin.terraform-provider-google_v3.13.0_x5: ---[ RESPONSE ]--------------------------------------
2020-04-16T08:09:40.765Z [DEBUG] plugin.terraform-provider-google_v3.13.0_x5: HTTP/2.0 200 OK
2020-04-16T08:09:40.765Z [DEBUG] plugin.terraform-provider-google_v3.13.0_x5: Alt-Svc: quic=":443"; ma=2592000; v="46,43",h3-Q050=":443"; ma=2592000,h3-Q049=":443"; ma=2592000,h3-Q048=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,h3-T050=":443"; ma=2592000
2020-04-16T08:09:40.765Z [DEBUG] plugin.terraform-provider-google_v3.13.0_x5: Cache-Control: private
2020-04-16T08:09:40.765Z [DEBUG] plugin.terraform-provider-google_v3.13.0_x5: Content-Type: application/json; charset=UTF-8
2020-04-16T08:09:40.765Z [DEBUG] plugin.terraform-provider-google_v3.13.0_x5: Date: Thu, 16 Apr 2020 08:09:40 GMT
2020-04-16T08:09:40.765Z [DEBUG] plugin.terraform-provider-google_v3.13.0_x5: Etag: A3DLckg4Wu8ErzOk9qqx63hrNSw=/VsMVDGNgtB_3ww9iCgb0-CDcE30=
2020-04-16T08:09:40.765Z [DEBUG] plugin.terraform-provider-google_v3.13.0_x5: Server: ESF
2020-04-16T08:09:40.765Z [DEBUG] plugin.terraform-provider-google_v3.13.0_x5: Vary: Origin
2020-04-16T08:09:40.765Z [DEBUG] plugin.terraform-provider-google_v3.13.0_x5: Vary: X-Origin
2020-04-16T08:09:40.765Z [DEBUG] plugin.terraform-provider-google_v3.13.0_x5: Vary: Referer
2020-04-16T08:09:40.765Z [DEBUG] plugin.terraform-provider-google_v3.13.0_x5: X-Content-Type-Options: nosniff
2020-04-16T08:09:40.765Z [DEBUG] plugin.terraform-provider-google_v3.13.0_x5: X-Frame-Options: SAMEORIGIN
2020-04-16T08:09:40.765Z [DEBUG] plugin.terraform-provider-google_v3.13.0_x5: X-Xss-Protection: 0
2020-04-16T08:09:40.765Z [DEBUG] plugin.terraform-provider-google_v3.13.0_x5: 
2020-04-16T08:09:40.765Z [DEBUG] plugin.terraform-provider-google_v3.13.0_x5: {
2020-04-16T08:09:40.765Z [DEBUG] plugin.terraform-provider-google_v3.13.0_x5:  "id": "3896326025922327949",
2020-04-16T08:09:40.765Z [DEBUG] plugin.terraform-provider-google_v3.13.0_x5:  "creationTimestamp": "2020-04-16T00:38:10.727-07:00",
2020-04-16T08:09:40.765Z [DEBUG] plugin.terraform-provider-google_v3.13.0_x5:  "name": "testing-30666-XXXXX",
2020-04-16T08:09:40.765Z [DEBUG] plugin.terraform-provider-google_v3.13.0_x5:  "region": "https://www.googleapis.com/compute/beta/projects/XXXXX/regions/europe-west4",
2020-04-16T08:09:40.765Z [DEBUG] plugin.terraform-provider-google_v3.13.0_x5:  "distributionPolicy": {
2020-04-16T08:09:40.765Z [DEBUG] plugin.terraform-provider-google_v3.13.0_x5:   "zones": [
2020-04-16T08:09:40.765Z [DEBUG] plugin.terraform-provider-google_v3.13.0_x5:    {
2020-04-16T08:09:40.765Z [DEBUG] plugin.terraform-provider-google_v3.13.0_x5:     "zone": "https://www.googleapis.com/compute/beta/projects/XXXXX/zones/europe-west4-b"
2020-04-16T08:09:40.765Z [DEBUG] plugin.terraform-provider-google_v3.13.0_x5:    },
2020-04-16T08:09:40.765Z [DEBUG] plugin.terraform-provider-google_v3.13.0_x5:    {
2020-04-16T08:09:40.765Z [DEBUG] plugin.terraform-provider-google_v3.13.0_x5:     "zone": "https://www.googleapis.com/compute/beta/projects/XXXXX/zones/europe-west4-c"
2020-04-16T08:09:40.765Z [DEBUG] plugin.terraform-provider-google_v3.13.0_x5:    },
2020-04-16T08:09:40.765Z [DEBUG] plugin.terraform-provider-google_v3.13.0_x5:    {
2020-04-16T08:09:40.765Z [DEBUG] plugin.terraform-provider-google_v3.13.0_x5:     "zone": "https://www.googleapis.com/compute/beta/projects/XXXXX/zones/europe-west4-a"
2020-04-16T08:09:40.765Z [DEBUG] plugin.terraform-provider-google_v3.13.0_x5:    }
2020-04-16T08:09:40.765Z [DEBUG] plugin.terraform-provider-google_v3.13.0_x5:   ]
2020-04-16T08:09:40.765Z [DEBUG] plugin.terraform-provider-google_v3.13.0_x5:  },
2020-04-16T08:09:40.765Z [DEBUG] plugin.terraform-provider-google_v3.13.0_x5:  "instanceTemplate": "https://www.googleapis.com/compute/beta/projects/XXXXX/global/instanceTemplates/testing-30666-XXXXX",
2020-04-16T08:09:40.765Z [DEBUG] plugin.terraform-provider-google_v3.13.0_x5:  "versions": [
2020-04-16T08:09:40.765Z [DEBUG] plugin.terraform-provider-google_v3.13.0_x5:   {
2020-04-16T08:09:40.765Z [DEBUG] plugin.terraform-provider-google_v3.13.0_x5:    "name": "testing-30666-XXXXX",
2020-04-16T08:09:40.765Z [DEBUG] plugin.terraform-provider-google_v3.13.0_x5:    "instanceTemplate": "https://www.googleapis.com/compute/beta/projects/XXXXX/global/instanceTemplates/testing-30666-XXXXX",
2020-04-16T08:09:40.765Z [DEBUG] plugin.terraform-provider-google_v3.13.0_x5:    "targetSize": {
2020-04-16T08:09:40.765Z [DEBUG] plugin.terraform-provider-google_v3.13.0_x5:     "calculated": 1
2020-04-16T08:09:40.765Z [DEBUG] plugin.terraform-provider-google_v3.13.0_x5:    }
2020-04-16T08:09:40.765Z [DEBUG] plugin.terraform-provider-google_v3.13.0_x5:   }
2020-04-16T08:09:40.765Z [DEBUG] plugin.terraform-provider-google_v3.13.0_x5:  ],
3.0_x5:  "instanceGroup": "https://www.googleapis.com/compute/beta/projects/XXXXX/regions/europe-west4/instanceGroups/testing-30666-XXXXX",
2020-04-16T08:09:40.765Z [DEBUG] plugin.terraform-provider-google_v3.13.0_x5:  "baseInstanceName": "testing-30666-XXXXX",
2020-04-16T08:09:40.765Z [DEBUG] plugin.terraform-provider-google_v3.13.0_x5:  "fingerprint": "DfMIXwcyn0M=",
2020-04-16T08:09:40.765Z [DEBUG] plugin.terraform-provider-google_v3.13.0_x5:  "currentActions": {
2020-04-16T08:09:40.765Z [DEBUG] plugin.terraform-provider-google_v3.13.0_x5:   "none": 0,
2020-04-16T08:09:40.765Z [DEBUG] plugin.terraform-provider-google_v3.13.0_x5:   "creating": 0,
2020-04-16T08:09:40.765Z [DEBUG] plugin.terraform-provider-google_v3.13.0_x5:   "creatingWithoutRetries": 0,
2020-04-16T08:09:40.765Z [DEBUG] plugin.terraform-provider-google_v3.13.0_x5:   "verifying": 1,
2020-04-16T08:09:40.765Z [DEBUG] plugin.terraform-provider-google_v3.13.0_x5:   "recreating": 0,
2020-04-16T08:09:40.765Z [DEBUG] plugin.terraform-provider-google_v3.13.0_x5:   "deleting": 0,
2020-04-16T08:09:40.765Z [DEBUG] plugin.terraform-provider-google_v3.13.0_x5:   "abandoning": 0,
2020-04-16T08:09:40.765Z [DEBUG] plugin.terraform-provider-google_v3.13.0_x5:   "restarting": 0,
2020-04-16T08:09:40.765Z [DEBUG] plugin.terraform-provider-google_v3.13.0_x5:   "refreshing": 0
2020-04-16T08:09:40.765Z [DEBUG] plugin.terraform-provider-google_v3.13.0_x5:  },
2020-04-16T08:09:40.765Z [DEBUG] plugin.terraform-provider-google_v3.13.0_x5:  "status": {
2020-04-16T08:09:40.766Z [DEBUG] plugin.terraform-provider-google_v3.13.0_x5:   "isStable": false,
2020-04-16T08:09:40.766Z [DEBUG] plugin.terraform-provider-google_v3.13.0_x5:   "versionTarget": {
2020-04-16T08:09:40.766Z [DEBUG] plugin.terraform-provider-google_v3.13.0_x5:    "isReached": true
2020-04-16T08:09:40.766Z [DEBUG] plugin.terraform-provider-google_v3.13.0_x5:   },
2020-04-16T08:09:40.766Z [DEBUG] plugin.terraform-provider-google_v3.13.0_x5:   "stateful": {
2020-04-16T08:09:40.766Z [DEBUG] plugin.terraform-provider-google_v3.13.0_x5:    "isStateful": false,
2020-04-16T08:09:40.766Z [DEBUG] plugin.terraform-provider-google_v3.13.0_x5:    "hasStatefulConfig": false
2020-04-16T08:09:40.766Z [DEBUG] plugin.terraform-provider-google_v3.13.0_x5:   }
2020-04-16T08:09:40.766Z [DEBUG] plugin.terraform-provider-google_v3.13.0_x5:  },
2020-04-16T08:09:40.766Z [DEBUG] plugin.terraform-provider-google_v3.13.0_x5:  "targetSize": 1,
2020-04-16T08:09:40.766Z [DEBUG] plugin.terraform-provider-google_v3.13.0_x5:  "selfLink": "https://www.googleapis.com/compute/beta/projects/XXXXX/regions/europe-west4/instanceGroupManagers/testing-30666-XXXXX",
2020-04-16T08:09:40.766Z [DEBUG] plugin.terraform-provider-google_v3.13.0_x5:  "autoHealingPolicies": [
2020-04-16T08:09:40.766Z [DEBUG] plugin.terraform-provider-google_v3.13.0_x5:   {
m-provider-google_v3.13.0_x5:    "healthCheck": "https://www.googleapis.com/compute/beta/projects/XXXXX/global/healthChecks/testing-30666-XXXXX-healthcheck-vm",
2020-04-16T08:09:40.766Z [DEBUG] plugin.terraform-provider-google_v3.13.0_x5:    "initialDelaySec": 60
2020-04-16T08:09:40.766Z [DEBUG] plugin.terraform-provider-google_v3.13.0_x5:   }
2020-04-16T08:09:40.766Z [DEBUG] plugin.terraform-provider-google_v3.13.0_x5:  ],
2020-04-16T08:09:40.766Z [DEBUG] plugin.terraform-provider-google_v3.13.0_x5:  "updatePolicy": {
2020-04-16T08:09:40.766Z [DEBUG] plugin.terraform-provider-google_v3.13.0_x5:   "type": "PROACTIVE",
2020-04-16T08:09:40.766Z [DEBUG] plugin.terraform-provider-google_v3.13.0_x5:   "instanceRedistributionType": "PROACTIVE",
2020-04-16T08:09:40.766Z [DEBUG] plugin.terraform-provider-google_v3.13.0_x5:   "minimalAction": "REPLACE",
2020-04-16T08:09:40.766Z [DEBUG] plugin.terraform-provider-google_v3.13.0_x5:   "maxSurge": {
2020-04-16T08:09:40.766Z [DEBUG] plugin.terraform-provider-google_v3.13.0_x5:    "fixed": 3,
2020-04-16T08:09:40.766Z [DEBUG] plugin.terraform-provider-google_v3.13.0_x5:    "calculated": 3
2020-04-16T08:09:40.766Z [DEBUG] plugin.terraform-provider-google_v3.13.0_x5:   },
2020-04-16T08:09:40.766Z [DEBUG] plugin.terraform-provider-google_v3.13.0_x5:   "maxUnavailable": {
2020-04-16T08:09:40.766Z [DEBUG] plugin.terraform-provider-google_v3.13.0_x5:    "fixed": 0,
2020-04-16T08:09:40.766Z [DEBUG] plugin.terraform-provider-google_v3.13.0_x5:    "calculated": 0
2020-04-16T08:09:40.766Z [DEBUG] plugin.terraform-provider-google_v3.13.0_x5:   },
2020-04-16T08:09:40.766Z [DEBUG] plugin.terraform-provider-google_v3.13.0_x5:   "minReadySec": 0,
2020-04-16T08:09:40.766Z [DEBUG] plugin.terraform-provider-google_v3.13.0_x5:   "replacementMethod": "SUBSTITUTE"
2020-04-16T08:09:40.766Z [DEBUG] plugin.terraform-provider-google_v3.13.0_x5:  },
2020-04-16T08:09:40.766Z [DEBUG] plugin.terraform-provider-google_v3.13.0_x5:  "namedPorts": [
2020-04-16T08:09:40.766Z [DEBUG] plugin.terraform-provider-google_v3.13.0_x5:   {
2020-04-16T08:09:40.766Z [DEBUG] plugin.terraform-provider-google_v3.13.0_x5:    "name": "http",
2020-04-16T08:09:40.766Z [DEBUG] plugin.terraform-provider-google_v3.13.0_x5:    "port": 80
2020-04-16T08:09:40.766Z [DEBUG] plugin.terraform-provider-google_v3.13.0_x5:   }
2020-04-16T08:09:40.766Z [DEBUG] plugin.terraform-provider-google_v3.13.0_x5:  ],
2020-04-16T08:09:40.766Z [DEBUG] plugin.terraform-provider-google_v3.13.0_x5:  "serviceAccount": "316799475005@cloudservices.gserviceaccount.com",oupManager"
2020-04-16T08:09:40.766Z [DEBUG] plugin.terraform-provider-google_v3.13.0_x5: }
2020-04-16T08:09:40.766Z [DEBUG] plugin.terraform-provider-google_v3.13.0_x5: -----------------------------------------------------
2020-04-16T08:09:40.766Z [DEBUG] plugin.terraform-provider-google_v3.13.0_x5: 2020/04/16 08:09:40 [TRACE] Waiting 10s before next try
2020-04-16T08:09:44.430Z [DEBUG] plugin.terraform-provider-google_v3.13.0_x5: 2020/04/16 08:09:44 [WARN] WaitForState timeout after 15m0s
2020-04-16T08:09:44.430Z [DEBUG] plugin.terraform-provider-google_v3.13.0_x5: 2020/04/16 08:09:44 [WARN] WaitForState starting 30s refresh grace period
2020/04/16 08:09:44 [ERROR] module.XXXXX-instances: eval: *terraform.EvalRefresh, err: timeout while waiting for state to become 'created' (last state: 'creating', timeout: 15m0s)
2020/04/16 08:09:44 [ERROR] module.XXXXX-instances: eval: *terraform.EvalSequence, err: timeout while waiting for state to become 'created' (last state: 'creating', timeout: 15m0s)
2020/04/16 08:09:44 [ERROR] timeout while waiting for state to become 'created' (last state: 'creating', timeout: 15m0s)

Expected Behavior

The instance group should be deleted, even not its in stable state.

Actual Behavior

It just hangs and timeouts after 15 minutes. I think it waits for instances to be stable, because I have configured wait_for_instances = true - but I don't think it should take that into consideration when we destroy resources.

Steps to Reproduce

  1. terraform apply -auto-approve
  2. terraform destroy -auto-approve
@ghost ghost added the bug label Apr 16, 2020
@venkykuberan venkykuberan self-assigned this Apr 16, 2020
@venkykuberan
Copy link
Contributor

Your debug log shows the currentActions.verifying =1, one of the instances are in verifying state. In order to delete the instance group, currentAction should have state NONE. No actions are being performed on the instances

Destroy makes a Get call to check the status before it destroys the IGM. Its working as expected. Let me know if its helps, i can close the issue.

@sorenhansendk
Copy link
Contributor Author

Ok, how can I then delete the instance group when its in the verifying state?

As I see it i cannot delete it with Terraform. But only, if I go to the Google Cloud Console and scale it down to 0 and then deletes the group after that.

@ghost ghost removed the waiting-response label Apr 18, 2020
@venkykuberan
Copy link
Contributor

Yeah, Delete it manually and clean up the terraform state file and start the resource creation again. s Tearraform doesn't make a destroy call when IMG have instances in some intermediate state. I am closing this issue as provider is working as expected

@ghost
Copy link

ghost commented May 21, 2020

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. If you feel I made an error 🤖 🙉 , please reach out to my human friends 👉 hashibot-feedback@hashicorp.com. Thanks!

@ghost ghost locked and limited conversation to collaborators May 21, 2020
@github-actions github-actions bot added forward/review In review; remove label to forward service/compute-managed labels Jan 14, 2025
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@venkykuberan venkykuberan

Labels
bug forward/review In review; remove label to forward service/compute-managed
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@sorenhansendk @venkykuberan