Deleting activated services on a project fails

[morgante]

I'm trying to delete a module which both created a project and activated some services on that project, but it fails because the project gets created first.

Terraform Version

Terraform v0.11.5
+ provider.external v1.0.0
+ provider.google v1.8.0
+ provider.gsuite (unversioned)
+ provider.null v1.0.0
+ provider.random v1.1.0

Affected Resource(s)

Please list the resources as a list, for example:

• google_project_service

Terraform Configuration Files

resource "google_project" "folder" {
  count           = "${var.folder_id != false ? 1 : 0}"
  name            = "${var.name}"
  project_id      = "${local.final_project_id}"
  folder_id       = "${var.folder_id}"
  billing_account = "${var.billing_account}"
}

locals {
  project_id       = "${element(coalescelist(google_project.folder.*.project_id, google_project.org.*.project_id), 0)}"
}

resource "google_project_service" "project_services" {
  count = "${length(var.activate_apis)}"

  project = "${local.project_id}"
  service = "${element(var.activate_apis,count.index)}"
}

Debug Output

Ping me privately if this is needed.

Expected Behavior

The services resource is "deleted" before the project (not that it actually needs to do something).

Actual Behavior

The service enters a zombie state where it constantly tries to get removed even though the project isn't even active any more.

Whenever I try to apply afterwards it's unable to fetch the state:

* module.application-xxx-xxx-xxx.module.project-xxx-xxx.google_project_service.project_services: google_project_service.project_services: googleapi: Error 404: Project 'marketo-xxx-xxx-xxx' not found or permission denied., notFound

[morgante]

Also running into this with usage bucket export:

* module.application-marketo-bigdata-solr.module.project-application-int.google_project_usage_export_bucket.usage_report_export: google_project_usage_export_bucket.usage_report_export: Error reading Project data for project marketo-bigdata-solr-int: googleapi: Error 403: Project 593012260542 has been scheduled for deletion and cannot be used for API calls. Visit https://console.developers.google.com/iam-admin/projects?pendingDeletion=true to undelete the project., accessNotConfigured

[rosbo]

We should add a field disable_on_destroy like we did for google_project_service.

Excerpt from the google_project_service documentation:

• disable_on_destroy - (Optional) If true, disable the service when the terraform resource is destroyed. Defaults to true. May be useful in the event that a project is long-lived but the infrastructure running in that project changes frequently.

[morgante]

Thanks @rosbo but I'm not sure that actually solves it since I'm already using google_project_service with the default parameter (disable_on_destory defaults to true).

[rosbo]

Ooops. I read too quickly and mistakenly read "google_project_services" for the resource you are using because the name of your resource is "project_services"...

I believe the problem you are experiencing is caused by the fact that google_project_service.project_services doesn't have a depends_on relationship with the google_project.folder so Terraform tries to delete both at the same time.

Usually, that relationship is created automatically when interpolating the project field in the google_project_service resource.

In your case, you will need to add an explicit dependency:

resource "google_project_service" "project_services" {
...
   depends_on = ["google_project.folder"]
}

Let me know if that fixes your problem.

[morgante]

I'm curious why the dependency isn't being created automatically? Do you know why the transitivity isn't working when it passes through the locals variable first?

The dependency did work when the project was created since I didn't get any errors about trying to activate services on a non-existent project.

[catsby]

I'm guessing it's because google_project_service depends on locals, which depends on google_project, so the dependencies are interpolated correctly for create.

I imagine the graph looks different for a complete destroy, IIRC that would read from state and plan to destroy everything, meaning it wouldn't consider the dependency on local et. al. I could be wrong though...

If I'm write, that would mean that project and service don't get a dependency and are deleted in parallel (or possibly deleted in parallel, depending on the number of resources). Which leads to this issue.

In that case, the depends_on forces the dependency in destroy, but is redundant in create.

Does changing _service to get it's project directly work?

resource "google_project" "folder" {
  count           = "${var.folder_id != false ? 1 : 0}"
  name            = "${var.name}"
  project_id      = "${local.final_project_id}"
  folder_id       = "${var.folder_id}"
  billing_account = "${var.billing_account}"
}

resource "google_project_service" "project_services" {
  count = "${length(var.activate_apis)}"

  project = "${element(coalescelist(google_project.folder.*.project_id, google_project.org.*.project_id), 0)}"
  service = "${element(var.activate_apis,count.index)}"
}

[u2mejc]

@danawillow Is this related to #1538 ?

[danawillow]

Nope- #1538 is specific to the release that occurred last week.

[paddycarver]

@morgante is this still an issue? Can I close this out? If this is still an issue, I think it's at least an upstream issue with Terraform--I'm not sure there's much the provider can do, unfortunately, except maybe consider the service deleted if the project can't be found.

[morgante]

@paddycarver Yes I still encounter this regularly.

except maybe consider the service deleted if the project can't be found.

That would help a lot. I regularly have to run terraform state rm to solve this today.

[paddycarver]

@morgante I was unable to successfully reproduce the 404 error, but I managed to get the 403. #2090 should handle both, even though I can't verify the 404.

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. If you feel I made an error 🤖 🙉 , please reach out to my human friends 👉 hashibot-feedback@hashicorp.com. Thanks!