Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

google_project_service fails to produce consistent results. #6182

Closed
mikhail-khodorovskiy opened this issue Apr 23, 2020 · 14 comments
Closed
Assignees
Labels

Comments

@mikhail-khodorovskiy
Copy link

mikhail-khodorovskiy commented Apr 23, 2020

Community Note

  • Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request.
  • Please do not leave +1 or me too comments, they generate extra noise for issue followers and do not help prioritize the request.
  • If you are interested in working on this issue or have submitted a pull request, please leave a comment.
  • If an issue is assigned to the modular-magician user, it is either in the process of being autogenerated, or is planned to be autogenerated soon. If an issue is assigned to a user, that user is claiming responsibility for the issue. If an issue is assigned to hashibot, a community member has claimed the issue already.

Terraform Version

Terraform v0.12.23

  • provider.external v1.2.0
  • provider.google v3.18.0
  • provider.google-beta v3.18.0
  • provider.null v2.1.2
  • provider.random v2.2.1

Affected Resource(s)

  • google_XXXXX
    Terraform v0.12.23
  • provider.external v1.2.0
  • provider.google v3.18.0
  • provider.google-beta v3.18.0
  • provider.null v2.1.2
  • provider.random v2.2.1

Terraform Configuration Files

# Copy-paste your Terraform configurations here.
#
# For large Terraform configs, please use a service like Dropbox and share a link to the ZIP file.
# For security, you can also encrypt the files using our GPG public key:
#    https://www.hashicorp.com/security
#
# If reproducing the bug involves modifying the config file (e.g., apply a config,
# change a value, apply the config again, see the bug), then please include both:
# * the version of the config before the change, and
# * the version of the config after the change.
variable "activate_apis" {
  type = list(string)

  default = [
    "storage-component.googleapis.com",
    "bigquery.googleapis.com",
    "oslogin.googleapis.com",
    "clouddebugger.googleapis.com",
    "runtimeconfig.googleapis.com",
    "serviceusage.googleapis.com",
    "pubsub.googleapis.com",
    "compute.googleapis.com",
    "cloudapis.googleapis.com",
    "servicemanagement.googleapis.com",
    "sqladmin.googleapis.com",
    "composer.googleapis.com",
    "deploymentmanager.googleapis.com",
    "iap.googleapis.com",
    "cloudtrace.googleapis.com",
    "monitoring.googleapis.com",
    "logging.googleapis.com",
    "iamcredentials.googleapis.com",
    "cloudresourcemanager.googleapis.com",
    "container.googleapis.com",
    "cloudbilling.googleapis.com",
    "sql-component.googleapis.com",
    "datastore.googleapis.com",
    "containerregistry.googleapis.com",
    "replicapool.googleapis.com",
    "replicapoolupdater.googleapis.com",
    "resourceviews.googleapis.com",
    "cloudfunctions.googleapis.com",
    "dataflow.googleapis.com",
    "appengine.googleapis.com",
    "storagetransfer.googleapis.com",
  ]
}

resource "google_project_service" "tools_project_apis" {
  count              = length(var.activate_apis)
  project            = var.tools_project_id
  service            = var.activate_apis[count.index]
  disable_on_destroy = false
  disable_dependent_services  = true
}

Debug Output

Error: Provider produced inconsistent result after apply

When applying changes to
module.chcs-dev.module.tools_project_setup.google_project_service.tools_project_apis[16],
provider "registry.terraform.io/-/google" produced an unexpected new value for
was present, but now absent.

This is a bug in the provider, which should be reported in the provider's own
issue tracker.

Error: Provider produced inconsistent result after apply

When applying changes to
module.chcs-dev.module.tools_project_setup.google_project_service.tools_project_apis[22],
provider "registry.terraform.io/-/google" produced an unexpected new value for
was present, but now absent.

This is a bug in the provider, which should be reported in the provider's own
issue tracker.

Error: Provider produced inconsistent result after apply

When applying changes to
module.chcs-dev.module.tools_project_setup.google_project_service.tools_project_apis[23],
provider "registry.terraform.io/-/google" produced an unexpected new value for
was present, but now absent.

This is a bug in the provider, which should be reported in the provider's own
issue tracker.

Error: Provider produced inconsistent result after apply

When applying changes to
module.chcs-dev.module.tools_project_setup.google_project_service.tools_project_apis[24],
provider "registry.terraform.io/-/google" produced an unexpected new value for
was present, but now absent.

This is a bug in the provider, which should be reported in the provider's own
issue tracker.

Error: Provider produced inconsistent result after apply

When applying changes to
module.chcs-dev.module.tools_project_setup.google_project_service.tools_project_apis[7],
provider "registry.terraform.io/-/google" produced an unexpected new value for
was present, but now absent.

This is a bug in the provider, which should be reported in the provider's own
issue tracker.

Error: Provider produced inconsistent result after apply

When applying changes to
module.chcs-dev.module.tools_project_setup.google_project_service.tools_project_apis[13],
provider "registry.terraform.io/-/google" produced an unexpected new value for
was present, but now absent.

This is a bug in the provider, which should be reported in the provider's own
issue tracker.

Error: Provider produced inconsistent result after apply

When applying changes to
module.chcs-dev.module.tools_project_setup.google_project_service.tools_project_apis[12],
provider "registry.terraform.io/-/google" produced an unexpected new value for
was present, but now absent.

This is a bug in the provider, which should be reported in the provider's own
issue tracker.

Error: Provider produced inconsistent result after apply

When applying changes to
module.chcs-dev.module.tools_project_setup.google_project_service.tools_project_apis[9],
provider "registry.terraform.io/-/google" produced an unexpected new value for
was present, but now absent.

This is a bug in the provider, which should be reported in the provider's own
issue tracker.

Error: Provider produced inconsistent result after apply

When applying changes to
module.chcs-dev.module.tools_project_setup.google_project_service.tools_project_apis[26],
provider "registry.terraform.io/-/google" produced an unexpected new value for
was present, but now absent.

This is a bug in the provider, which should be reported in the provider's own
issue tracker.

Error: Provider produced inconsistent result after apply

When applying changes to
module.chcs-dev.module.tools_project_setup.google_project_service.tools_project_apis[10],
provider "registry.terraform.io/-/google" produced an unexpected new value for
was present, but now absent.

This is a bug in the provider, which should be reported in the provider's own
issue tracker.

Error: Provider produced inconsistent result after apply

When applying changes to
module.chcs-dev.module.tools_project_setup.google_project_service.tools_project_apis[1],
provider "registry.terraform.io/-/google" produced an unexpected new value for
was present, but now absent.

This is a bug in the provider, which should be reported in the provider's own
issue tracker.

Error: Provider produced inconsistent result after apply

When applying changes to
module.chcs-dev.module.tools_project_setup.google_project_service.tools_project_apis[2],
provider "registry.terraform.io/-/google" produced an unexpected new value for
was present, but now absent.

This is a bug in the provider, which should be reported in the provider's own
issue tracker.

Error: Provider produced inconsistent result after apply

When applying changes to
module.chcs-dev.module.tools_project_setup.google_project_service.tools_project_apis[30],
provider "registry.terraform.io/-/google" produced an unexpected new value for
was present, but now absent.

This is a bug in the provider, which should be reported in the provider's own
issue tracker.

Error: Provider produced inconsistent result after apply

When applying changes to
module.chcs-dev.module.tools_project_setup.google_project_service.tools_project_apis[4],
provider "registry.terraform.io/-/google" produced an unexpected new value for
was present, but now absent.

This is a bug in the provider, which should be reported in the provider's own
issue tracker.

Error: Provider produced inconsistent result after apply

When applying changes to
module.chcs-dev.module.tools_project_setup.google_project_service.tools_project_apis[25],
provider "registry.terraform.io/-/google" produced an unexpected new value for
was present, but now absent.

This is a bug in the provider, which should be reported in the provider's own
issue tracker.

Error: Provider produced inconsistent result after apply

When applying changes to
module.chcs-dev.module.tools_project_setup.google_project_service.tools_project_apis[28],
provider "registry.terraform.io/-/google" produced an unexpected new value for
was present, but now absent.

This is a bug in the provider, which should be reported in the provider's own
issue tracker.

Error: Provider produced inconsistent result after apply

When applying changes to
module.chcs-dev.module.tools_project_setup.google_project_service.tools_project_apis[5],
provider "registry.terraform.io/-/google" produced an unexpected new value for
was present, but now absent.

This is a bug in the provider, which should be reported in the provider's own
issue tracker.

Error: Provider produced inconsistent result after apply

When applying changes to
module.chcs-dev.module.tools_project_setup.google_project_service.tools_project_apis[19],
provider "registry.terraform.io/-/google" produced an unexpected new value for
was present, but now absent.

This is a bug in the provider, which should be reported in the provider's own
issue tracker.

Error: Provider produced inconsistent result after apply

When applying changes to
module.chcs-dev.module.tools_project_setup.google_project_service.tools_project_apis[17],
provider "registry.terraform.io/-/google" produced an unexpected new value for
was present, but now absent.

This is a bug in the provider, which should be reported in the provider's own
issue tracker.

Error: Provider produced inconsistent result after apply

When applying changes to
module.chcs-dev.module.tools_project_setup.google_project_service.tools_project_apis[21],
provider "registry.terraform.io/-/google" produced an unexpected new value for
was present, but now absent.

This is a bug in the provider, which should be reported in the provider's own
issue tracker.

Error: Provider produced inconsistent result after apply

When applying changes to
module.chcs-dev.module.tools_project_setup.google_project_service.tools_project_apis[15],
provider "registry.terraform.io/-/google" produced an unexpected new value for
was present, but now absent.

This is a bug in the provider, which should be reported in the provider's own
issue tracker.

Error: Provider produced inconsistent result after apply

When applying changes to
module.chcs-dev.module.tools_project_setup.google_project_service.tools_project_apis[20],
provider "registry.terraform.io/-/google" produced an unexpected new value for
was present, but now absent.

This is a bug in the provider, which should be reported in the provider's own
issue tracker.

Error: Provider produced inconsistent result after apply

When applying changes to
module.chcs-dev.module.tools_project_setup.google_project_service.tools_project_apis[8],
provider "registry.terraform.io/-/google" produced an unexpected new value for
was present, but now absent.

This is a bug in the provider, which should be reported in the provider's own
issue tracker.

Error: Provider produced inconsistent result after apply

When applying changes to
module.chcs-dev.module.tools_project_setup.google_project_service.tools_project_apis[0],
provider "registry.terraform.io/-/google" produced an unexpected new value for
was present, but now absent.

This is a bug in the provider, which should be reported in the provider's own
issue tracker.

Error: Provider produced inconsistent result after apply

When applying changes to
module.chcs-dev.module.tools_project_setup.google_project_service.tools_project_apis[3],
provider "registry.terraform.io/-/google" produced an unexpected new value for
was present, but now absent.

This is a bug in the provider, which should be reported in the provider's own
issue tracker.

Error: Provider produced inconsistent result after apply

When applying changes to
module.chcs-dev.module.tools_project_setup.google_project_service.tools_project_apis[29],
provider "registry.terraform.io/-/google" produced an unexpected new value for
was present, but now absent.

This is a bug in the provider, which should be reported in the provider's own
issue tracker.

Error: Provider produced inconsistent result after apply

When applying changes to
module.chcs-dev.module.tools_project_setup.google_project_service.tools_project_apis[27],
provider "registry.terraform.io/-/google" produced an unexpected new value for
was present, but now absent.

This is a bug in the provider, which should be reported in the provider's own
issue tracker.

Error: Provider produced inconsistent result after apply

When applying changes to
module.chcs-dev.module.tools_project_setup.google_project_service.tools_project_apis[14],
provider "registry.terraform.io/-/google" produced an unexpected new value for
was present, but now absent.

This is a bug in the provider, which should be reported in the provider's own
issue tracker.

Error: Provider produced inconsistent result after apply

When applying changes to
module.chcs-dev.module.tools_project_setup.google_project_service.tools_project_apis[11],
provider "registry.terraform.io/-/google" produced an unexpected new value for
was present, but now absent.

This is a bug in the provider, which should be reported in the provider's own
issue tracker.

Error: Provider produced inconsistent result after apply

When applying changes to
module.chcs-dev.module.tools_project_setup.google_project_service.tools_project_apis[6],
provider "registry.terraform.io/-/google" produced an unexpected new value for
was present, but now absent.

This is a bug in the provider, which should be reported in the provider's own
issue tracker.

Error: Provider produced inconsistent result after apply

When applying changes to
module.chcs-dev.module.tools_project_setup.google_project_service.tools_project_apis[18],
provider "registry.terraform.io/-/google" produced an unexpected new value for
was present, but now absent.

This is a bug in the provider, which should be reported in the provider's own
issue tracker.

Error: Error reading Google Cloud Storage Transfer service account not found: googleapi: Error 403: Storage Transfer API has not been used in project 644911759278 before or it is disabled. Enable it by visiting https://console.developers.google.com/apis/api/storagetransfer.googleapis.com/overview?project=644911759278 then retry. If you enabled this API recently, wait a few minutes for the action to propagate to our systems and retry., accessNotConfigured

on ../../modules/data_transfer_project/main.tf line 40, in data "google_storage_transfer_project_service_account" "storage_transfer_sa":
40: data "google_storage_transfer_project_service_account" "storage_transfer_sa" {

Error: Error reading Google Cloud Storage Transfer service account not found: googleapi: Error 403: Storage Transfer API has not been used in project 644911759278 before or it is disabled. Enable it by visiting https://console.developers.google.com/apis/api/storagetransfer.googleapis.com/overview?project=644911759278 then retry. If you enabled this API recently, wait a few minutes for the action to propagate to our systems and retry., accessNotConfigured

on ../../modules/logging/main.tf line 42, in data "google_storage_transfer_project_service_account" "org_logs":
42: data "google_storage_transfer_project_service_account" "org_logs" {

Panic Output

Expected Behavior

Actual Behavior

Steps to Reproduce

  1. terraform apply

Important Factoids

References

  • #0000
@ghost ghost added the bug label Apr 23, 2020
@edwardmedia edwardmedia self-assigned this Apr 23, 2020
@edwardmedia
Copy link
Contributor

@mikhail-khodorovskiy I noticed that "}" is missing before resource. After adding that, your code works fine for me. Let me know if this can address your problem. If not, would please specify the steps how to reach the error?

@mikhail-khodorovskiy
Copy link
Author

@mikhail-khodorovskiy I noticed that "}" is missing before resource. After adding that, your code works fine for me. Let me know if this can address your problem. If not, would please specify the steps how to reach the error?

I made a mistake copying the snippet of the code. I tried to only give you relevant parts where locally I have the API being enabled as part of an internal module. I tried destroying and recreating the API enablement a dozen of times and it always fails with the error I posted on the first apply. Did you try the exact versions of terraform and provider combination? I even tried with the google-beta provider and still got the same error.

@ghost ghost removed the waiting-response label Apr 23, 2020
@edwardmedia
Copy link
Contributor

@mikhail-khodorovskiy your versions should be fine. I noticed your code are in modules. Can you test by putting your above code in a single main.tf, removing everything else?

@mikhail-khodorovskiy
Copy link
Author

I briefly moved the enabling of the service to my main and I still get the same error.
After I moved it back I captured the trace logs:

2020/04/24 01:42:27 [TRACE] module.chcs-dev.module.tools_project_setup: eval: *terraform.EvalReduceDiff
2020/04/24 01:42:27 [TRACE] module.chcs-dev.module.tools_project_setup: eval: *terraform.EvalIf
2020/04/24 01:42:27 [TRACE] module.chcs-dev.module.tools_project_setup: eval: terraform.EvalNoop
2020/04/24 01:42:27 [TRACE] module.chcs-dev.module.tools_project_setup: eval: *terraform.EvalApplyPre
2020/04/24 01:42:27 [TRACE] module.chcs-dev.module.tools_project_setup: eval: *terraform.EvalApply
2020/04/24 01:42:27 [DEBUG] module.chcs-dev.module.tools_project_setup.google_project_service.tools_project_apis[7]: applying the planned Create change
2020/04/24 01:42:27 [TRACE] GRPCProvider: ApplyResourceChange
2020-04-24T01:42:27.750Z [DEBUG] plugin.terraform-provider-google_v3.18.0_x5: 2020/04/24 01:42:27 [DEBUG] Adding batch request "List Project Services projects/chcs-dev-tools-54c9" to existing batch "project/projects/chcs-dev-tools-54c9/services"
2020-04-24T01:42:27.750Z [DEBUG] plugin.terraform-provider-google_v3.18.0_x5: 2020/04/24 01:42:27 [DEBUG] Added batch request "List Project Services projects/chcs-dev-tools-54c9" to batch. New batch body: <nil>
2020-04-24T01:42:29.326Z [DEBUG] plugin.terraform-provider-google_v3.18.0_x5: 2020/04/24 01:42:29 [DEBUG] Sending batch "project/projects/chcs-dev-tools-54c9/services" combining 10 requests)
2020-04-24T01:42:29.326Z [DEBUG] plugin.terraform-provider-google_v3.18.0_x5: 2020/04/24 01:42:29 [DEBUG] Listing enabled services for project projects/chcs-dev-tools-54c9
2020-04-24T01:42:29.327Z [DEBUG] plugin.terraform-provider-google_v3.18.0_x5: 2020/04/24 01:42:29 [DEBUG] Waiting for state to become: [success]
2020-04-24T01:42:29.328Z [DEBUG] plugin.terraform-provider-google_v3.18.0_x5: 2020/04/24 01:42:29 [DEBUG] Retry Transport: starting RoundTrip retry loop
2020-04-24T01:42:29.328Z [DEBUG] plugin.terraform-provider-google_v3.18.0_x5: 2020/04/24 01:42:29 [DEBUG] Retry Transport: request attempt 0
2020-04-24T01:42:29.329Z [DEBUG] plugin.terraform-provider-google_v3.18.0_x5: 2020/04/24 01:42:29 [DEBUG] Google API Request Details:
2020-04-24T01:42:29.329Z [DEBUG] plugin.terraform-provider-google_v3.18.0_x5: ---[ REQUEST ]---------------------------------------
2020-04-24T01:42:29.329Z [DEBUG] plugin.terraform-provider-google_v3.18.0_x5: GET /v1/projects/projects/chcs-dev-tools-54c9/services?alt=json&fields=services%2Fname%2CnextPageToken&filter=state%3AENABLED&prettyPrint=false HTTP/1.1
2020-04-24T01:42:29.329Z [DEBUG] plugin.terraform-provider-google_v3.18.0_x5: Host: serviceusage.googleapis.com
2020-04-24T01:42:29.329Z [DEBUG] plugin.terraform-provider-google_v3.18.0_x5: User-Agent: google-api-go-client/0.5 HashiCorp Terraform/0.12.23 (+https://www.terraform.io) Terraform Plugin SDK/1.4.0 terraform-provider-google/3.18.0
2020-04-24T01:42:29.329Z [DEBUG] plugin.terraform-provider-google_v3.18.0_x5: X-Goog-Api-Client: gl-go/1.13.7 gdcl/20200203
2020-04-24T01:42:29.329Z [DEBUG] plugin.terraform-provider-google_v3.18.0_x5: Accept-Encoding: gzip
2020-04-24T01:42:29.329Z [DEBUG] plugin.terraform-provider-google_v3.18.0_x5: 
2020-04-24T01:42:29.329Z [DEBUG] plugin.terraform-provider-google_v3.18.0_x5: 
2020-04-24T01:42:29.329Z [DEBUG] plugin.terraform-provider-google_v3.18.0_x5: -----------------------------------------------------
2020-04-24T01:42:29.761Z [DEBUG] plugin.terraform-provider-google_v3.18.0_x5: 2020/04/24 01:42:29 [DEBUG] Google API Response Details:
2020-04-24T01:42:29.761Z [DEBUG] plugin.terraform-provider-google_v3.18.0_x5: ---[ RESPONSE ]--------------------------------------
2020-04-24T01:42:29.761Z [DEBUG] plugin.terraform-provider-google_v3.18.0_x5: HTTP/1.1 404 Not Found
2020-04-24T01:42:29.761Z [DEBUG] plugin.terraform-provider-google_v3.18.0_x5: Connection: close
2020-04-24T01:42:29.761Z [DEBUG] plugin.terraform-provider-google_v3.18.0_x5: Content-Length: 1711
2020-04-24T01:42:29.761Z [DEBUG] plugin.terraform-provider-google_v3.18.0_x5: Alt-Svc: quic=":443"; ma=2592000; v="46,43",h3-Q050=":443"; ma=2592000,h3-Q049=":443"; ma=2592000,h3-Q048=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,h3-T050=":443"; ma=2592000
2020-04-24T01:42:29.761Z [DEBUG] plugin.terraform-provider-google_v3.18.0_x5: Content-Type: text/html; charset=UTF-8
2020-04-24T01:42:29.761Z [DEBUG] plugin.terraform-provider-google_v3.18.0_x5: Date: Fri, 24 Apr 2020 01:42:29 GMT
2020-04-24T01:42:29.761Z [DEBUG] plugin.terraform-provider-google_v3.18.0_x5: Server: ESF
2020-04-24T01:42:29.761Z [DEBUG] plugin.terraform-provider-google_v3.18.0_x5: X-Content-Type-Options: nosniff
2020-04-24T01:42:29.761Z [DEBUG] plugin.terraform-provider-google_v3.18.0_x5: X-Frame-Options: SAMEORIGIN
2020-04-24T01:42:29.761Z [DEBUG] plugin.terraform-provider-google_v3.18.0_x5: X-Xss-Protection: 0
2020-04-24T01:42:29.761Z [DEBUG] plugin.terraform-provider-google_v3.18.0_x5: 
2020-04-24T01:42:29.761Z [DEBUG] plugin.terraform-provider-google_v3.18.0_x5: <!DOCTYPE html>
2020-04-24T01:42:29.761Z [DEBUG] plugin.terraform-provider-google_v3.18.0_x5: <html lang=en>
2020-04-24T01:42:29.761Z [DEBUG] plugin.terraform-provider-google_v3.18.0_x5:   <meta charset=utf-8>
2020-04-24T01:42:29.761Z [DEBUG] plugin.terraform-provider-google_v3.18.0_x5:   <meta name=viewport content="initial-scale=1, minimum-scale=1, width=device-width">
2020-04-24T01:42:29.761Z [DEBUG] plugin.terraform-provider-google_v3.18.0_x5:   <title>Error 404 (Not Found)!!1</title>
2020-04-24T01:42:29.761Z [DEBUG] plugin.terraform-provider-google_v3.18.0_x5:   <style>
2020-04-24T01:42:29.761Z [DEBUG] plugin.terraform-provider-google_v3.18.0_x5:     *{margin:0;padding:0}html,code{font:15px/22px arial,sans-serif}html{background:#fff;color:#222;padding:15px}body{margin:7% auto 0;max-width:390px;min-height:180px;padding:30px 0 15px}* > body{background:url(//www.google.com/images/errors/robot.png) 100% 5px no-repeat;padding-right:205px}p{margin:11px 0 22px;overflow:hidden}ins{color:#777;text-decoration:none}a img{border:0}@media screen and (max-width:772px){body{background:none;margin-top:0;max-width:none;padding-right:0}}#logo{background:url(//www.google.com/images/branding/googlelogo/1x/googlelogo_color_150x54dp.png) no-repeat;margin-left:-5px}@media only screen and (min-resolution:192dpi){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat 0% 0%/100% 100%;-moz-border-image:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) 0}}@media only screen and (-webkit-min-device-pixel-ratio:2){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat;-webkit-background-size:100% 100%}}#logo{display:inline-block;height:54px;width:150px}
2020-04-24T01:42:29.761Z [DEBUG] plugin.terraform-provider-google_v3.18.0_x5:   </style>
2020-04-24T01:42:29.761Z [DEBUG] plugin.terraform-provider-google_v3.18.0_x5:   <a href=//www.google.com/><span id=logo aria-label=Google></span></a>
2020-04-24T01:42:29.761Z [DEBUG] plugin.terraform-provider-google_v3.18.0_x5:   <p><b>404.</b> <ins>That’s an error.</ins>
2020-04-24T01:42:29.761Z [DEBUG] plugin.terraform-provider-google_v3.18.0_x5:   <p>The requested URL <code>/v1/projects/projects/chcs-dev-tools-54c9/services?alt=json&amp;fields=services%2Fname%2CnextPageToken&amp;filter=state%3AENABLED&amp;prettyPrint=false</code> was not found on this server.  <ins>That’s all we know.</ins>
2020-04-24T01:42:29.761Z [DEBUG] plugin.terraform-provider-google_v3.18.0_x5: 
2020-04-24T01:42:29.761Z [DEBUG] plugin.terraform-provider-google_v3.18.0_x5: -----------------------------------------------------
2020-04-24T01:42:29.761Z [DEBUG] plugin.terraform-provider-google_v3.18.0_x5: 2020/04/24 01:42:29 [DEBUG] Retry Transport: Stopping retries, last request failed with non-retryable error: googleapi: got HTTP response code 404 with body: HTTP/1.1 404 Not Found
2020-04-24T01:42:29.761Z [DEBUG] plugin.terraform-provider-google_v3.18.0_x5: Connection: close
2020-04-24T01:42:29.761Z [DEBUG] plugin.terraform-provider-google_v3.18.0_x5: Content-Length: 1711
2020-04-24T01:42:29.761Z [DEBUG] plugin.terraform-provider-google_v3.18.0_x5: Alt-Svc: quic=":443"; ma=2592000; v="46,43",h3-Q050=":443"; ma=2592000,h3-Q049=":443"; ma=2592000,h3-Q048=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,h3-T050=":443"; ma=2592000
2020-04-24T01:42:29.761Z [DEBUG] plugin.terraform-provider-google_v3.18.0_x5: Content-Type: text/html; charset=UTF-8
2020-04-24T01:42:29.761Z [DEBUG] plugin.terraform-provider-google_v3.18.0_x5: Date: Fri, 24 Apr 2020 01:42:29 GMT
2020-04-24T01:42:29.761Z [DEBUG] plugin.terraform-provider-google_v3.18.0_x5: Server: ESF
2020-04-24T01:42:29.761Z [DEBUG] plugin.terraform-provider-google_v3.18.0_x5: X-Content-Type-Options: nosniff
2020-04-24T01:42:29.761Z [DEBUG] plugin.terraform-provider-google_v3.18.0_x5: X-Frame-Options: SAMEORIGIN
2020-04-24T01:42:29.761Z [DEBUG] plugin.terraform-provider-google_v3.18.0_x5: X-Xss-Protection: 0
2020-04-24T01:42:29.761Z [DEBUG] plugin.terraform-provider-google_v3.18.0_x5: 
2020-04-24T01:42:29.762Z [DEBUG] plugin.terraform-provider-google_v3.18.0_x5: <!DOCTYPE html>
2020-04-24T01:42:29.762Z [DEBUG] plugin.terraform-provider-google_v3.18.0_x5: <html lang=en>
2020-04-24T01:42:29.762Z [DEBUG] plugin.terraform-provider-google_v3.18.0_x5:   <meta charset=utf-8>
2020-04-24T01:42:29.762Z [DEBUG] plugin.terraform-provider-google_v3.18.0_x5:   <meta name=viewport content="initial-scale=1, minimum-scale=1, width=device-width">
2020-04-24T01:42:29.762Z [DEBUG] plugin.terraform-provider-google_v3.18.0_x5:   <title>Error 404 (Not Found)!!1</title>
2020-04-24T01:42:29.762Z [DEBUG] plugin.terraform-provider-google_v3.18.0_x5:   <style>
2020-04-24T01:42:29.762Z [DEBUG] plugin.terraform-provider-google_v3.18.0_x5:     *{margin:0;padding:0}html,code{font:15px/22px arial,sans-serif}html{background:#fff;color:#222;padding:15px}body{margin:7% auto 0;max-width:390px;min-height:180px;padding:30px 0 15px}* > body{background:url(//www.google.com/images/errors/robot.png) 100% 5px no-repeat;padding-right:205px}p{margin:11px 0 22px;overflow:hidden}ins{color:#777;text-decoration:none}a img{border:0}@media screen and (max-width:772px){body{background:none;margin-top:0;max-width:none;padding-right:0}}#logo{background:url(//www.google.com/images/branding/googlelogo/1x/googlelogo_color_150x54dp.png) no-repeat;margin-left:-5px}@media only screen and (min-resolution:192dpi){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat 0% 0%/100% 100%;-moz-border-image:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) 0}}@media only screen and (-webkit-min-device-pixel-ratio:2){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat;-webkit-background-size:100% 100%}}#logo{display:inline-block;height:54px;width:150px}
2020-04-24T01:42:29.762Z [DEBUG] plugin.terraform-provider-google_v3.18.0_x5:   </style>
2020-04-24T01:42:29.762Z [DEBUG] plugin.terraform-provider-google_v3.18.0_x5:   <a href=//www.google.com/><span id=logo aria-label=Google></span></a>
2020-04-24T01:42:29.762Z [DEBUG] plugin.terraform-provider-google_v3.18.0_x5:   <p><b>404.</b> <ins>That’s an error.</ins>
2020-04-24T01:42:29.762Z [DEBUG] plugin.terraform-provider-google_v3.18.0_x5:   <p>The requested URL <code>/v1/projects/projects/chcs-dev-tools-54c9/services?alt=json&amp;fields=services%2Fname%2CnextPageToken&amp;filter=state%3AENABLED&amp;prettyPrint=false</code> was not found on this server.  <ins>That’s all we know.</ins>
2020-04-24T01:42:29.762Z [DEBUG] plugin.terraform-provider-google_v3.18.0_x5: 2020/04/24 01:42:29 [DEBUG] Retry Transport: Returning after 1 attempts
2020-04-24T01:42:29.762Z [DEBUG] plugin.terraform-provider-google_v3.18.0_x5: 2020/04/24 01:42:29 [DEBUG] Batch failed with error: Failed to list enabled services for project projects/chcs-dev-tools-54c9: googleapi: got HTTP response code 404 with body: <!DOCTYPE html>
2020-04-24T01:42:29.762Z [DEBUG] plugin.terraform-provider-google_v3.18.0_x5: <html lang=en>
2020-04-24T01:42:29.762Z [DEBUG] plugin.terraform-provider-google_v3.18.0_x5:   <meta charset=utf-8>
2020-04-24T01:42:29.762Z [DEBUG] plugin.terraform-provider-google_v3.18.0_x5:   <meta name=viewport content="initial-scale=1, minimum-scale=1, width=device-width">
2020-04-24T01:42:29.762Z [DEBUG] plugin.terraform-provider-google_v3.18.0_x5:   <title>Error 404 (Not Found)!!1</title>
2020-04-24T01:42:29.762Z [DEBUG] plugin.terraform-provider-google_v3.18.0_x5:   <style>
2020-04-24T01:42:29.762Z [DEBUG] plugin.terraform-provider-google_v3.18.0_x5:     *{margin:0;padding:0}html,code{font:15px/22px arial,sans-serif}html{background:#fff;color:#222;padding:15px}body{margin:7% auto 0;max-width:390px;min-height:180px;padding:30px 0 15px}* > body{background:url(//www.google.com/images/errors/robot.png) 100% 5px no-repeat;padding-right:205px}p{margin:11px 0 22px;overflow:hidden}ins{color:#777;text-decoration:none}a img{border:0}@media screen and (max-width:772px){body{background:none;margin-top:0;max-width:none;padding-right:0}}#logo{background:url(//www.google.com/images/branding/googlelogo/1x/googlelogo_color_150x54dp.png) no-repeat;margin-left:-5px}@media only screen and (min-resolution:192dpi){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat 0% 0%/100% 100%;-moz-border-image:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) 0}}@media only screen and (-webkit-min-device-pixel-ratio:2){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat;-webkit-background-size:100% 100%}}#logo{display:inline-block;height:54px;width:150px}
2020-04-24T01:42:29.762Z [DEBUG] plugin.terraform-provider-google_v3.18.0_x5:   </style>
2020-04-24T01:42:29.762Z [DEBUG] plugin.terraform-provider-google_v3.18.0_x5:   <a href=//www.google.com/><span id=logo aria-label=Google></span></a>
2020-04-24T01:42:29.762Z [DEBUG] plugin.terraform-provider-google_v3.18.0_x5:   <p><b>404.</b> <ins>That’s an error.</ins>
2020-04-24T01:42:29.762Z [DEBUG] plugin.terraform-provider-google_v3.18.0_x5:   <p>The requested URL <code>/v1/projects/projects/chcs-dev-tools-54c9/services?alt=json&amp;fields=services%2Fname%2CnextPageToken&amp;filter=state%3AENABLED&amp;prettyPrint=false</code> was not found on this server.  <ins>That’s all we know.</ins>
2020-04-24T01:42:29.762Z [DEBUG] plugin.terraform-provider-google_v3.18.0_x5: 2020/04/24 01:42:29 [DEBUG] Sending each request in batch separately
2020-04-24T01:42:29.762Z [DEBUG] plugin.terraform-provider-google_v3.18.0_x5: 2020/04/24 01:42:29 [DEBUG] Retrying single request "List Project Services projects/chcs-dev-tools-54c9"
2020-04-24T01:42:29.762Z [DEBUG] plugin.terraform-provider-google_v3.18.0_x5: 2020/04/24 01:42:29 [DEBUG] Listing enabled services for project projects/chcs-dev-tools-54c9
2020-04-24T01:42:29.762Z [DEBUG] plugin.terraform-provider-google_v3.18.0_x5: 2020/04/24 01:42:29 [DEBUG] Waiting for state to become: [success]
2020-04-24T01:42:29.763Z [DEBUG] plugin.terraform-provider-google_v3.18.0_x5: 2020/04/24 01:42:29 [DEBUG] Retry Transport: starting RoundTrip retry loop
2020-04-24T01:42:29.763Z [DEBUG] plugin.terraform-provider-google_v3.18.0_x5: 2020/04/24 01:42:29 [DEBUG] Retry Transport: request attempt 0
2020-04-24T01:42:29.764Z [DEBUG] plugin.terraform-provider-google_v3.18.0_x5: 2020/04/24 01:42:29 [DEBUG] Google API Request Details:
2020-04-24T01:42:29.764Z [DEBUG] plugin.terraform-provider-google_v3.18.0_x5: ---[ REQUEST ]---------------------------------------
2020-04-24T01:42:29.764Z [DEBUG] plugin.terraform-provider-google_v3.18.0_x5: GET /v1/projects/projects/chcs-dev-tools-54c9/services?alt=json&fields=services%2Fname%2CnextPageToken&filter=state%3AENABLED&prettyPrint=false HTTP/1.1
2020-04-24T01:42:29.764Z [DEBUG] plugin.terraform-provider-google_v3.18.0_x5: Host: serviceusage.googleapis.com
2020-04-24T01:42:29.764Z [DEBUG] plugin.terraform-provider-google_v3.18.0_x5: User-Agent: google-api-go-client/0.5 HashiCorp Terraform/0.12.23 (+https://www.terraform.io) Terraform Plugin SDK/1.4.0 terraform-provider-google/3.18.0
2020-04-24T01:42:29.764Z [DEBUG] plugin.terraform-provider-google_v3.18.0_x5: X-Goog-Api-Client: gl-go/1.13.7 gdcl/20200203
2020-04-24T01:42:29.764Z [DEBUG] plugin.terraform-provider-google_v3.18.0_x5: Accept-Encoding: gzip
2020-04-24T01:42:29.764Z [DEBUG] plugin.terraform-provider-google_v3.18.0_x5: 
2020-04-24T01:42:29.764Z [DEBUG] plugin.terraform-provider-google_v3.18.0_x5: 
2020-04-24T01:42:29.764Z [DEBUG] plugin.terraform-provider-google_v3.18.0_x5: -----------------------------------------------------
2020-04-24T01:42:30.162Z [DEBUG] plugin.terraform-provider-google_v3.18.0_x5: 2020/04/24 01:42:30 [DEBUG] Google API Response Details:
2020-04-24T01:42:30.162Z [DEBUG] plugin.terraform-provider-google_v3.18.0_x5: ---[ RESPONSE ]--------------------------------------
2020-04-24T01:42:30.162Z [DEBUG] plugin.terraform-provider-google_v3.18.0_x5: HTTP/1.1 404 Not Found
2020-04-24T01:42:30.162Z [DEBUG] plugin.terraform-provider-google_v3.18.0_x5: Connection: close
2020-04-24T01:42:30.162Z [DEBUG] plugin.terraform-provider-google_v3.18.0_x5: Content-Length: 1711
2020-04-24T01:42:30.162Z [DEBUG] plugin.terraform-provider-google_v3.18.0_x5: Alt-Svc: quic=":443"; ma=2592000; v="46,43",h3-Q050=":443"; ma=2592000,h3-Q049=":443"; ma=2592000,h3-Q048=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,h3-T050=":443"; ma=2592000
2020-04-24T01:42:30.162Z [DEBUG] plugin.terraform-provider-google_v3.18.0_x5: Content-Type: text/html; charset=UTF-8
2020-04-24T01:42:30.162Z [DEBUG] plugin.terraform-provider-google_v3.18.0_x5: Date: Fri, 24 Apr 2020 01:42:30 GMT
2020-04-24T01:42:30.162Z [DEBUG] plugin.terraform-provider-google_v3.18.0_x5: Server: ESF
2020-04-24T01:42:30.162Z [DEBUG] plugin.terraform-provider-google_v3.18.0_x5: X-Content-Type-Options: nosniff
2020-04-24T01:42:30.162Z [DEBUG] plugin.terraform-provider-google_v3.18.0_x5: X-Frame-Options: SAMEORIGIN
2020-04-24T01:42:30.162Z [DEBUG] plugin.terraform-provider-google_v3.18.0_x5: X-Xss-Protection: 0
2020-04-24T01:42:30.162Z [DEBUG] plugin.terraform-provider-google_v3.18.0_x5: 
2020-04-24T01:42:30.162Z [DEBUG] plugin.terraform-provider-google_v3.18.0_x5: <!DOCTYPE html>
2020-04-24T01:42:30.162Z [DEBUG] plugin.terraform-provider-google_v3.18.0_x5: <html lang=en>
2020-04-24T01:42:30.162Z [DEBUG] plugin.terraform-provider-google_v3.18.0_x5:   <meta charset=utf-8>
2020-04-24T01:42:30.162Z [DEBUG] plugin.terraform-provider-google_v3.18.0_x5:   <meta name=viewport content="initial-scale=1, minimum-scale=1, width=device-width">
2020-04-24T01:42:30.162Z [DEBUG] plugin.terraform-provider-google_v3.18.0_x5:   <title>Error 404 (Not Found)!!1</title>
2020-04-24T01:42:30.162Z [DEBUG] plugin.terraform-provider-google_v3.18.0_x5:   <style>
2020-04-24T01:42:30.162Z [DEBUG] plugin.terraform-provider-google_v3.18.0_x5:     *{margin:0;padding:0}html,code{font:15px/22px arial,sans-serif}html{background:#fff;color:#222;padding:15px}body{margin:7% auto 0;max-width:390px;min-height:180px;padding:30px 0 15px}* > body{background:url(//www.google.com/images/errors/robot.png) 100% 5px no-repeat;padding-right:205px}p{margin:11px 0 22px;overflow:hidden}ins{color:#777;text-decoration:none}a img{border:0}@media screen and (max-width:772px){body{background:none;margin-top:0;max-width:none;padding-right:0}}#logo{background:url(//www.google.com/images/branding/googlelogo/1x/googlelogo_color_150x54dp.png) no-repeat;margin-left:-5px}@media only screen and (min-resolution:192dpi){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat 0% 0%/100% 100%;-moz-border-image:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) 0}}@media only screen and (-webkit-min-device-pixel-ratio:2){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat;-webkit-background-size:100% 100%}}#logo{display:inline-block;height:54px;width:150px}
2020-04-24T01:42:30.162Z [DEBUG] plugin.terraform-provider-google_v3.18.0_x5:   </style>
2020-04-24T01:42:30.162Z [DEBUG] plugin.terraform-provider-google_v3.18.0_x5:   <a href=//www.google.com/><span id=logo aria-label=Google></span></a>
2020-04-24T01:42:30.162Z [DEBUG] plugin.terraform-provider-google_v3.18.0_x5:   <p><b>404.</b> <ins>That’s an error.</ins>
2020-04-24T01:42:30.162Z [DEBUG] plugin.terraform-provider-google_v3.18.0_x5:   <p>The requested URL <code>/v1/projects/projects/chcs-dev-tools-54c9/services?alt=json&amp;fields=services%2Fname%2CnextPageToken&amp;filter=state%3AENABLED&amp;prettyPrint=false</code> was not found on this server.  <ins>That’s all we know.</ins>
2020-04-24T01:42:30.162Z [DEBUG] plugin.terraform-provider-google_v3.18.0_x5: 
2020-04-24T01:42:30.162Z [DEBUG] plugin.terraform-provider-google_v3.18.0_x5: -----------------------------------------------------
2020-04-24T01:42:30.162Z [DEBUG] plugin.terraform-provider-google_v3.18.0_x5: 2020/04/24 01:42:30 [DEBUG] Retry Transport: Stopping retries, last request failed with non-retryable error: googleapi: got HTTP response code 404 with body: HTTP/1.1 404 Not Found
2020-04-24T01:42:30.162Z [DEBUG] plugin.terraform-provider-google_v3.18.0_x5: Connection: close
2020-04-24T01:42:30.162Z [DEBUG] plugin.terraform-provider-google_v3.18.0_x5: Content-Length: 1711
2020-04-24T01:42:30.163Z [DEBUG] plugin.terraform-provider-google_v3.18.0_x5: Alt-Svc: quic=":443"; ma=2592000; v="46,43",h3-Q050=":443"; ma=2592000,h3-Q049=":443"; ma=2592000,h3-Q048=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,h3-T050=":443"; ma=2592000
2020-04-24T01:42:30.163Z [DEBUG] plugin.terraform-provider-google_v3.18.0_x5: Content-Type: text/html; charset=UTF-8
2020-04-24T01:42:30.163Z [DEBUG] plugin.terraform-provider-google_v3.18.0_x5: Date: Fri, 24 Apr 2020 01:42:30 GMT
2020-04-24T01:42:30.163Z [DEBUG] plugin.terraform-provider-google_v3.18.0_x5: Server: ESF
2020-04-24T01:42:30.163Z [DEBUG] plugin.terraform-provider-google_v3.18.0_x5: X-Content-Type-Options: nosniff
2020-04-24T01:42:30.163Z [DEBUG] plugin.terraform-provider-google_v3.18.0_x5: X-Frame-Options: SAMEORIGIN
2020-04-24T01:42:30.163Z [DEBUG] plugin.terraform-provider-google_v3.18.0_x5: X-Xss-Protection: 0
2020-04-24T01:42:30.163Z [DEBUG] plugin.terraform-provider-google_v3.18.0_x5: 
2020-04-24T01:42:30.163Z [DEBUG] plugin.terraform-provider-google_v3.18.0_x5: <!DOCTYPE html>
2020-04-24T01:42:30.163Z [DEBUG] plugin.terraform-provider-google_v3.18.0_x5: <html lang=en>
2020-04-24T01:42:30.163Z [DEBUG] plugin.terraform-provider-google_v3.18.0_x5:   <meta charset=utf-8>
2020-04-24T01:42:30.163Z [DEBUG] plugin.terraform-provider-google_v3.18.0_x5:   <meta name=viewport content="initial-scale=1, minimum-scale=1, width=device-width">
2020-04-24T01:42:30.163Z [DEBUG] plugin.terraform-provider-google_v3.18.0_x5:   <title>Error 404 (Not Found)!!1</title>
2020-04-24T01:42:30.163Z [DEBUG] plugin.terraform-provider-google_v3.18.0_x5:   <style>
2020-04-24T01:42:30.163Z [DEBUG] plugin.terraform-provider-google_v3.18.0_x5:     *{margin:0;padding:0}html,code{font:15px/22px arial,sans-serif}html{background:#fff;color:#222;padding:15px}body{margin:7% auto 0;max-width:390px;min-height:180px;padding:30px 0 15px}* > body{background:url(//www.google.com/images/errors/robot.png) 100% 5px no-repeat;padding-right:205px}p{margin:11px 0 22px;overflow:hidden}ins{color:#777;text-decoration:none}a img{border:0}@media screen and (max-width:772px){body{background:none;margin-top:0;max-width:none;padding-right:0}}#logo{background:url(//www.google.com/images/branding/googlelogo/1x/googlelogo_color_150x54dp.png) no-repeat;margin-left:-5px}@media only screen and (min-resolution:192dpi){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat 0% 0%/100% 100%;-moz-border-image:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) 0}}@media only screen and (-webkit-min-device-pixel-ratio:2){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat;-webkit-background-size:100% 100%}}#logo{display:inline-block;height:54px;width:150px}
2020-04-24T01:42:30.163Z [DEBUG] plugin.terraform-provider-google_v3.18.0_x5:   </style>
2020-04-24T01:42:30.163Z [DEBUG] plugin.terraform-provider-google_v3.18.0_x5:   <a href=//www.google.com/><span id=logo aria-label=Google></span></a>
2020-04-24T01:42:30.163Z [DEBUG] plugin.terraform-provider-google_v3.18.0_x5:   <p><b>404.</b> <ins>That’s an error.</ins>
2020-04-24T01:42:30.163Z [DEBUG] plugin.terraform-provider-google_v3.18.0_x5:   <p>The requested URL <code>/v1/projects/projects/chcs-dev-tools-54c9/services?alt=json&amp;fields=services%2Fname%2CnextPageToken&amp;filter=state%3AENABLED&amp;prettyPrint=false</code> was not found on this server.  <ins>That’s all we know.</ins>
2020-04-24T01:42:30.163Z [DEBUG] plugin.terraform-provider-google_v3.18.0_x5: 2020/04/24 01:42:30 [DEBUG] Retry Transport: Returning after 1 attempts
2020-04-24T01:42:30.163Z [DEBUG] plugin.terraform-provider-google_v3.18.0_x5: 2020/04/24 01:42:30 [DEBUG] Retried single request "List Project Services projects/chcs-dev-tools-54c9" returned response: {map[] 0xc000706a20}
2020-04-24T01:42:30.163Z [DEBUG] plugin.terraform-provider-google_v3.18.0_x5: 2020/04/24 01:42:30 [DEBUG] Retrying single request "List Project Services projects/chcs-dev-tools-54c9"
2020-04-24T01:42:30.163Z [DEBUG] plugin.terraform-provider-google_v3.18.0_x5: 2020/04/24 01:42:30 [DEBUG] Listing enabled services for project projects/chcs-dev-tools-54c9
2020-04-24T01:42:30.163Z [DEBUG] plugin.terraform-provider-google_v3.18.0_x5: 2020/04/24 01:42:30 [DEBUG] Waiting for state to become: [success]
2020-04-24T01:42:30.163Z [DEBUG] plugin.terraform-provider-google_v3.18.0_x5: 2020/04/24 01:42:30 [WARN] Removing Project Service  because it's gone
2020/04/24 01:42:30 [DEBUG] module.chcs-dev.module.tools_project_setup.google_project_service.tools_project_apis[26]: apply errored, but we're indicating that via the Error pointer rather than returning it: Provider produced inconsistent result after apply: When applying changes to module.chcs-dev.module.tools_project_setup.google_project_service.tools_project_apis[26], provider "registry.terraform.io/-/google" produced an unexpected new value for was present, but now absent.

This is a bug in the provider, which should be reported in the provider's own issue tracker.

@ghost ghost removed the waiting-response label Apr 24, 2020
@mikhail-khodorovskiy
Copy link
Author

I think I need to explain a little more of what's going on:
I use the project factory to activate "essential" for my terraform setup services:

variable "activate_apis" {
  type = list(string)

  default = [
    "cloudbuild.googleapis.com",
    "cloudresourcemanager.googleapis.com",
    "iamcredentials.googleapis.com",
  ]
}
module "project" {
  source              = "terraform-google-modules/project-factory/google"
  version             = "~> 8.0"
  random_project_id   = true
  name                = local.environment_name_with_suffix
  billing_account     = var.billing_account
  auto_create_network = true
  org_id              = var.organization_id
  folder_id           = module.environment.environment_folder_id
  lien                = true
  activate_apis       = var.activate_apis

  # the account should be deleted when the issue is resolved https://github.com/terraform-google-modules/terraform-google-project-factory/issues/243
  default_service_account     = "keep"
  labels                      = local.common_labels
  disable_services_on_destroy = false
  skip_gcloud_download        = true
}

then in a different repo this time using the project editor terraform service account I want to add the enabled service list with the code from the issue above and this is when I get the problem. I know the service list is additive but some services may turn other ones on as well if the depend on them - not sure if this is the problem. I tried to find the ones that are enabled currently with the project factory setup and added the one in using the project services resource that's not on the currently enabled list and I still get that problem. Really hope to get some help here.

@edwardmedia
Copy link
Contributor

@mikhail-khodorovskiy I am not sure how many projects are involved. Where do you run the terraform? What credential does it use? Does the credential has the required permissions in the projects where these services are intended to be enabled? Please double check if you have proper settings for the terraform runner. https://www.terraform.io/docs/providers/google/guides/getting_started.html#adding-credentials
Please let me know if this helps address your issue.

@mikhail-khodorovskiy
Copy link
Author

Would I not be getting the authentication error if the credentials were not specified or not privileged enough? As you can see in the logs above I am getting a number of HTTP responses with 404 errors codes while getting the services enabled. Does this look something that should be happening in the provider to you?

@ghost ghost removed the waiting-response label Apr 24, 2020
@morgante
Copy link

morgante commented Apr 24, 2020

@mikhail-khodorovskiy How are you managing the dependency ordering between project creation and API activation? This could also be happening if API activation is attempted before project creation is finished.

@c2thorn
Copy link
Collaborator

c2thorn commented Apr 24, 2020

@mikhail-khodorovskiy
Looking at the debug logs, it looks like the GET URL is malformed:
/v1/projects/projects/chcs-dev-tools-54c9/services...
Do you mind double-checking what var.tools_project_id is set to?

@edwardmedia
Copy link
Contributor

@mikhail-khodorovskiy I think it all depends. It sounds like a bug as you said I am getting a number of HTTP responses with 404 errors codes while getting the services enabled. Do you know which specific service(s) were enabled by this call? I assume you have watched the changes on these services before and after the call.

@danawillow
Copy link
Contributor

@c2thorn is absolutely correct on this. We could mitigate this in the provider by accepting both values (projects/{{project}} and {{project}}) in the google_project_service resource, but it should be super easy to work around on your end, @mikhail-khodorovskiy, by making sure the project getting sent over doesn't have a projects/ prefix.

@mikhail-khodorovskiy
Copy link
Author

Apply complete! Resources: 128 added, 0 changed, 0 destroyed.

Thank you so much for all your help!!!

It turns out all I had to do while looking up the project id is to change:

data "google_projects" "tools_projects" {
  filter = "labels.environment-name:${var.environment_name} AND labels.purpose:tools"
}

data "google_project" "tools_project" {
  project_id = data.google_projects.tools_projects.projects[0].project_id # used to be id
}

and it worked.

The only feedback I have is I wish the provider gave me a better error to figure this out without your help.

Close the issue please!

@ghost ghost removed the waiting-response label Apr 24, 2020
@edwardmedia
Copy link
Contributor

@mikhail-khodorovskiy Thank for using the product and your feedback

@ghost
Copy link

ghost commented May 25, 2020

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. If you feel I made an error 🤖 🙉 , please reach out to my human friends 👉 hashibot-feedback@hashicorp.com. Thanks!

@ghost ghost locked and limited conversation to collaborators May 25, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
Projects
None yet
Development

No branches or pull requests

5 participants