google_project_services creates dataproc-control.googleapis.com without it being specified

[mogronalol]

Terraform Version

Terraform v0.9.1

Affected Resource(s)

Please list the resources as a list, for example:

• google_project
• google_project_services

Terraform Configuration Files

resource "google_project" "project" {
  name = "foo"
  project_id = "foo"
  org_id = "180710039370"
  billing_account = "006F6F-62E19A-81CF8C"
}

resource "google_project_services" "project" {
  project = "${google_project.project.id}"
  services = [
    "cloudresourcemanager.googleapis.com",
    "container.googleapis.com",
    "compute-component.googleapis.com",
    "deploymentmanager.googleapis.com",
    "dns.googleapis.com",
    "monitoring.googleapis.com",
    "replicapool.googleapis.com",
    "replicapoolupdater.googleapis.com",
    "resourceviews.googleapis.com",
    "servicemanagement.googleapis.com",
    "storage-api.googleapis.com",
    "storage-component.googleapis.com",
    "dataproc-control.googleapis.com", // Breaks when present or omitted
    "dataproc.googleapis.com"]
}

Debug Output

google_project_services.project: Error creating services: Error enabling service "dataproc-control.googleapis.com" for project "foo": googleapi: Error 403: Service 'dataproc-control.googleapis.com' not found or permission denied., forbidden

Expected Behavior

If I don't specify dataproc-control.googleapis.com and dataproc.googleapis.com then it should not assign these services. If I do specify them then it should be able to create them.

Actual Behavior

If I don't specify dataproc-control.googleapis.com and dataproc.googleapis.com, then Google will still set them anyway. This means a subsequent plan or apply will do a diff, and think that I am removing these two services (Because they are present Google side, but not in the Terraform template, despite them never being specified). I then get a 403 when applying because it's not possible to disable dataproc-control.googleapis.com, which is Terraform tries to do because it sees it as being removed.

If I do specify dataproc-control.googleapis.com and dataproc.googleapis.com, then I get 403 when applying because it's not possible to enable dataproc-control.googleapis.com via an API call, which it Terraform tries to do. However I can run apply again, and it will work. This is because Google assigned it anyway, so a subsequent diff during plan will return no diff.

Steps to Reproduce

Please list the steps required to reproduce the issue, for example:

1. terraform apply
2. terraform plan // Notice Google has added these services, and Terraform thinks there is a diff

[danawillow]

Hey @mogronalol, I took a look at this and I don't have a good answer for you as to why this is happening :-/ I'm not able to reproduce it since I don't have permissions to create projects programmatically, and our tests that do so are passing just fine. Would you mind sharing a gist of the full debug output as specified in https://www.terraform.io/docs/internals/debugging.html? That might help.

[joe-boyce]

I get this exact same error as well as receiving one for the following:

google_project_services.project: Error creating services: Error enabling service "containeranalysis.googleapis.com" googleapi: Error 403: Service 'containeranalysis.googleapis.com' not found or permission denied., forbidden

For both the dataproc & containeranalysis services if I keep running terraform apply it finally goes through

My complete list of services being enabled via terraform are as follows:

"cloudresourcemanager.googleapis.com", "iam.googleapis.com", "container.googleapis.com", "sourcerepo.googleapis.com", "spanner.googleapis.com", "containeranalysis.googleapis.com", "cloudbuild.googleapis.com", "compute-component.googleapis.com", "deploymentmanager.googleapis.com", "dns.googleapis.com", "monitoring.googleapis.com", "resourceviews.googleapis.com", "servicemanagement.googleapis.com", "storage-api.googleapis.com", "storage-component.googleapis.com", "pubsub.googleapis.com", "replicapool.googleapis.com", "replicapoolupdater.googleapis.com", "resourceviews.googleapis.com", "dataproc-control.googleapis.com", "dataproc.googleapis.com"

[paddycarver]

I just tried reproducing this myself, enabling only dataproc.googleapis.com. My hope was to get a minimal reproduction.

In doing so, apply worked fine, a follow-up plan yielded this:

~ google_project_services.project
    services.#:          "10" => "1"
    services.133405307:  "storage-component.googleapis.com" => ""
    services.2692275209: "dataproc.googleapis.com" => "dataproc.googleapis.com"
    services.2966512281: "deploymentmanager.googleapis.com" => ""
    services.3010261123: "replicapool.googleapis.com" => ""
    services.3075019877: "replicapoolupdater.googleapis.com" => ""
    services.3077910291: "resourceviews.googleapis.com" => ""
    services.3731214611: "compute-component.googleapis.com" => ""
    services.3740470850: "container.googleapis.com" => ""
    services.3875785048: "storage-api.googleapis.com" => ""
    services.83957402:   "dataproc-control.googleapis.com" => ""

I updated the config to use those services--which, presumably, dataproc.googleapis.com relies on and just helpfully toggled for you--and another plan showed no changes, and apply worked as expected, with no changes.

Starting from the config with all those APIs enabled, a plan works as expected, and the apply gives me the forbidden error:

Error applying plan:

1 error(s) occurred:

* google_project_services.project: 1 error(s) occurred:

* google_project_services.project: Error creating services: Error enabling service "dataproc-control.googleapis.com" for project "paddy-13004-3ca09670": googleapi: Error 403: Service 'dataproc-control.googleapis.com' not found or permission denied., forbidden

Here's the config I'm working with:

resource "random_id" "id" {
  byte_length = 4
  prefix      = "paddy-13004-"
}

resource "google_project" "project" {
  project_id      = "${random_id.id.hex}"
  name            = "Paddy 13004 Repro"
  org_id          = "{OUR ORG ID}"
  billing_account = "{OUR BILLING ACCOUNT}"
}

resource "google_project_services" "project" {
  project = "${google_project.project.id}"

  services = [
    "dataproc.googleapis.com",
    "container.googleapis.com",
    "compute-component.googleapis.com",
    "deploymentmanager.googleapis.com",
    "replicapool.googleapis.com",
    "replicapoolupdater.googleapis.com",
    "resourceviews.googleapis.com",
    "storage-api.googleapis.com",
    "storage-component.googleapis.com",
    "dataproc-control.googleapis.com",
  ]
}

Now that I've seen it in action, it looks like dataproc-control.googleapis.com may not be something you can enable yourself, but is something dataproc.googleapis.com can enable for you? If that's the case, we may be able to filter in a workaround for this. In the meantime, a (gross) workaround would be to create the project not specifying dataproc-control.googleapis.com at first, then adding it to the config after the project is created. That seemed to work for me, though it's certainly not nice.

[joe-boyce]

Unfortunately this isn't just isolated to the dataproc-control.googleapis.com service

I enabled the containerregistry.googleapis.com service a moment ago and this brings with it the source.googleapis.com service so on a subsequent plan you receive a diff showing it being removed:

services.1229593865: "source.googleapis.com" => ""

Obviously if you add the service at this point to your code it works fine but if you try to add this in the original plan/apply you receive the 403 error:

* google_project_services.project: Error updating services: Error enabling service "source.googleapis.com" for project "i-cnet-qa": googleapi: Error 403: Service 'source.googleapis.com' not found or permission denied., forbidden

So this seems to affect at least the following that we know of:

dataproc-control.googleapis.com
source.googleapis.com
containeranalysis.googleapis.com

Thanks,

Joe

[paddycarver]

Thanks for the additional information @joe-boyce. I'm going to look into compiling a list of the services this affects, and getting Terraform to just ignore them. The idea being that you can't manually add (or, presumably, remove) them, so Terraform shouldn't worry about drift (because it can't do anything about them, anyways). So the real issue is just getting them to not show up in the diff, which fortunately we have a tool for. I just need to figure out which APIs it needs to ignore.

[joe-boyce]

Thanks for the update, sound good!

As you said the various services which can't be enabled/disabled are automatically enabled by other services anyway so they don't need to be referenced

Thanks again,

Joe

[mogronalol]

The other problem is being 'private' which means you can't enable them at all through the API. Unfortunately I couldn't find documentation on the rules governing this on Google's side (At least that I could see), so I'm not sure how you are supposed to know which services meet this criteria.

[danawillow]

I'm trying to find out internally whether there are any others like this. However, ignoring services that can't be disabled only solves this in the case where you actively wish to disable the service. Some APIs enable other APIs (that we probably don't want to ignore) and will show a diff anyway. For example, I enabled just dataproc.googleapis.com on a project, and ended up with 10 enabled services:

google_project_services.terraform-test:
  id = dana-tf-13004-d69858b3
  project = dana-tf-13004-d69858b3
  services.# = 10
  services.133405307 = storage-component.googleapis.com
  services.2692275209 = dataproc.googleapis.com
  services.2966512281 = deploymentmanager.googleapis.com
  services.3010261123 = replicapool.googleapis.com
  services.3075019877 = replicapoolupdater.googleapis.com
  services.3077910291 = resourceviews.googleapis.com
  services.3731214611 = compute-component.googleapis.com
  services.3740470850 = container.googleapis.com
  services.3875785048 = storage-api.googleapis.com
  services.83957402 = dataproc-control.googleapis.com

We definitely don't want to ignore things like container.googleapis.com- at the same time, we don't want to disable them on the next apply, since (presumably) the API we did enable requires that one to be enabled as well. So either way, users will have to update their config to look like what's in state :-/

[ctavan]

I see a similar issue.

First apply:

module.environment.google_project_services.main: Creating...
  project:             "" => "contentpass-staging"
  services.#:          "" => "10"
  services.1610229196: "" => "bigquery-json.googleapis.com"
  services.2117420113: "" => "pubsub.googleapis.com"
  services.238136042:  "" => "cloudapis.googleapis.com"
  services.2631575801: "" => "sqladmin.googleapis.com"
  services.323125032:  "" => "cloudtrace.googleapis.com"
  services.3237295688: "" => "monitoring.googleapis.com"
  services.3355193353: "" => "logging.googleapis.com"
  services.3731214611: "" => "compute-component.googleapis.com"
  services.3740470850: "" => "container.googleapis.com"
  services.729731224:  "" => "dataflow.googleapis.com"

Second apply:

module.environment.google_project_services.main: Modifying... (ID: contentpass-staging)
  services.#:          "21" => "10"
  services.133405307:  "storage-component.googleapis.com" => ""
  services.1610229196: "bigquery-json.googleapis.com" => "bigquery-json.googleapis.com"
  services.1712537408: "containerregistry.googleapis.com" => ""
  services.1793140363: "clouddebugger.googleapis.com" => ""
  services.2117420113: "pubsub.googleapis.com" => "pubsub.googleapis.com"
  services.238136042:  "cloudapis.googleapis.com" => "cloudapis.googleapis.com"
  services.2471815660: "servicemanagement.googleapis.com" => ""
  services.2631575801: "sqladmin.googleapis.com" => "sqladmin.googleapis.com"
  services.2966512281: "deploymentmanager.googleapis.com" => ""
  services.3010261123: "replicapool.googleapis.com" => ""
  services.3075019877: "replicapoolupdater.googleapis.com" => ""
  services.3077910291: "resourceviews.googleapis.com" => ""
  services.323125032:  "cloudtrace.googleapis.com" => "cloudtrace.googleapis.com"
  services.3237295688: "monitoring.googleapis.com" => "monitoring.googleapis.com"
  services.3355193353: "logging.googleapis.com" => "logging.googleapis.com"
  services.3731214611: "compute-component.googleapis.com" => "compute-component.googleapis.com"
  services.3740470850: "container.googleapis.com" => "container.googleapis.com"
  services.3875785048: "storage-api.googleapis.com" => ""
  services.729731224:  "dataflow.googleapis.com" => "dataflow.googleapis.com"
  services.77316126:   "sql-component.googleapis.com" => ""
  services.932568001:  "datastore.googleapis.com" => ""

Afterwards I'm good.

The output above was created on a blank project where all services were enabled via terraform.

I have a second project (which is now also managed by terraform) where we initially enabled some APIs through cloud console as well (I cannot recall exactly which ones) and only later introduced terraform to manage the APIs as well. On this second project I also have dataproc-control.googleapis.com active and cannot remove it, as described in the rest of this issue. Even though I never intended to use dataproc…

Unfortunately the whole Google API thing doesn't seem to be really well documented... So maybe somebody knows someone inside Google who can find out more about this mysterious behavior? 😉

[danawillow]

@ctavan the dataproc-control issue will be fixed with #13730. Unfortunately the double-apply issue probably won't ever be fixed. Some GCP APIs require (or at least claim that they require) others to be enabled in order to work, and the way they've decided to implement that is to go ahead and enable them for you (instead of requesting that you enable them yourself). Because of this, we have two options in terraform:

1. make you do the double apply / update your config to match state (current behavior)
2. not put the side-effect APIs into the state file

#2 is dangerous since that means the state file doesn't actually represent GCP state, so we're left with #1.

As for the Google side of this, I can tell you that people are talking about it internally, but that's about all the detail I can go into. If you have specific places where you'd like more/better documentation (and an idea on how to make it better), I'm happy to file the appropriate issues.

[ctavan]

Hey @danawillow, #13730 is already a great improvement as it at least solve the most prominent problem of the apis which cannot even be manually created or deleted, so that's awesome!

As for the other issue I think having to re-apply a second is acceptable since it only affects the very first apply and it's definitely better to have the correct representation of Google-side state in terraform.

To me it appears that Google is pretty generous in choosing which APIs to auto-enable. For example I have no clue why deploymentmanager got enabled which is clearly not necessary for any of the apis that I use to work. At least I haven't found any issues with the current more restrictive set of APIs that I'm specifying: For my workload the additional auto-enabled APIs simply don't seem to be required. Hence it's fine for me to just have terraform disable those APIs upon the second terraform apply.

So I think given #13730 we're all good now! Thank you for your work on this!

[ctavan]

BTW, does this mean that – given #13730 has been merged yesterday – can this issue be closed?

[danawillow]

Ah yes. Closing now!

[joe-boyce]

Great thanks Dana, look forward to when this is released! 👍

Cheers,

Joe

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.