google_project_service 403 - not enabling API

[stankolubomir]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request.
• Please do not leave +1 or me too comments, they generate extra noise for issue followers and do not help prioritize the request.
• If you are interested in working on this issue or have submitted a pull request, please leave a comment.
• If an issue is assigned to the modular-magician user, it is either in the process of being autogenerated, or is planned to be autogenerated soon. If an issue is assigned to a user, that user is claiming responsibility for the issue. If an issue is assigned to hashibot, a community member has claimed the issue already.

Terraform Version

Terraform v0.12.24
Terraform Google Provider 3.15.0

Affected Resource(s)

• google_project_service

Terraform Configuration Files

resource "google_project_service" "service" {
  for_each = toset([
    "compute.googleapis.com",
    "cloudresourcemanager.googleapis.com"
  ])

  service = each.key

  project            = "project-x-tf-123456"
  disable_on_destroy = false
}

Debug Output

https://gist.github.com/stankolubomir/59c0741c93778b793411d151a9495514

Expected Behavior

Resource should be able to enable google API after terraform google provider upgrade from 3.9.0 to 3.15.0.

Actual Behavior

Resource is not able to enable google API after terraform google provider upgrade from 3.9.0 to 3.15.0. There is also no luck using versions 3.12.0, 3.13.0 or 3.14.0.
Although after downgrade back to 3.9.0, resource is able to enable google API.

Steps to Reproduce

1. terraform apply

[edwardmedia]

@stankolubomir I can't repro the issue. After I ran your code, I verified the services were enabled. I don't see POST requests in your log. 403 was on GET request. Can you post the entire debug log so I can take a closer look? Thanks

[stankolubomir]

@edwardmedia I've updated the gist above with the full debug log.

[edwardmedia]

@stankolubomir what do you get tf plan? Can you post your full debug log for the plan?

[stankolubomir]

@edwardmedia I think that tf plan runs correctly.
Full debug log: https://gist.github.com/stankolubomir/59c0741c93778b793411d151a9495514#file-terraform-plan-debug-google_project_service-not-enabling-api-log

[edwardmedia]

@stankolubomir I can repro your issue after I removed certain permissions on the terraform account. Not sure if that is the reason that causes your problem. Can you double check if the service account your terraform uses has related permissions? I am not sure how it relates to the different results among provider versions. Did you change any other settings (configs) when you switch to different provider versions ( 3.9.0 <==> 3.12.0, 3.13.0, 3.14.0, 3.15.0)?

[danawillow]

Took a look at the debug logs and I see what's going on. @edwardmedia I'm going to explain it thoroughly so that you can get some ideas on how to do similar sleuthing in the future.

The 403 is actually happening on the request to read the project, not on any project service reads/writes, and even though it's a 403, it's not a permissions error. It just says that the API hasn't been enabled yet:

{
  "error": {
    "code": 403,
    "message": "Cloud Resource Manager API has not been used in project XXXXXXXXXX before or it is disabled. Enable it by visiting https://console.developers.google.com/apis/api/cloudresourcemanager.googleapis.com/overview?project=XXXXXXXXXX then retry. If you enabled this API recently, wait a few minutes for the action to propagate to our systems and retry.",
    "errors": [
      {
        "message": "Cloud Resource Manager API has not been used in project XXXXXXXXXX before or it is disabled. Enable it by visiting https://console.developers.google.com/apis/api/cloudresourcemanager.googleapis.com/overview?project=XXXXXXXXXX then retry. If you enabled this API recently, wait a few minutes for the action to propagate to our systems and retry.",
        "domain": "usageLimits",
        "reason": "accessNotConfigured",
        "extendedHelp": "https://console.developers.google.com"
      }
    ],
    "status": "PERMISSION_DENIED"
  }
}

We can see from the config that that cloudresourcemanager.googleapis.com is the exact service that's trying to get enabled, so it makes sense that it hasn't been enabled yet. So, at this point, the problem is that we're trying to call an API before we enable services.

Next question is why did it work in 3.9.0 and not 3.12.0? We can check the changelog to see if anything stands out as affecting the resource. Sure enough, in the 3.12.0 changelog entry, there's a change for project_services that says google_project_service no longer attempts to enable a service that is already enabled. That change added an additional read before adding the google_project_service to check if it's enabled, and our existing read code also reads the project to check whether or not it's deleted. So, that change introduced the error here as a side effect.

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. If you feel I made an error 🤖 🙉 , please reach out to my human friends 👉 hashibot-feedback@hashicorp.com. Thanks!