Too many concurrent requests for cloud sql

[thusoy]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request.
• Please do not leave +1 or me too comments, they generate extra noise for issue followers and do not help prioritize the request.
• If you are interested in working on this issue or have submitted a pull request, please leave a comment.
• If an issue is assigned to the modular-magician user, it is either in the process of being autogenerated, or is planned to be autogenerated soon. If an issue is assigned to a user, that user is claiming responsibility for the issue. If an issue is assigned to hashibot, a community member has claimed the issue already.

Terraform Version

Terraform v0.14.5
Google provider v3.53.0

Affected Resource(s)

• google_sql_user, possibly others

Terraform Configuration Files

variable db_users = {
    "foo" : {
      name     = "foo",
      password = "foopw",
    },
    "bar" : {
      name     = "bar",
      password = "barpw",
    },
}

resource "google_sql_user" "users" {
  for_each = var.db_users
  name     = each.value.name
  instance = "some-instance-name"
  host     = "%"
  password = each.value.password
}

Debug Output

Too much to filter out in the full log, but this is the extract with the response from GCP:

19:35:23.874Z 2021/01/25 19:35:23 [DEBUG]: Looking for member "serviceAccount:SOMEUSER@FOOPROJECT.iam.gserviceaccount.com" in found binding
19:35:24.146Z 2021/01/25 19:35:24 [DEBUG] Google API Response Details:
19:35:24.146Z ---[ RESPONSE ]--------------------------------------
19:35:24.146Z HTTP/1.1 400 Bad Request
19:35:24.147Z Connection: close
19:35:24.147Z Transfer-Encoding: chunked
19:35:24.147Z Cache-Control: private
19:35:24.147Z Content-Type: application/json; charset=UTF-8
19:35:24.147Z Date: Mon, 25 Jan 2021 19:35:24 GMT
19:35:24.147Z Server: ESF
19:35:24.147Z Vary: Origin
19:35:24.147Z Vary: X-Origin
19:35:24.147Z Vary: Referer
19:35:24.147Z X-Content-Type-Options: nosniff
19:35:24.147Z X-Frame-Options: SAMEORIGIN
19:35:24.147Z X-Xss-Protection: 0
19:35:24.148Z
19:35:24.148Z 170
19:35:24.148Z {
19:35:24.148Z   "error": {
19:35:24.148Z     "code": 400,
19:35:24.148Z     "message": "Invalid request: Too many operations are being executed on the instance. Please try again later..",
19:35:24.148Z     "errors": [
19:35:24.148Z       {
19:35:24.148Z         "message": "Invalid request: Too many operations are being executed on the instance. Please try again later..",
19:35:24.148Z         "domain": "global",
19:35:24.148Z         "reason": "invalid"
19:35:24.148Z       }
19:35:24.148Z     ]
19:35:24.149Z   }
19:35:24.149Z }
19:35:24.149Z
19:35:24.149Z 0
19:35:24.149Z
19:35:24.149Z
19:35:24.149Z -----------------------------------------------------
19:35:24.149Z 2021/01/25 19:35:24 [DEBUG] Retry Transport: Stopping retries, last request failed with non-retryable error: googleapi: got HTTP response code 400 with body: HTTP/1.1 400 Bad Request```

Expected Behavior

Terraform should have created all the users.

Actual Behavior

The plan fails due to (presumably) too many concurrent requests.

Steps to Reproduce

This doesn't reproduce reliably across all of our environments, but if you try to manage enough users at the same time (for us it's only six), you'll get the 400 from GCP. If we add the -parallelism=3 flag the plan will succeed. Any value higher than 3 fails for us. Guessing this is new behavior from Google as we didn't have issues with more than 3 users before January 14th.

1. terraform plan

[edwardmedia]

@thusoy the error comes from API. Besides the features Terraform provides like limiting the size of concurrent requests, there are not much the Provider can do to avoid the issue. Closing the issue then

[thusoy]

There's no way the provider could automatically apply limits to concurrent sql operations or something like that? Is there somewhere else I should file this issue, like terraform proper? Resolving this was not obvious, and it's unfortunate that the workaround involves restricting parallelism for the entire terraform run just to avoid a limitation in the sql api.

[MrEcco]

This is work only for -parallelism=1 for user and/or databases listing via API in my case. It can be too slow for large states.

I understand what it is not terraform issue, but can anybody give info about how to resolve this/same issue? May be another issue thread in google provider repo?

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. If you feel I made an error 🤖 🙉 , please reach out to my human friends 👉 hashibot-feedback@hashicorp.com. Thanks!