-
Notifications
You must be signed in to change notification settings - Fork 1.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
google provider is rejected from gcp API for frequent request #6021
Comments
@toshitanian what version of the provider are you on? We have several improvements in our recent versions to take care of these sorts of issues. If you're on an up-to-date version, it would help also to get some of the information that we ask for in the bug template since this feels more like a bug to me. |
I added the version |
Does this happen consistently or only sometimes? Does it happen with every resource / data source, or just GKE? |
It happened only on GKE cloud cluster for me as I don't use other resource with terraform. |
@emilymye any idea whether there's anything that can be done here? You know more than I do about this type of error. In the meantime, @toshitanian, keep us posted on whether this seems to happen often or just during certain times. Normally, quota errors show up as successful requests but with an error code (rather than this, where the request doesn't seem to have actually made it to a GCP backend). |
@toshitanian, if this happens again, can you post debug logs (env var TF_LOG=DEBUG)? I just ran into this myself where it started returning that error after retrying on several 503s in a row, and I'm curious whether yours are the same or if it happens on the very first request. |
@danawillow yes. when it happens again, i will report more detail logs. Thank you for your help. |
@danawillow I am seeing this more often as well for our GKE module CI |
Interesting, thanks @bharathkkb. You don't happen to have debug logs (showing the full request/response) for those, do you? I'd be curious to see what sorts of responses we're getting back from the API and retrying on before we finally get that error. |
Unfortunately not since its in CB, I will post if I encounter locally. |
@danawillow I have collected debug logs and attached to b/164189639 |
Thank you, those are super helpful! I have some changes I'd like to make that should improve this (it might not fix it 100% but should make it better in some cases). Will keep you informed. |
@danawillow @bharathkkb This is happening for me too quite frequently when running tf plan
Run tf plan again
Run tf plan again
Run tf plan again It works It appears randomly, I keep running tf plan again and again and it gets works automatically at some point. Running v0.13.4 of Terraform with |
@tvvignesh can you try increasing the value for |
@danawillow Ahh. Did not know that a timeout option existed. My bad. Tried it and it works great most of the times now (it does fail in some instances though but the error changed from timeout to network unreachable), Thanks. PS: The interesting thing is I am running TF within a GCP VM and applying only GCP resources so, basically I am using everything within the google network but still there are timeouts and failures. |
No worries! It actually took me a decent amount of time debugging this a month or two back to realize that that specific error would be fixed by increasing the timeout. And yeah, they happen sometimes for non-networking related reasons (sometimes things just queue up on the service end for whatever reasons) |
Given the broad scope of the issue, the age, and the fact that I believe this has been addressed over time, i.e. with the changes Dana outlined having made above. I'm going to close out this issue, but if folks are running into it please open new one(s)! I recognise that having open source issues closed without a clear resolution can be annoying, and I'm sorry about that. The reality is that this issue is pretty far into our backlog, and won't realistically have action taken on it- a newer issue is much more likely to drive progress on a solution if the issue is still present. |
I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. |
Community Note
Description
When I was applying google provider to GCP,
terraform plan
failed with API timeout.Finally I found that I have to reduce parallelism of terraform.
If it's possible I want google-provider to handle parallelism by itself.
Version
Terraform v0.12.24
New or Affected Resource(s)
Potential Terraform Configuration
References
The text was updated successfully, but these errors were encountered: