-
Notifications
You must be signed in to change notification settings - Fork 1.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Develop mechanism to overcome 429 "hits the concurrent operations quota" #15579
Develop mechanism to overcome 429 "hits the concurrent operations quota" #15579
Comments
+1, but as i says it has some limit at 5 (and turned out it performed those first 5 changes, at least in my case) │ , rateLimitExceeded |
+1 |
Note for the service team: We might need to introduce some sort of mutex or batching if we can only have 5 concurrent operations. |
Hi, replying here from the team behind this change. The retrying of operations is intentional, as we've recently made improvements such that most node pool operations can now be performed concurrently. But rather than try to rate limit client side or duplicate the complicated logic for determining which operations can be concurrent with other operations, we opted to have Terraform simply retry the creation over and over until it eventually succeeds. In other words, it's like there is a mutex but it's server side. This isn't supposed to cause overall TF apply failures, though. I believe what's happening is that it's hitting the timeout for overall creation, which defaults to 30 minutes but can be overridden in your config with a standard field (https://developer.hashicorp.com/terraform/language/resources/syntax#operation-timeouts). That said, there is a bug for us to fix here (or at least an unintentional change for us to decide how we should handle). Previously (with the one operation per cluster limit), the deadline started after the client side mutex, but now it starts after. That also means some of the deadline is consumed by this period of waiting for operations, which was unintentional. Until we can address this, I suggest increasing the In some cases, this error may be a red herring due to a different resource also failing. e.g., with the current quota of 5 operations per cluster, you could create 10 node pools. Only 5 would actually start creating and the other 5 would be blocked waiting for the first 5 to finish. If those first 5 take too long to finish, the pending 5 could timeout first, causing the error to mention the pending 5 even though the 5 that were being created (but are taking too long) are the real issue. Sufficiently large node pools failing to become healthy or stockouts could cause this. Bumping the timeout would also help if this is the case. For most users, Terraform apply shouldn't fail. If you simply see these HTTP 429 errors in the logs but everything overall succeeds, that's working as intended. Since the operations can now be done concurrently instead of having to be sequential (the previous status quo), the overall time to completion should be lower. We also intend to increase the default concurrency quota soon (which is one reason we didn't limit this client side), with a September ETA. That should also significantly help with this as well as further improving time to completion. As an aside, the |
@KatrinaHoffert, we can increase the default timeout on these resources for what it's worth if you think that is a good fix in the interim. But I'm not sure what a good value would be. Could someone from your team help to make this change? |
Actually, while the above is an issue while there's op conflicts, after having more details provided about an instance of this, I've realized the more immediate issue is that the error checking in https://github.com/hashicorp/terraform-provider-google/blob/main/google/services/container/resource_container_node_pool.go#L520 handles only the case of operation conflicts (which are a "failed precondition" error). The quota errors, on the other hand, are "resource exhausted" (HTTP 429), so are not being retried when they're meant to be. I have someone on my team looking into this and have escalated our internal ticket. The |
We have a quota of 1. how can we increase it? example: |
5 doesn't work, only 1 does which slows down deployments to an unacceptable level. Is there any movement on the linked PR? |
My apologies, that shouldn't have happened. A bad flag rollout that was intended to increase the quota resulted in a temporary period of falling back to an unintended fallback value, which tragically makes this issue worse. We'll have a fix for that rolling out shortly and is expected to finish before EOD. This is likely also why there's a mention that |
Hi @KatrinaHoffert has the fix for this issue already been rolled out? |
I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. |
Community Note
Description
While using terraform ("~> 1.4.6"), and google terraform provider (v4.78.0), I hit the following error many times:
11:50:40 │ Error: error creating NodePool: googleapi: Error 429: Too many operations: cluster "clustername" hits the concurrent operations quota 5, please try again later. 11:50:40 │ Details: 11:50:40 │ [ 11:50:40 │ { 11:50:40 │ "@type": "type.googleapis.com/google.rpc.RequestInfo", 11:50:40 │ "requestId": "*********" 11:50:40 │ } 11:50:40 │ ] 11:50:40 │ , rateLimitExceeded 11:50:40 │ 11:50:40 │ with google_container_node_pool.workload-node-pool[2], 11:50:40 │ on main.tf line 122, in resource "google_container_node_pool" "workload-node-pool": 11:50:40 │ 122: resource "google_container_node_pool" "workload-node-pool" {
I can see this error a lot in my Audit logs:
{ "protoPayload": { "@type": "type.googleapis.com/google.cloud.audit.AuditLog", "status": { "code": 8, "message": "Too many operations: cluster \"****\" hits the concurrent operations quota 5, please try again later." }, "authenticationInfo": { "principalEmail": "****", "serviceAccountKeyName": "****", "principalSubject": "****" }, "requestMetadata": { "callerIp": "****", "callerSuppliedUserAgent": "google-api-go-client/0.5 Terraform/1.3.7 (+https://www.terraform.io) Terraform-Plugin-SDK/2.10.1 terraform-provider-google/4.78.0,gzip(gfe)", "requestAttributes": { "time": "2023-08-20T16:49:31.313550903Z", "auth": {} }, "destinationAttributes": {} }, "serviceName": "container.googleapis.com", "methodName": "google.container.v1.ClusterManager.CreateNodePool", "authorizationInfo": [ { "permission": "container.clusters.update", "granted": true, "resourceAttributes": {} } ], "resourceName": "projects/****/zones/****/clusters/****/nodePools/****", "request": { "parent": "projects/****/locations/****/clusters/****", "@type": "type.googleapis.com/google.container.v1alpha1.CreateNodePoolRequest", "nodePool": { "name": "****", "config": { "imageType": "UBUNTU_CONTAINERD", "labels": { "aquanode": "****" }, "diskType": "pd-ssd", "loggingConfig": {}, "machineType": "n2-standard-32", "diskSizeGb": 100, "oauthScopes": [ "https://www.googleapis.com/auth/devstorage.read_only" ] }, "initialNodeCount": 1, "networkConfig": {} } }, "response": { "@type": "type.googleapis.com/google.container.v1alpha1.Operation" }, "resourceLocation": { "currentLocations": [ "****" ] }, "policyViolationInfo": { "orgPolicyViolationInfo": {} } }, "insertId": "****", "resource": { "type": "gke_nodepool", "labels": { "location": "****", "cluster_name": "****", "nodepool_name": "****", "project_id": "****" } }, "timestamp": "2023-08-20T16:49:31.923885772Z", "severity": "ERROR", "logName": "projects/****logs/cloudaudit.googleapis.com%2Factivity", "receiveTimestamp": "2023-08-20T16:49:32.282783167Z" }
New or Affected Resource(s)
References
(#15205)
b/298050622
The text was updated successfully, but these errors were encountered: