Bump default discovery rate limit to 300rps

[negz]

Hi folks,

I'd like to encourage you to update Helm's Kubernetes discovery client rate limiter to allow bursts of up to 300 rps (or more - but 300 is the new client-go default). This allows the discovery process to complete much faster when there are up to 300 API groups (e.g. due to many CRDs being installed).

• Encourage Kubernetes clients to bump discovery rate limits to 300+ rps crossplane/crossplane#3272

See the above issue for the full details. I note that #10842 addressed something similar (CC @isutton) but at a glance it seems like this is bumping the "regular" client rate limits, which are different from the discovery client rate limits.

Running Helm 3.9.0 with many CRDs I see discovery taking about a minute when I run helm install:

$ kubectl --kubeconfig=$HOME/.kube/config.aks -n upbound-system get crds|wc -l
1872

$ helm version
version.BuildInfo{Version:"v3.9.0", GitCommit:"7ceeda6c585217a19a1131663d8cd1f7d641b2a7", GitTreeState:"", GoVersion:"go1.17.10"}

$ helm --kubeconfig=$HOME/.kube/config.aks install my-release bitnami/wordpress
I0825 22:50:43.429866 3795560 request.go:601] Waited for 1.035998917s due to client-side throttling, not priority and fairness, request: GET:https://negz-scale-test-dns-a2a05ba0.hcp.westus2.azmk8s.io:443/apis/ecr.aws.jet.crossplane.io/v1alpha1?timeout=32s
I0825 22:50:53.427214 3795560 request.go:601] Waited for 11.036005505s due to client-side throttling, not priority and fairness, request: GET:https://negz-scale-test-dns-a2a05ba0.hcp.westus2.azmk8s.io:443/apis/ram.aws.jet.crossplane.io/v1alpha1?timeout=32s
I0825 22:51:03.627182 3795560 request.go:601] Waited for 21.235791344s due to client-side throttling, not priority and fairness, request: GET:https://negz-scale-test-dns-a2a05ba0.hcp.westus2.azmk8s.io:443/apis/organizations.aws.jet.crossplane.io/v1alpha1?timeout=32s
I0825 22:51:13.627183 3795560 request.go:601] Waited for 31.235723889s due to client-side throttling, not priority and fairness, request: GET:https://negz-scale-test-dns-a2a05ba0.hcp.westus2.azmk8s.io:443/apis/ecs.aws.jet.crossplane.io/v1alpha2?timeout=32s
I0825 22:51:23.827000 3795560 request.go:601] Waited for 41.435376145s due to client-side throttling, not priority and fairness, request: GET:https://negz-scale-test-dns-a2a05ba0.hcp.westus2.azmk8s.io:443/apis/management.azure.jet.crossplane.io/v1alpha2?timeout=32s
I0825 22:51:34.026662 3795560 request.go:601] Waited for 51.634867774s due to client-side throttling, not priority and fairness, request: GET:https://negz-scale-test-dns-a2a05ba0.hcp.westus2.azmk8s.io:443/apis/maps.azure.jet.crossplane.io/v1alpha1?timeout=32s
NAME: my-release
LAST DEPLOYED: Thu Aug 25 22:51:35 2022
NAMESPACE: default
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
CHART NAME: wordpress
CHART VERSION: 15.1.3
APP VERSION: 6.0.1

This could be addressed by:

• Updating to client-go v0.25.0 or greater to take advantage of the new 300rps default.
• Manually specifying a burst of 300rps or higher.
• Replicating the pattern established by Adjust Discovery throttling kptdev/kpt#3368 to disable rate limits completely.

[phil9909]

client-go version 0.25.x is planed to be included in the Helm 3.10 release.

I searched for calls to WithDiscoveryQPS in the repo and couldn't find any matches, so Helm seems to use the default and updating to v0.25.x should bring the new default automatically.

[negz]

Ack, thanks @phil9909!

[hickeyma]

@negz client-go has been bumped to v0.25.0. It is included in the 3.10.0 release. Let us know if it works as expected.

[mattfarina]

Closing as this should be completed. Please let us know if it's not.