Configurable Kubernetes Client Throttling/Rate Limiting

[AaronFriel]

Affected area

All Pulumi operations over Kubernetes resources. This would affect the construction of clients from the Kubernetes client-go package.

Based on logs emitted from deploys, it appears that Pulumi may be using a default rate limiting configuration that is too conservative. In particular, it appears client-go supports server-side rate limit detection and retries, and Pulumi implements its own retry mechanism.

I tried to follow the client-go/dynamic source code to figure out where the rate limiter was specified, but Go is not a language I work with regularly. It does seem like there ought to be, somewhere, a place to swap the RateLimiter used.

I think the dynamic.NewForConfig call here:

pulumi-kubernetes/provider/pkg/clients/clients.go

Lines 67 to 70 in 0072954

	client, err := dynamic.NewForConfig(clientConfig)
	if err != nil {
	return nil, fmt.Errorf("failed to initialize dynamic client: %v", err)
	}

Pulumi may be able to supply a "QPS" and "Burst" higher than 5 and 10, respectively: https://github.com/kubernetes/client-go/blob/f6ce18ae578c8cca64d14ab9687824d9e1305a67/rest/config.go#L115-L121

An environment variable would be perfect for us to evaluate the change, such as PULUMI_KUBE_QPS and PULUMI_KUBE_BURST, but I imagine the Pulumi team has test clusters on major platforms to evaluate for better defaults.

A moderate increase could dramatically reduce deploy times while work is ongoing to reduce the number of API calls necessary by sharing informers / watches on resources a la #1639.