Bug: Client-side throttling

[chobostar]

Contact Details

No response

Describe bug

A lot of messages in logs from Velero's pod:

I0214 09:50:17.550266       1 request.go:665] Waited for 1.044379662s due to client-side throttling, not priority and fairness, request: GET:https://172.30.0.1:443/apis/kibana.k8s.elastic.co/v1?timeout=32s
I0214 09:50:27.550734       1 request.go:665] Waited for 11.043717333s due to client-side throttling, not priority and fairness, request: GET:https://172.30.0.1:443/apis/azure.upbound.io/v1alpha1?timeout=32s
I0214 09:50:37.550903       1 request.go:665] Waited for 8.345544199s due to client-side throttli

That's slow down velero pod routine.

700+ CRDs in cluster:

$ kubectl get crds | wc -l
747

here is similar discussion - kubernetes/kubectl#1126
here is solution kubernetes/kubernetes#109141

It is required to update client-go for Velero.

What happened?

A bug happened!

OADP Version

1.0.3 (Stable)

OpenShift Version

4.10

Velero pod logs

I0214 09:50:17.550266       1 request.go:665] Waited for 1.044379662s due to client-side throttling, not priority and fairness, request: GET:https://172.30.0.1:443/apis/kibana.k8s.elastic.co/v1?timeout=32s
I0214 09:50:27.550734       1 request.go:665] Waited for 11.043717333s due to client-side throttling, not priority and fairness, request: GET:https://172.30.0.1:443/apis/azure.upbound.io/v1alpha1?timeout=32s
I0214 09:50:37.550903       1 request.go:665] Waited for 8.345544199s due to client-side throttling, not priority and fairness, request: GET:https://172.30.0.1:443/apis/postgres-operator.crunchydata.com/v1beta1?timeout=32s

Restic pod logs

No response

Operator pod logs

No response

New issue

• This issue is new

[kaovilai]

Point taken. Will work with velero to get it updated. Not sure yet if it will make it to 1.0 version.

[kaovilai]

vmware-tanzu/velero#5907

[kaovilai]

closed via vmware-tanzu/velero#5907

[kaovilai]

This has been addressed upstream and just need to be included in our fork in future versions