Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

✨ Add flags for configuring rate limits #8579

Merged
merged 1 commit into from
May 2, 2023

Conversation

lentzi90
Copy link
Contributor

@lentzi90 lentzi90 commented Apr 27, 2023

What this PR does / why we need it:

All the controllers have built-in rate limits. They throttle themselves if they hit this limit. So far it has not been possible to configure these limits. This commit adds flags to the controllers for setting both the QPS and the burst for the rate limits. The default remains the same as before (20 QPS, 30 burst).

New flags (for each controller, including CAPD):

--kube-api-qps
--kube-api-burst

Also adds .devcontainer to .gitignore.

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #8052

@k8s-ci-robot k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Apr 27, 2023
@k8s-ci-robot k8s-ci-robot added the size/M Denotes a PR that changes 30-99 lines, ignoring generated files. label Apr 27, 2023
Copy link
Contributor

@killianmuldoon killianmuldoon left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@lentzi90 could you explain a bit more how this impacts the cluster creation issue #8052 ? I think it's a good idea to expose these - but we should be clear what the rate limits are for.

bootstrap/kubeadm/main.go Outdated Show resolved Hide resolved
@lentzi90
Copy link
Contributor Author

Thanks for the fast review @killianmuldoon !
From my experiments with scaling the number of clusters, I found that the default rate limits were slowing things down considerably. The logs were full of messages like

I0302 15:50:57.343130       1 request.go:601] Waited for 4.870221494s due to client-side throttling, not priority and fairness, request: GET:https://10.96.0.1:443/api/v1/namespaces/test-697/secrets/test-697-sa

Tuning these values fixed the issue for me.

Copy link
Contributor

@killianmuldoon killianmuldoon left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A couple of nits, but otherwise this looks good. Thanks for opening this PR!

.gitignore Outdated Show resolved Hide resolved
bootstrap/kubeadm/main.go Outdated Show resolved Hide resolved
bootstrap/kubeadm/main.go Outdated Show resolved Hide resolved
All the controllers have built-in rate limits. They throttle themselves
if they hit this limit. So far it has not been possible to configure
these limits. This commit adds flags to the controllers for setting both
the QPS and the burst for the rate limits. The default remains the same
as before (20 QPS, 30 burst).

New flags (for each controller, including CAPD):

--kube-api-qps
--kube-api-burst

Also adds .devcontainer to .gitignore.
@sbueringer
Copy link
Member

Thx!

/lgtm
/approve

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label May 2, 2023
@k8s-ci-robot
Copy link
Contributor

LGTM label has been added.

Git tree hash: a5f86648aaaa9b052f6a1db2c7df050697b28747

@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: sbueringer

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label May 2, 2023
@k8s-ci-robot k8s-ci-robot merged commit 883d52c into kubernetes-sigs:main May 2, 2023
@k8s-ci-robot k8s-ci-robot added this to the v1.5 milestone May 2, 2023
@lentzi90 lentzi90 deleted the lentzi90/rate-limits branch May 2, 2023 12:57
@sbueringer sbueringer mentioned this pull request Jun 12, 2023
27 tasks
@johannesfrey
Copy link
Contributor

/area provider/core provider/control-plane-kubeadm provider/infrastructure-docker

@k8s-ci-robot k8s-ci-robot added area/provider/core Issues or PRs related to the core provider area/provider/control-plane-kubeadm Issues or PRs related to KCP area/provider/infrastructure-docker Issues or PRs related to the docker infrastructure provider labels Jun 16, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. area/provider/control-plane-kubeadm Issues or PRs related to KCP area/provider/core Issues or PRs related to the core provider area/provider/infrastructure-docker Issues or PRs related to the docker infrastructure provider cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. lgtm "Looks good to me", indicates that a PR is ready to be merged. size/M Denotes a PR that changes 30-99 lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Slow cluster creation when working with hundreds of clusters
5 participants