-
Notifications
You must be signed in to change notification settings - Fork 1.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Explore alternatives to rate limited GitHub API for clusterctl #3982
Comments
/milestone v0.4.0 |
/help |
@fabriziopandini: Please ensure the request meets the requirements listed here. If this request no longer meets these requirements, the label can be removed In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
I'd make use of Go module proxy + |
Issues go stale after 90d of inactivity. If this issue is safe to close now please do so with Send feedback to sig-contributor-experience at kubernetes/community. |
/lifecycle frozen |
Can we prioritize this issue a bit more? GitHub rate limits are pretty fast to hit and that might impact the user experience, could we move to use go modules like we do in cluster-api/hack/tools/mdbook/releaselink/releaselink.go Lines 61 to 84 in d2a110e
|
/milestone v1.0 |
/priority important-soon Users (especially developers) are regularly hitting this issue. |
/triage accepted |
Hm I think that should not happen. Never had the problem on my machine. Maybe there is a way to confirm that the token / your account is really over the rate limit? Sorry super stupid question but I assume the GITHUB_TOKEN env var is exported? |
I'll take a look and try to at least implement the improvement mentioned by @vincepri here: #3982 (comment) 😃 |
I created a fresh PAT and it seems to work (I may have tested with an old PAT). I will keep an eye and report if it happens again. The PAT requirement should be documented though. |
+1. I will open an issue for this. |
After reading the list of releases from go modules, the next big step is to find a way to download release assets without using GitHub API |
Might be a stupid question but as we only have the sources in the go modules (https://pkg.go.dev/sigs.k8s.io/cluster-api#section-directories), from where else can we get the built/rendered release assets? |
I did not research the topic extensively, but if we can build an url like https://github.com/kubernetes-sigs/cluster-api/releases/download/v1.2.2/bootstrap-components.yaml with the info we have, I assume we can download from there without passing from the GitHub API, similar to when I click to links in the asset section in https://github.com/kubernetes-sigs/cluster-api/releases/tag/v1.2.2 page In other words, we will continue to get the assets from GitHub but using HTTP calls instead of the the GitHub APIs |
This issue is labeled with You can:
For more details on the triage process, see https://www.kubernetes.dev/docs/guide/issue-triage/ /remove-triage accepted |
/triage accepted |
Any updates on this ? |
@mjnovice - nobody has followed up on this as a general case as far as I know. Have you run into this issue? |
@killianmuldoon we have a pipeline which creates and destroys clusterAPI on kind cluster, and we constantly get throttled by this. |
Any way we can circumvent this ? |
Circumvention for this is to use a github token, which should get rid of most of the rate limiting, or else use a local repository with the yamls that github normally tries to get from the internet. You should also pin versions to prevent clustertctl from looking for it. |
+1 to what @killianmuldoon said. The bulk of the calls that clusterctl makes are to look up versions and download files from the github releases. There were some improvements made to try and redude the gihtub aoi calls by using goporxy to look up versions but the best ways to avoid rate limiting are as Killian mentioned:
|
This issue is labeled with You can:
For more details on the triage process, see https://www.kubernetes.dev/docs/guide/issue-triage/ /remove-triage accepted |
@fabriziopandini: Closing this issue. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
User Story
As a user, I would like to use clusterctl without the risk of incurring in GitHub API rate limits.
Detailed Description
Clusterctl fetches artifacts from the provider's repository, and if the repository is hosted on GitHub this operation is subject to GitHub API rate limits.
#2450 already raised this problem, and #2848 implemented caching so now clusterctl init executes 8 API calls (+3 for each additional provider), which is ~1/6 of the total number of allowed calls in 1hr (without a GitHub token).
While this is generally ok for the majority of the users, there is still a chance that user (especially developers or CI tools) incur in this limit, so we should explore alternatives for overcoming these limits e.g.
/kind feature
/area clusterctl
The text was updated successfully, but these errors were encountered: