Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

TotalNumberOfQueuedAndInProgressWorkflowRuns cant scale beyond 60 runners due to GH API pagination #990

Closed
2 tasks
larhauga opened this issue Dec 8, 2021 · 0 comments
Labels
enhancement New feature or request good first issue Good for newcomers

Comments

@larhauga
Copy link
Contributor

larhauga commented Dec 8, 2021

Describe the bug
When scaling runners based on TotalNumberOfQueuedAndInProgressWorkflowRuns it seems that the maximum number of possible runners are currently 60.

Having testet with a workflow with 170 jobs (created by matrix), we only reached 50 runners.

We think the issue is related to missing pagination handling of the ListWorkflowJobs api here: https://github.com/actions-runner-controller/actions-runner-controller/blob/48923fec562126e2ffe575c9d7694d18c0a56f3d/controllers/autoscaling.go#L70
The default pagination of that API is 30, and that correlates with what we see in the controller manager logs.

Checks

  • My actions-runner-controller version (v0.x.y) does support the feature
  • I'm using an unreleased version of the controller I built from HEAD of the default branch

To Reproduce

  1. Create a HorizontalRunnerAutoscaler that uses TotalNumberOfQueuedAndInProgressWorkflowRuns, with a limit of more than 60.
  2. Trigger a workflow that has more than 60 parallel jobs.
  3. Expect the number of runners to level out below what should be expected.

Expected behavior
Expected the number of runners to be equal to the number of jobs created in the workflow.

Environment (please complete the following information):

  • Deployment Method Helm
  • Helm Chart Version: 0.13.2
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request good first issue Good for newcomers
Projects
None yet
Development

No branches or pull requests

2 participants