Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Disable autoscaling if no metrics are supplied #728

Closed
toast-gear opened this issue Aug 16, 2021 · 1 comment
Closed

Disable autoscaling if no metrics are supplied #728

toast-gear opened this issue Aug 16, 2021 · 1 comment
Labels
enhancement New feature or request good first issue Good for newcomers help wanted Extra attention is needed
Milestone

Comments

@toast-gear
Copy link
Collaborator

toast-gear commented Aug 16, 2021

Current Behaviour
If no metric is provided in the HRA (HorizontalRunnerAutoscaler) kind then it defaults to applying the TotalNumberOfQueuedAndInProgressWorkflowRuns scaling metric logic

Suggestion
If no metric is provided in the HRA then no attempt is made to perform any autoscale activity other than those defined in the scheduled overrides configuration

Rationale
The original behaviour of applying a default metric was helpful as it meant you could deploy HRAs with less knowledge making it easier for the end user as well as reducing the amount of yaml in a manifest. Additionally, the project original only had a single scaling metric (TotalNumberOfQueuedAndInProgressWorkflowRuns) and so it made sense to just apply it by default.

Since this original logic was implemented the project has added support for scheduled overrides of the replica counts. The scheduled overrides feature allows you to adjust the min / max replica count of a RunnerDeployment based on a schedule. One use case (and I suspect a common one) of this feature is scaling replica counts down to 0 outside of a time range such as core business hours, this then allows you to scale the underlying node group down to 0 saving you £££££.

If you are attempting to save money by scaling down your nodes but not by scaling up because you can't save money scaling up due to how your cluster is setup then will want to deploy a RunnerDeployment with a min / max replica count set to the same value and back it with a HRA to take advantage of the scheduled override feature. With the current behaviour the controller will make API calls, consuming API calls (which are rate limited) unnecessarily as you are using the HRA more like a schedular than an autoscaler.

Seen as this is a breaking change to the default behaviour if anyone has any concerns please do highlight on this issue, additionally, you may want to add a metrics to all your HRAs now to prevent issues post upgrading.

@toast-gear toast-gear added the enhancement New feature or request label Aug 16, 2021
@toast-gear toast-gear added help wanted Extra attention is needed good first issue Good for newcomers labels Feb 14, 2022
@toast-gear toast-gear added this to the v0.23.0 milestone Mar 16, 2022
mumoshu added a commit that referenced this issue Apr 22, 2022
Previously, omitting hra.spec.metrics at all resulted in ARC enabling the TotalNumberOfQueuedAndInProgressWorkflowRuns.
That turned out not a good idea so since this change it is not enabled by default.

Ref #728
mumoshu added a commit that referenced this issue Apr 24, 2022
Previously, omitting hra.spec.metrics at all resulted in ARC enabling the TotalNumberOfQueuedAndInProgressWorkflowRuns.
That turned out not a good idea so since this change it is not enabled by default.

Ref #728
mumoshu added a commit that referenced this issue Apr 24, 2022
…#1372)

Previously, omitting hra.spec.metrics at all resulted in ARC enabling the TotalNumberOfQueuedAndInProgressWorkflowRuns.
That turned out not a good idea so since this change it is not enabled by default.

Ref #728
@mumoshu
Copy link
Collaborator

mumoshu commented Apr 24, 2022

Implemented in #1372

@mumoshu mumoshu closed this as completed Apr 24, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request good first issue Good for newcomers help wanted Extra attention is needed
Projects
None yet
Development

No branches or pull requests

2 participants