Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use InPlacePodVerticalScaling feature gate instead of tweaking cgroups #14

Merged
merged 5 commits into from
Oct 28, 2023

Conversation

norbjd
Copy link
Owner

@norbjd norbjd commented Oct 28, 2023

The current version (also tagged v0.1.0) is tweaking cgroups to update the CPU requests/limits.

It works, but there are many downsides:

  • depending on the cgroups version (v1 or v2), the code is different
  • depending on the Kubernetes distribution (kind or kapsule), the code is also different
  • the most important: by tweaking cgroups (and bypassing Kubernetes), Kubernetes is not aware of the changes and can start scheduling pods on a node, thinking it has the place to do, while actually all the CPU of the node is already allocated to the existing manually tweaked pods

On Kubernetes 1.27, the alpha feature gate InPlacePodVerticalScaling was introduced (docs here), and this allows to change the CPU without restarting the pod, accomplishing exactly what we're trying to achieve.

The only downsides of using this (that I can see) are:

  • the InPlacePodVerticalScaling feature gate is only available from 1.27, so older versions might need to use the "tweaking cgroups" solution in order to boost the pods at startup
  • it seems like the pod can take longer to start even with the boost (~1 second with cgroups tweaking, ~3 seconds with the feature gate)

Still, there are more benefits overall than downsides.

@norbjd norbjd merged commit 2bf4894 into main Oct 28, 2023
2 checks passed
@norbjd norbjd deleted the use-feature-gate-InPlacePodVerticalScaling branch October 28, 2023 15:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant