Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add scale out experiment scenario #128

Merged
merged 15 commits into from
Feb 5, 2024
Merged

Add scale out experiment scenario #128

merged 15 commits into from
Feb 5, 2024

Conversation

timebertt
Copy link
Owner

What this PR does / why we need it:

This PR adds a second experiment scenario: scale-out.
The new scenario has a high churn rate and is designed to overload 5 webhosting-operator instances with 5 workers.
It is executed for 1 to 5 instances each. For every instance count, the maximum load capacity is calculated where the SLOs are still satisfied.

To make the high churn rate work, several things need to be improved:

  • kube-controller-manager needs to get guaranteed resources (on the dedicated high-cpu worker pool)
  • disable HTTP2 in kube-controller-manager as well
  • set GOMAXPROCS to match CPU quota (prevents CPU throttling and decreases tail latency)
  • align histogram buckets with SLOs (as soon as percentile estimate crosses the SLO, we are sure the SLO is not satisfied anymore)

@timebertt timebertt added the enhancement New feature or request label Feb 5, 2024
@timebertt timebertt merged commit fd39c7d into main Feb 5, 2024
5 checks passed
@timebertt timebertt deleted the scale-out branch February 5, 2024 19:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant