This repository has been archived by the owner on Jul 21, 2023. It is now read-only.
terraform: autoscale k8s deployments #536
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
To spare ourselves the hassle of manually resizing deployments, we configure Kubernetes to automatically resize `intake-batch` and `aggregate` worker pools. This requires deploying the Stackdriver custom metrics adapter to make PubSub metrics visible to a Kubernetes Horizontal Pod Autoscaler, then configuring an HPA for each deployment that consults the PubSub `num_undelivered_messages` metric. See `terraform/README.md` for more details and discussion of config parameter choices. This commit also modifies the `integration-tester` deployment so that it emits more batches, forcing dev and staging clusters to exercise the autoscaling feature. We also amend the alert for `intake-batch` task queue size: we no longer expect that queue to periodically empty since we have configured Kubernetes to keep it at a steady state of ~150 messages. Resolves #484
aaomidi
approved these changes
Apr 5, 2021
|
Leaving this here for reference: https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/ I'm somewhat more confident now that we can revise the metric target here to improve autoscaling behaviors. |
|
Obsoleted by #1042 |
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
To spare ourselves the hassle of manually resizing deployments, we
configure Kubernetes to automatically resize
intake-batchandaggregateworker pools. This requires deploying the Stackdriver custommetrics adapter to make PubSub metrics visible to a Kubernetes
Horizontal Pod Autoscaler, then configuring an HPA for each deployment
that consults the PubSub
num_undelivered_messagesmetric. Seeterraform/README.mdfor more details and discussion of configparameter choices.
This commit also modifies the
integration-testerdeployment so that itemits more batches, forcing dev and staging clusters to exercise the
autoscaling feature. We also amend the alert for
intake-batchtaskqueue size: we no longer expect that queue to periodically empty since
we have configured Kubernetes to keep it at a steady state of ~150
messages.
This is essentially a revert of the revert of #507, but corrects a couple mistakes in that PR:
Resolves #484