Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Emit max concurrency metric #5362

Merged
merged 1 commit into from
May 30, 2023

Conversation

harry671003
Copy link
Contributor

What this PR does:

This PR exposes the max_concurrent config param through prometheus metrics.
In AWS, we are autoscaling queriers based on consumed concurrency. When the system runs out of concurrency, we need to provide more concurrency by scaling up queriers.

  • consumed_concurrency_percentage = 100 * current_concurrency / total_concurrency.
  • To determine total_concurrency, we need to know the MaxConcurrent.
  • After this change the HPA metric for total_concurrency can be calculated as:
min(kube_deployment_status_replicas_ready{deployment="querier"}) * avg(cortex_querier_max_concurrent)

Which issue(s) this PR fixes:
Fixes #

Checklist

  • Tests updated
  • Documentation added
  • CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

@yeya24
Copy link
Contributor

yeya24 commented May 25, 2023

@harry671003 The integration test from ruler is related to this change. Can you take a look?

@harry671003 harry671003 force-pushed the concurrency_metric branch from 025be2c to 00414bc Compare May 25, 2023 17:41
Copy link
Member

@friedrichg friedrichg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!
Can we add the feature to the changelog?

Signed-off-by: 🌲 Harry 🌊 John 🏔 <johrry@amazon.com>
@harry671003 harry671003 force-pushed the concurrency_metric branch from 00414bc to 5cd9662 Compare May 29, 2023 15:53
@friedrichg friedrichg merged commit 1506b14 into cortexproject:master May 30, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants