Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow rule definition for promql/range_query case (possible enhancement) #1064

Closed
RainbowHerbicides opened this issue Aug 12, 2024 · 0 comments
Assignees

Comments

@RainbowHerbicides
Copy link

Our monitoring stack consists out of multiple segmented prometheus instances with retention period set from 6 month till 1 year (depending on metrics type) in addition to Thanos in receiver mode.

Current behaviour of promql/range_query is to take data via API from defined Prometheus hosts and perform check for each alert, which is not usually desirable behaviour. Allowing user to define range_query that is equal to retention period - might be ok for smaller installation but in case when you have hundreds or thousands alerts against metrics with varying lvls of cardinality, it might lead to undesirable consequences (long evaluation times and resource starvation). Such cases can be mitigated by usage of query/cost but with same case (high amount of alerts) we will have a tradeoff of long pipeline execution time due to each alert execution against designated Prometheus/Thanos endpoint. Having possibility to define same rules as currently implemented in rule/for might be beneficial for cases when allowing users to set range_query equal to Prometheus retention is not a desirable outcome.

TL:DR version:
Allowing manually define rule for promql/range_query in same way as rule/for might be beneficial for larger installations

@prymitive prymitive self-assigned this Aug 12, 2024
prymitive added a commit that referenced this issue Aug 20, 2024
prymitive added a commit that referenced this issue Aug 20, 2024
prymitive added a commit that referenced this issue Aug 20, 2024
prymitive added a commit that referenced this issue Aug 20, 2024
This issue was closed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants