Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Blocking certain queries in Thanos #7579

Open
rkdutta opened this issue Jul 31, 2024 · 4 comments
Open

Blocking certain queries in Thanos #7579

rkdutta opened this issue Jul 31, 2024 · 4 comments

Comments

@rkdutta
Copy link

rkdutta commented Jul 31, 2024

Is your proposal related to a problem?

I want to block certain patterns of queries which are very expensive and lead to OOM errors.

Describe the solution you'd like

It would be great if we could provide a config containing a list of queries or patterns we would like to block. I see grafana mimir has support for this but so far I did not find any such option in Thanos.
https://grafana.com/docs/mimir/latest/configure/configure-blocked-queries/#configure-queries-to-block

Describe alternatives you've considered

(Write your answer here.)

Additional context

(Write your answer here.)

@harry671003
Copy link
Contributor

This feature was also implemented recently in Cortex. cortexproject/cortex#6005
It's just a matter of porting the changes to Thanos.

@rkdutta
Copy link
Author

rkdutta commented Jul 31, 2024

ok thanks for the quick response, I am not sure if there is a separate thread about porting the changes to Thanos.

@harry671003
Copy link
Contributor

It's not being worked on currently. I was merely pointing out that we can copy most of the Cortex code as is into Thanos.

Thanks for submitting this feature request.

@rkdutta
Copy link
Author

rkdutta commented Aug 2, 2024

Thinking from user behavior..

If you go to explorer page in Grafana, select a thanos datasource and then you specify the labels before you set the metric, this creates a very open-ended query to the backend and leads to OOM. I have seen people doing this many times. I don't know yet how to block this behavioral pattern.

Maybe it's a good idea to have a solution that does not allow querying if the metric name is missing or if the minimum number of labels is less than an acceptable number. Just thought of sharing my 2 cents.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants