Bug: SGW/Querier is not timing out the query even with lower querier timeout setting

[rishabhkumar92]

What is the bug?

Hello team,

We try to cancel long expensive query by having lower than default timeout(2m). We are setting it to 1m (reference) but we are seeing long term expensive historical queries taking 100s which is causing resource starvation issues and impact other queries.

How to reproduce it?

1. Start Mimir 2.14.3
2. Set querier.timeout to 1m under querier block (ref), execute some expensive query which takes more than a minute and it's not timing out.

What did you think would happen?

Expected behavior: Query should timeout after a min if timeout is being set to 1m.
Actual behavior: Expensive query are taking > 100s to execute causing SGW resource starvation and impacting other smaller query.

What was your environment?

Kubernetes

Any additional context to share?

No response