Add max-inflight-requests limit to store gateway

[justinjung04]

What this PR does:

Store gateway can only handle set number of requests at a given time, and rest of the requests are just queued up, waiting in the queryGate (channel). Since queriers select store gateways randomly without any information about their load, lots of requests could flood to one store gateway instance and slows down overall query latency (or could even lead to timeouts).

This PR add a new configuration -blocks-storage.bucket-store.max-inflight-request that allows requests to be rejected fast, which then queriers could try other store gateway replicas without waiting.

Which issue(s) this PR fixes:
n/a

Checklist

• Tests updated
• Documentation added
• CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

[friedrichg]

I like what you are doing.
Isn't this more like an instance limit?
It would be nice to have metrics like it was done in #4071

[yeya24]

It would be nice to have metrics like it was done in #4071

We already have metrics for current usage in https://github.com/cortexproject/cortex/blob/master/vendor/github.com/weaveworks/common/server/server.go#L316

[friedrichg]

right that is one, the other one was instanceLimitsMetric which shows the current limit. If it is indeed an instance limit.

I mean, I am just helping 😄 , not blocking. The requests for changes is for the tiny nit on the flag.

[yeya24]

@friedrichg Thanks for the review and help. I was just saying we already have the usage metric, agree we can add the limit metrics if necessary. All good, not blocking at all.

[justinjung04]

I liked the idea of grouping all instance limits, so on top of this new limit also created metrics for existing limits:

• Max concurrent
• Max chunk pool bytes

[yeya24]

LGTM

[yeya24]

@justinjung04 test failure seems related to the new metrics added. Once we fix that we can merge