Improve ProxyStore timeouts

[IKSIN]

We want to increase stability Thanos infrastructure with many stores/sidecars (49 in our case) when some stores respond slowly.

Changes

• Fix and improve TestProxyStore_SeriesSlowStores:
  • Fixed broken test
  • Add some use-cases
  • check on elapsed time on get data from stores.
• Change logic in ProxyStore:
  • store response timeout moved from MergedSet.Next() function to goroutine in startStreamSeriesSet to check timeouts in parallel
  • add ProxyStore metrics

More details in comments.

Verification

• Tests
• On production in our system with 49 store backends already 2 weeks.

Details

Old ProxyStore logic:

New ProxyStore logic:

This PR fixes the case when 2 or more stores are responding slowly.
It is the case if, for example,
a) common S3 storage degraded -> all stores become slow
b) multiple prom instances deployed to the same host and the host is under high CPU pressure.
new messages

This PR also fixes double timeout in case of warnings

Also, we’ve separated RT metrics with/without payload:
before this change we’ve seen pretty low RT for all stores because most of queries doesn’t return any data. But if store return any payload - RT usually is much more.

This PR change works well in most of cases because if store responding slowly - it’s usually since first data chunk.

Example:
You have 3 stores backed by Ceph and 3 fast proms/sidecars
Ceph is degraded and responding very slowly (~infinite time).
Query timeout = 2m, response timeout = 10s
As it was before:
We will wait at least 10s X 3 because we’ve started response timeout timer inside MergedSet -> Next()
Now:
We will wait at least 10s, because timeout works in parallel way for all stores.

[IKSIN]

This is recreated PR from #1789 (with fixed commits)

[bwplotka]

hm you can rebase the previous branch (: I already commented there ):