[Query] Partial response doesn't work if one of stores returned timeout

[d-ulyanov]

Hi, Thanos team!

What happened:
Thanos Query returns error (context deadline exceeded) if one of requested stores failed with timeout.
Partial response parameter is enabled.

Expected behaviour:
Thanos returns part of successfully fetched data and warning that one of stores failed.

In the same time there is an other case: if one of stores is not available at all (process is not running e.g.) Thanos Query returns data partially and warning.
It seems that behaviour should be the same in both cases.

@bwplotka what do you think?

Thanos v0.2.1

[bwplotka]

Well, is it because the request to one of the stor actually took too long to wait, and either your client or Thanos Query server timeout killed the request? (: So it's not partial erorr - whole request took too long, even though the root cause was one slow store.

So now is the question what would you expect in this case? (:

[d-ulyanov]

Yes, In this case Thanos Query killed request, but some stores already returned data.
I think its better to support partial degradation and return some data and warning that some of stores timed out.

I think this case should not be different with behaviour if one of stores is down at all

[d-ulyanov]

@bwplotka any ideas?)
One of possible solutions is to add additional parameter like store.read_timeout and use it for closing requests to particular stores.

[bwplotka]

Hm.. yes, as long as store.read_timeout is smaller the client timeout - that would make sense. But overall it's a tradeoff between availability & conistency. Partial response can indeed surprise you, especially when you use 3rd paty tools like Grafana where warning is not even there.

I am fine to accept PR with your idea though to have this internal timeout to be configurable for Thanos Querier 👍

[d-ulyanov]

@bwplotka Hey! Here it is :)
#895

[bwplotka]

Hm.. so @povilasv fixed this issue IMO with this: #928

[bwplotka]

so what #895 is adding? ;p

[d-ulyanov]

Moving discussion to #1453