Engine issue: different results with thanos and prometheus engines. group_left?

[ahurtaud]

Hello,

We have 2 similar thanos query (observee / observer) architectures. (query->queriers->sidecars):
One with the prometheus promql engine and one with the thanos promql engine and the following query gave different results with the thanos promql engines compared to thanos with prometheus engine and directly executed in prometheus also:

avg by (hrStorageDescr) (hrStorageUsed{nodename =~"xxxxx", } * on (instance, hrStorageIndex ) group_left(hrStorageDescr) (sum by (instance, hrStorageIndex, hrStorageDescr) (hrStorageDescr{nodename =~"xxxxx", hrStorageDescr!~".*Data.*"} ) ))

results with thanos engine (wrong one):

results with prometheus engine (ok one):

results in prometheus ui directly:

Sorry I had no time yet to try to reproduce with a "generic" exporter/query. I may try in the coming days.
I suspect group_left queries (maybe only with negative matchers?). Are those supported by the engine yet?

Thanks

[ahurtaud]

It is the top avg by aggregation that is "messing" the results. If I remove it in the thanos promql engine one. We can see it gets properly the 5 dimensions, but avg by wrongly aggregates them all:

[fpetkovski]

Hi @ahurtaud , thanks for reporting this. Which version of Thanos are you using?

[ahurtaud]

Hi @ahurtaud , thanks for reporting this. Which version of Thanos are you using?

oh sorry. v0.31.0-rc.1 which I hope is quite similar to 0.31.0. I can upgrade but I dont think it has diff regarding this (not even sure it has diff at all) :)

[fpetkovski]

I think they have the same engine version so updating won't help unfortunately. Is avg alone returning correct results?

[ahurtaud]

I think they have the same engine version so updating won't help unfortunately. Is avg alone returning correct results?

on the right-hand side of the query avg by alone works:

the left-hand side metrics does not contain this label key (thats the purpose of group_left here), but if we avg by another dimension label contained by the left-side metrics, it seems ok:

So I guess the issue would be when trying to "add" a label from an *_info metrics using group_left and then aggregating using this label. like explained in https://www.robustperception.io/why-info-style-metrics-have-a-value-of-1/

[fpetkovski]

Thanks for the detailed description, I'll try to reproduce the issue using these steps.