Fix panic in instant query splitting when using unwrapped rate #6348

chaudum · 2022-06-09T08:34:47Z

What this PR does / why we need it:

The range aggregation rate() supports both log ranges and unwrapped ranges, e.g.

rate({app="foo"} [$__interval]) and rate({app="foo"} | unwrap bar [$__interval])

Since rate() was split into multiple count_over_time() over total duration, but count_over_time() does not support unwrap, unwrapped rate queries caused panics.

This fix changes the splitting of rate({app="foo"} | unwrap bar [$__interval] into multiple sum_over_time() over total duration.

Signed-off-by: Christian Haudum christian.haudum@gmail.com

Which issue(s) this PR fixes:

Fixes #6344

Special notes for your reviewer:

Checklist

Documentation added
Tests updated
Is this an important fix or new feature? Add an entry in the CHANGELOG.md.
Changes that require user attention or interaction to upgrade are documented in docs/sources/upgrading/_index.md

grafanabot · 2022-06-09T08:40:24Z

./tools/diff_coverage.sh ../loki-main/test_results.txt test_results.txt ingester,distributor,querier,querier/queryrange,iter,storage,chunkenc,logql,loki

Change in test coverage per package. Green indicates 0 or positive change, red indicates that test coverage for a package fell.

+           ingester	0%
+        distributor	0%
+            querier	0%
+ querier/queryrange	0%
+               iter	0%
+            storage	0%
+           chunkenc	0%
-              logql	-0.1%
+               loki	0%

ssncferreira · 2022-06-09T08:43:25Z

pkg/logql/rangemapper.go

+		if expr.Left.Unwrap != nil {
+			// rate({app="foo"} | unwrap bar [2m])
+			// => (sum without (sum_over_time({app="foo"}[1m]) ++ sum_over_time({app="foo"}[1m]) offset 1m) / 120)
+			op = syntax.OpRangeTypeSum


I'm a bit confused 😕 are we sure this is equivalent to sum_over_time?
In this case, it would also be useful to add a test to pkg/logql/downstream_test.go to compare with the normal engine

Yes, I should add a downstream test 👍

Instead of counting log lines for the rate, it sums the individual unwrap values.

🤔 Maybe not ...

I think I understood unwrapped rate incorrectly. It does not sum the unwrapped values, but treats them as counter, which is treated as reset, when a subsequent value is smaller than the preceding one.
See
https://github.com/grafana/loki/blob/main/pkg/logql/range_vector.go#L227
and
https://github.com/grafana/loki/blob/main/pkg/logql/range_vector.go#L248-L256

Waiting for #6361 to revert rate calculation.

@ssncferreira Rebased this PR with main and now the added tests pass :)

The range aggregation `rate()` supports both log ranges and unwrapped ranges, e.g. `rate({app="foo"} [$__interval])` and `rate({app="foo"} | unwrap bar [$__interval])` Since `rate()` was split into multiple `count_over_time()` over total duration, but `count_over_time()` does not support `unwrap`, unwrapped rate queries caused panics. This fix changes the splitting of `rate({app="foo"} | unwrap bar [$__interval]` into multiple `sum_over_time()` over total duration. Fixes #6344 Signed-off-by: Christian Haudum <christian.haudum@gmail.com>

Signed-off-by: Christian Haudum <christian.haudum@gmail.com>

grafanabot · 2022-06-19T19:32:53Z

./tools/diff_coverage.sh ../loki-main/test_results.txt test_results.txt ingester,distributor,querier,querier/queryrange,iter,storage,chunkenc,logql,loki

Change in test coverage per package. Green indicates 0 or positive change, red indicates that test coverage for a package fell.

+           ingester	0%
+        distributor	0.3%
+            querier	0%
+ querier/queryrange	0%
+               iter	0%
+            storage	0%
+           chunkenc	0%
+              logql	0%
+               loki	0%

ssncferreira

LGTM! Great job 🚀
Just a small nit on the test to ease readability

pkg/logql/rangemapper_test.go

Co-authored-by: Susana Ferreira <susana.ferreira@grafana.com>

grafanabot · 2022-06-20T11:20:20Z

./tools/diff_coverage.sh ../loki-main/test_results.txt test_results.txt ingester,distributor,querier,querier/queryrange,iter,storage,chunkenc,logql,loki

Change in test coverage per package. Green indicates 0 or positive change, red indicates that test coverage for a package fell.

+           ingester	0%
+        distributor	0%
+            querier	0%
+ querier/queryrange	0%
+               iter	0%
+            storage	0%
+           chunkenc	0%
+              logql	0%
+               loki	0%

* Fix panic in instant query splitting when using unwrapped rate The range aggregation `rate()` supports both log ranges and unwrapped ranges, e.g. `rate({app="foo"} [$__interval])` and `rate({app="foo"} | unwrap bar [$__interval])` Since `rate()` was split into multiple `count_over_time()` over total duration, but `count_over_time()` does not support `unwrap`, unwrapped rate queries caused panics. This fix changes the splitting of `rate({app="foo"} | unwrap bar [$__interval]` into multiple `sum_over_time()` over total duration. Fixes #6344 Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Add tests Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Integrate review feedback Co-authored-by: Susana Ferreira <susana.ferreira@grafana.com> Co-authored-by: Susana Ferreira <susana.ferreira@grafana.com> (cherry picked from commit 798677a)

#6557) * Fix panic in instant query splitting when using unwrapped rate The range aggregation `rate()` supports both log ranges and unwrapped ranges, e.g. `rate({app="foo"} [$__interval])` and `rate({app="foo"} | unwrap bar [$__interval])` Since `rate()` was split into multiple `count_over_time()` over total duration, but `count_over_time()` does not support `unwrap`, unwrapped rate queries caused panics. This fix changes the splitting of `rate({app="foo"} | unwrap bar [$__interval]` into multiple `sum_over_time()` over total duration. Fixes #6344 Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Add tests Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Integrate review feedback Co-authored-by: Susana Ferreira <susana.ferreira@grafana.com> Co-authored-by: Susana Ferreira <susana.ferreira@grafana.com> (cherry picked from commit 798677a) Co-authored-by: Christian Haudum <christian.haudum@gmail.com>

chaudum requested a review from a team as a code owner June 9, 2022 08:34

pull-request-size bot added the size/M label Jun 9, 2022

chaudum requested review from owen-d, ssncferreira and cstyan June 9, 2022 08:35

ssncferreira reviewed Jun 9, 2022

View reviewed changes

chaudum mentioned this pull request Jun 9, 2022

[RFC]: Sum values in unwrapped rate aggregation instead of treating them as counter #6351

Open

chaudum added 2 commits June 19, 2022 21:20

Add tests

ef72a4c

Signed-off-by: Christian Haudum <christian.haudum@gmail.com>

chaudum force-pushed the chaudum/issue-6344 branch from 6b1b1cf to ef72a4c Compare June 19, 2022 19:27

chaudum requested a review from ssncferreira June 19, 2022 19:27

ssncferreira approved these changes Jun 20, 2022

View reviewed changes

pkg/logql/rangemapper_test.go Outdated Show resolved Hide resolved

Integrate review feedback

ae6c44f

Co-authored-by: Susana Ferreira <susana.ferreira@grafana.com>

slim-bean merged commit 798677a into main Jun 22, 2022

slim-bean deleted the chaudum/issue-6344 branch June 22, 2022 14:38

ssncferreira added the backport release-2.6.x Tag a PR with this label to create a PR which cherry pics it into the release-2.6.x branch label Jun 30, 2022

grafanabot mentioned this pull request Jun 30, 2022

[release-2.6.x] Fix panic in instant query splitting when using unwrapped rate #6557

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix panic in instant query splitting when using unwrapped rate #6348

Fix panic in instant query splitting when using unwrapped rate #6348

chaudum commented Jun 9, 2022

grafanabot commented Jun 9, 2022

ssncferreira Jun 9, 2022

chaudum Jun 9, 2022

chaudum Jun 9, 2022

chaudum Jun 9, 2022

chaudum Jun 9, 2022

chaudum Jun 10, 2022

chaudum Jun 19, 2022

grafanabot commented Jun 19, 2022

ssncferreira left a comment

grafanabot commented Jun 20, 2022

Fix panic in instant query splitting when using unwrapped rate #6348

Fix panic in instant query splitting when using unwrapped rate #6348

Conversation

chaudum commented Jun 9, 2022

grafanabot commented Jun 9, 2022

ssncferreira Jun 9, 2022

Choose a reason for hiding this comment

chaudum Jun 9, 2022

Choose a reason for hiding this comment

chaudum Jun 9, 2022

Choose a reason for hiding this comment

chaudum Jun 9, 2022

Choose a reason for hiding this comment

chaudum Jun 9, 2022

Choose a reason for hiding this comment

chaudum Jun 10, 2022

Choose a reason for hiding this comment

chaudum Jun 19, 2022

Choose a reason for hiding this comment

grafanabot commented Jun 19, 2022

ssncferreira left a comment

Choose a reason for hiding this comment

grafanabot commented Jun 20, 2022