Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Put ingester disable chunk trimming a feature flag #6300

Merged
merged 6 commits into from
Nov 1, 2024

Conversation

yeya24
Copy link
Contributor

@yeya24 yeya24 commented Nov 1, 2024

Signed-off-by: Ben Ye benye@amazon.com<!-- Thanks for sending a pull request! Before submitting:

  1. Read our CONTRIBUTING.md guide
  2. Rebase your PR if it gets out of sync with master
    -->

What this PR does:

This changes #6270 to mark the behavior optional under a feature flag.

We noticed that query result got changed if disabling chunk trimming for the following test case, captured by fuzz test.

https://github.com/cortexproject/cortex/actions/runs/11621920479/job/32367371250?pr=6298

The failure seems correctly captured as timestamp function returns timestamp of the sample so if certain samples are not trimmed then the result changes.

I think the behavior is not intended but let's make disable chunk trimming a feature flag for now and users can enable it at their own risk

query_fuzz_test.go:188: case 296 results mismatch.
        range query: timestamp({__name__="test_series_3"} @ end() offset 3m21s)
        res1: {job="test"} =>
        1730421141.171 @[1730419941.189]
        1730421141.171 @[1730419971.189]
        1730421141.171 @[1730420001.189]
        1730421141.171 @[1730420031.189]
        1730421141.171 @[1730420061.189]
        1730421141.171 @[1730420091.189]
        1730421141.171 @[1730420121.189]
        1730421141.171 @[1730420151.189]
        1730421141.171 @[1730420181.189]
        1730421141.171 @[1730420211.189]
        1730421141.171 @[1730420241.189]
        1730421141.171 @[1730420271.189]
        1730421141.171 @[1730420301.189]
        1730421141.171 @[1730420331.189]
        1730421141.171 @[1730420361.189]
        1730421141.171 @[1730420391.189]
        1730421141.171 @[1730420421.189]
        1730421141.171 @[1730420451.189]
        1730421141.171 @[1730420481.189]
        1730421141.171 @[1730420511.189]
        1730421141.171 @[1730420[54](https://github.com/cortexproject/cortex/actions/runs/11621920479/job/32367371250?pr=6298#step:10:55)1.189]
        1730421141.171 @[1730420571.189]
        1730421141.171 @[1730420601.189]
        1730421141.171 @[1730420631.189]
        1730421141.171 @[1730420661.189]
        1730421141.171 @[1730420691.189]
        1730421141.171 @[1730420721.189]
        1730421141.171 @[1730420751.189]
        1730421141.171 @[1730420781.189]
        1730421141.171 @[1730420811.189]
        1730421141.171 @[1730420841.189]
        1730421141.171 @[1730420871.189]
        1730421141.171 @[1730420901.189]
        1730421141.171 @[1730420931.189]
        1730421141.171 @[1730420961.189]
        1730421141.171 @[1730420991.189]
        1730421141.171 @[1730421021.189]
        1730421141.171 @[1730421051.189]
        1730421141.171 @[1730421081.189]
        1730421141.171 @[1730421111.189]
        1730421141.171 @[1730421141.189]
        res2: {job="test"} =>
        1730420931.171 @[1730419941.189]
        1730420931.171 @[1730419971.189]
        1730420931.171 @[1730420001.189]
        1730420931.171 @[1730420031.189]
        1730420931.171 @[1730420061.189]
        1730420931.171 @[1730420091.189]
        1730420931.171 @[1730420121.189]
        1730420931.171 @[1730420151.189]
        1730420931.171 @[1730420181.189]
        1730420931.171 @[1730420211.189]
        1730420931.171 @[1730420241.189]
        1730420931.171 @[1730420271.189]
        1730420931.171 @[1730420301.189]
        1730420931.171 @[1730420331.189]
        1730420931.171 @[1730420361.189]
        1730420931.171 @[1730420391.189]
        1730420931.171 @[1730420421.189]
        1730420931.171 @[1730420451.189]
        1730420931.171 @[1730420481.189]
        1730420931.171 @[1730420511.189]
        1730420931.171 @[1730420541.189]
        1730420931.171 @[1730420[57](https://github.com/cortexproject/cortex/actions/runs/11621920479/job/32367371250?pr=6298#step:10:58)1.189]
        1730420931.171 @[1730420601.189]
        1730420931.171 @[1730420631.189]
        1730420931.171 @[1730420661.189]
        1730420931.171 @[1730420691.189]
        1730420931.171 @[1730420721.189]
        1730420931.171 @[1730420751.189]
        1730420931.171 @[1730420781.189]
        1730420931.171 @[1730420811.189]
        1730420931.171 @[1730420841.189]
        1730420931.171 @[1730420871.189]
        1730420931.171 @[1730420901.189]
        1730420931.171 @[1730420931.189]
        1730420931.171 @[17304209[61](https://github.com/cortexproject/cortex/actions/runs/11621920479/job/32367371250?pr=6298#step:10:62).189]
        1730420931.171 @[1730420991.189]
        1730420931.171 @[1730421021.189]
        1730420931.171 @[1730421051.189]
        1730420931.171 @[1730421081.189]
        1730420931.171 @[1730421111.189]
        1730420931.171 @[1730421141.189]
    query_fuzz_test.go:193: 
        	Error Trace:	/home/runner/work/cortex/cortex/integration/query_fuzz_test.go:193
        	Error:      	finished query fuzzing tests
        	Test:       	TestDisableChunkTrimmingFuzz
        	Messages:   	1 test cases failed

Which issue(s) this PR fixes:
Fixes #

Checklist

  • Tests updated
  • Documentation added
  • CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

Signed-off-by: Ben Ye <benye@amazon.com>
Signed-off-by: Ben Ye <benye@amazon.com>
Signed-off-by: Ben Ye <benye@amazon.com>
Signed-off-by: Ben Ye <benye@amazon.com>
Signed-off-by: Ben Ye <benye@amazon.com>
@pull-request-size pull-request-size bot added size/M and removed size/S labels Nov 1, 2024
Signed-off-by: Ben Ye <benye@amazon.com>
@yeya24 yeya24 force-pushed the disable-chunk-trimming branch from bce5678 to 5bbe234 Compare November 1, 2024 03:41
for {
expr = ps.WalkRangeQuery()
query = expr.Pretty(0)
if !strings.Contains(query, "timestamp") {
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We on purposely ignore timestamp to reduce flakiness as it is a know imcompatible behavior

Copy link
Member

@alanprot alanprot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks

@dosubot dosubot bot added the lgtm This PR has been approved by a maintainer label Nov 1, 2024
@yeya24 yeya24 merged commit 4063773 into cortexproject:master Nov 1, 2024
16 checks passed
@yeya24 yeya24 deleted the disable-chunk-trimming branch November 1, 2024 16:36
CharlieTLe pushed a commit to CharlieTLe/cortex that referenced this pull request Dec 3, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
component/ingester lgtm This PR has been approved by a maintainer size/M
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants