storage: reset compaction window start offset when offset map isn't built
#24323
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
… built
Consider the following situation:
[0] [1] [2] [3] [4] [5]
^
|----- _last_compaction_window_start_offset
In the event that segment [2] cannot have its keys completely added
to the offset map due to memory constraints, if we do not reset
the window start offset, compaction will become somewhat stuck as it
repeatedly tries to index segment [2] in future compaction rounds.
By resetting the window's value in this situation, we can allow
compaction to continue to make forward progression.
|
ducktape was retried in https://buildkite.com/redpanda/redpanda/builds/58828#01936a7e-ff7b-4c6d-8aa3-93d21eec7fd6 |
|
This may be superseded by #24367 |
andrwng
approved these changes
Dec 3, 2024
|
/backport v24.3.x |
dotnwat
reviewed
Dec 3, 2024
Comment on lines
-554
to
+555
| segment_set segs(std::move(buf)); | ||
| if (segs.empty()) { | ||
| return segs; | ||
| } | ||
|
|
||
| segment_set segs(std::move(buf)); |
The idea struck me when I was on a walk actually 🚶 The test case came next to confirm the theory, then the fix |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The behavior of the sliding window considered for compaction was changed in this PR: #23380.
However, consider the following situation:
In the event that segment [2] cannot have its keys completely added to the offset map due to memory constraints, if we do not reset the window start offset, compaction will become somewhat stuck as it repeatedly tries to index segment [2] in future compaction rounds.
By resetting the window's value in this situation, we can allow compaction to continue to make forward progression.
Backports Required
Release Notes
Bug Fixes