[v23.1.x] storage: fix race between segment.ms and appends #12715
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This adds a mechanism for adding artificial stress to Redpanda. A stress_payload is defined as one or multiple fibers that run in the background, repeatedly counting up to some bounded random values/delays, until stopped. This will be exposed via admin endpoint in a subsequent patch. It's expected that only one stress payload run at a time. The public interface (stress_fiber_manager) reflects that. The public interface exposes configuration by delay or by count. In practice, I found the former helpful in reliably injecting reactor stalls, and the latter helpful in having finer grained control over how many cycles are "wasted". This will be useful in exercising timings and reorderings of tasks that happen when under load. (cherry picked from commit d42e0ac)
andrwng
force-pushed
the
v23.1.x-storage-append-segment-ms-race
branch
4 times, most recently
from
2e7ea25 to
3313a71
Compare
CONFLICTS: - replace http with httpd This adds an admin endpoint to kick off some number of fibers to run busy loops (i.e. loops that just hog CPU time without doing anything meaningful). The goal is to put redpanda in a state that emulates heavy load. (cherry picked from commit 4e46473)
In planning some new tests with tiered storage, I've been wanting to write single-node unit tests that exercise produce and consume paths from the Kafka layer down to underlying storage. It is particularly important for tiered storage to test this bridge since the Kafka layer is where we make key decisions regarding what storage subsystems to direct to. This commit adds some utilities to facilitate this: it wraps the kafka::client::transport and abstracts away details of Kafka requests, instead exposing a simple key-value interface for producing and consuming. (cherry picked from commit f578b4f)
The test consumer can only read the first record batch. This commit fixes this so it could read the entire log. It also adds extra logging. (cherry picked from commit 21fe29b)
The consume utility was previously not throwing on per-partition errors, which is useful see about in tests. (cherry picked from commit b89a058)
CONFLICT: - makes the change for mem_log_impl too Because we are splitting the compact() function into separate functions for compaction() and gc(), we need a name that can be used to represent the most common function which is the combination of the two without updating every call site to call both primitives instead. Housekeeping is the prefect name. So we renamed the previous housekeeping method to reflect its exact functionality. Notice that in the comments it is clear that in the future apply_segment_ms will likely be folded directly into the housekeeping functionality itself, but this commit does not attempt to address those TODO items. Signed-off-by: Noah Watkins <noahwatkins@gmail.com> (cherry picked from commit 61de825)
This will make it easier to tweak locking in the area. I also removed the `_bytes_left_in_segment > 0` stop condition, which is true if `!needs_to_roll_log(term)` (cherry picked from commit d2a7f18)
I found it easier to reason about the implementation when considering whether the current segment is appendable, rather than whether we need to roll. In a subsequent commit, I will update the condition further to include whether the current segment has an appender, which isn't quite as intuitive when thinking about rolling. This patch updates the method and condition to reflect whether the segment is appendable. (cherry picked from commit 8a77b82)
Following a coroutinization and a rename, it's now clearer that the while loop in the method can further be simplified. This patch cleans it up. (cherry picked from commit f4c4593)
H/T to VladLazar for a similar change that inspired this one: VladLazar@682aea5 The problem statement: ``` We have seen a couple of races between the application of `segment.ms` and the normal append path. They had the following pattern in common: 1. application of `segment.ms` begins 2. a call to `segment::append` is interleaved 3. the append finishes first and and advances the dirty offsets, which the rolling logic in `segment.ms` does not expect -- or -- 4. `segment.ms` releases the current appender while the append is ongoing, which the append logic does not expect ``` The proposed fix was to introduce a new appender lock to the segment, and ensure that it is held while appending an while segment.ms rolling. This addressed problem #3, but wasn't sufficient to address redpanda-data#4. The issue with introducing another lock to the segment is that the unexpected behavior when appending to a segment happens in the context of an already referenced segment. I.e. the appending fiber may proceed to reference an appender, only for it to be destructed by the housekeeping fiber before segment::append() is called, resulting in a segfault. This patch extends usage of the existing disk_log_impl::_segments_rolling_lock to cover the entire duration of append (i.e. not just the underlying segment::append() call), ensuring that segment.ms rolls and appends are mutually exclusive. (cherry picked from commit 78a9749)
Pulls out some fixture methods to use in e2e fixture test. The original commit includes a cloud storage test and some changes to cloud storage that I've left out.
CONFLICT: - does not bring in test_basic_delete_around_batch case or the new cloud_storage_e2e_test.cc This will make the type easier to extend, and make test code slightly easier to grok immediately. (cherry picked from commit 8fd082a)
It has been useful in tests to generate a specific number of segments. This pulls out some utilities to be useful in more upcoming tests. (cherry picked from commit 21311a0)
This introduces a single-node fixture test that races segment.ms rolling with appends. This previously resulted in a crash. (cherry picked from commit 1ae1527)
andrwng
force-pushed
the
v23.1.x-storage-append-segment-ms-race
branch
from
3313a71 to
c089746
Compare
|
CI failure: #9697 |
abhijat
approved these changes
Aug 17, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Backport of #12565
Pulls in:
Backports Required
Release Notes
Bug fixes