[backport] [v23.1.x] raft: dynamic learner recovery rate #10505
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This implementation periodically coordinates with other shards and reallocates any unused rate from idle shards among shards with deficit. More details in raft::coordinated_recovery_throttle. (cherry picked from commit 8087915)
Update all the call sites that used to rely on legacy raft::recovery_throttle implementation to use the new implementation. Includes callsites in test fixtures. (cherry picked from commit 4c406ae)
Adds unit tests for the new recovery throttle implementation. The unit tests disable the coordination and instead invoke it on demand to have more control of the test state. The logic encapsulated in a test fixture. (cherry picked from commit e2ac37a)
Adds the following per shard metrics. (1) partition_movement_available_bandwidth (2) partition_movement_assigned_bandwidth (1) is available unused bandwidth on the shard in bytes for callers (2) is total assigned bandwidth on the shard in bytes in the last coordination tick. (cherry picked from commit dc8d2f0)
(cherry picked from commit 82fdf3e)
With the changes in the patch, the new recovery rate is applied in the next coordinator tick (may take as long as 1s). Reflect that wait in the tests that update the rate during testing. (cherry picked from commit 91baa42)
|
/ci-repeat 5 |
mmaslankaprv
approved these changes
May 5, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Implements a new recovery throttle that coordinates the available bandwidth among shards and reallocates it dynamically based on load. It replaces the current static recovery throttle that assigns bandwidth per shard up front.
Ran this patch on a 5 node cluster in ec2 with 7-8tb of a test topic. Ran a targeted test for reconfiguring a 500gb partition and made sure it consumed the entire bandwidth on the node.
Also decommissioned a node and reconfigurations progressed much faster with higher bandwidth share than they otherwise would with static rate assignment. Here is what rate allocation looks like visually for a node. Decommissioning starts at 21:30 and 21:40-21:45 is when we start to have laggers that benefit from reassignment as they can consume the total available bandwidth. 22:05 is when the decommission finished and the rate distribution if back to fair rate.
Backport of PR #10339
Fixes #10494
Backports Required
Release Notes
Improvements