Allow partition balancing in presence of moving partitions #10724
Conversation
No need for separate movement and cancellations_planned
The method checks if the set of replica nodes for the partition changed (has_changes() method by contrast takes shard changes into account too).
The purpose of this class is to present a unified interface for the functions that do the balancing (instead of them iterating over topic table and reallocating replicas manually).
|
/ci-repeat 2 |
|
Would it be possible to add a commit comment on how moving partitions are handled ? |
If we are going to allow balancer operation in presence of moving partitions, we must take into account their contribution to assigned/released disk sizes: moving partitions will add some disk space to new replicas and release some disk space on old ones; and cancelling partitions will release some disk space on target replicas.
Split the partition class into 3 variants, depending on which actions we can do to it: * reassignable_partition - partition is not in progress, we can move replicas * moving_partition - partition movement is in progress, we can cancel it * immutable_partition - we can do nothing only report failure
Our handling of moving partitions is pretty conservative - we don't touch them, except when we are cancelling movements to unavailable nodes (which are unlikely to ever finish, so this is required to make progress).
ztlpn
force-pushed
the
pb-moving-partitions
branch
from
daf5dac to
ef07885
Compare
|
PartitionBalancerTest.test_fuzz_admin_ops suspiciously failed but looks like this is just an instance of #9315 - partition movements are being scheduled, they are just slow in debug mode. |
|
/ci-repeat 2 |
|
/ci-repeat 2 |
For some reason (possibly linked to this PR) the rate of partition balancer tests failures has recently increased quite a bit. This doesn't look like a bug, partition movements simply can't proceed because RPC append_entries calls are too slow. To unblock the PR, skip all partition balancing tests in debug mode.
|
So I ran all partition balancer tests 50 times, and they only failed once (#9788), which looks pretty good in my opinion. To unblock this PR (and lower the number of ci failures in general), I propose to disable running partition balancer tests (and possibly other tests that move a lot of partitions) in debug mode. Some of them are already disabled anyway. |
partitioninterface for use in balancing/repair operators - thepartitionvariant makes possible only those actions that make sense - e.g. a moving partition can only be cancelled and a partition already in the process of cancellation can only be observed.Backports Required
Release Notes
Improvements