[improve] [broker] make system topic distribute evenly. #22953
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
PTAL, thanks. @heesung-sn @Demogorgon314 @congbobo184 @liangyepianzhou @lhotari |
thetumbled
changed the title
heesung-sn
approved these changes
Jun 22, 2024
Demogorgon314
approved these changes
Jun 24, 2024
|
/pulsarbot rerun-failure-checks |
lhotari
approved these changes
Jun 24, 2024
lhotari
pushed a commit
that referenced
this pull request
Jun 24, 2024
(cherry picked from commit 263c694)
lhotari
pushed a commit
that referenced
this pull request
Jun 24, 2024
(cherry picked from commit 263c694)
lhotari
pushed a commit
that referenced
this pull request
Jun 24, 2024
(cherry picked from commit 263c694)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Motivation
The loading of TC onto which broker depends on which broker the
transaction_coordinator_assign-partition-${TC ID}is loaded onto.In order to improve service availability, we need to distribute TC evenly across the entire cluster as much as possible, avoiding all TCs from being loaded onto a few machines.
For example, if all 16 TCs are loaded onto two or three brokers, restarting the broker during cluster rolling upgrades will result in a large number of TCs becoming unavailable simultaneously, which will significantly affect the transaction client.
All shedding algorithm pick up the bundles based on throughput or message rate from the top to the bottom. However, the system topic has almost no traffic, which means system topic can't be re-distribute evenly with shedding algo. Placement strategy always want to distribute bundles with the least load broker, so there is high possibility that the bundles of system topic are loaded onto few of brokers.
For example:
count(pulsar_txn_created_total) by (instance)One single broker serve for all 16 TCs.
Modifications
Distribute bundles of system topic evenly forcefully.
Verifying this change
(Please pick either of the following options)
This change is a trivial rework / code cleanup without any test coverage.
Does this pull request potentially affect one of the following parts:
If the box was checked, please highlight the changes
Documentation
docdoc-requireddoc-not-neededdoc-completeMatching PR in forked repository
PR in forked repository: thetumbled#60