Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[improve][broker] PIP-364: Introduce a new load balance algorithm AvgShedder #22949

Merged
merged 31 commits into from
Jul 11, 2024
Merged
Show file tree
Hide file tree
Changes from 26 commits
Commits
Show all changes
31 commits
Select commit Hold shift + click to select a range
8f79af0
add AvgShedder.
thetumbled Jun 20, 2024
be38d4d
fix typo.
thetumbled Jun 20, 2024
a153eb0
Merge remote-tracking branch 'apache/master' into AvgShedder
thetumbled Jun 27, 2024
7944159
replace bundle name + random number with random number only.
thetumbled Jun 27, 2024
4cf4f2b
add test code.
thetumbled Jun 28, 2024
99caa97
add test code.
thetumbled Jun 28, 2024
b5e444b
update conf.
thetumbled Jun 28, 2024
6d4678a
fix typo.
thetumbled Jun 28, 2024
59907a8
add test code.
thetumbled Jun 28, 2024
0ac2b99
fix checkstyle.
thetumbled Jun 28, 2024
1bf6346
fix header.
thetumbled Jun 28, 2024
d2dbf6c
fix checkstyle.
thetumbled Jun 28, 2024
1b60887
fix checkstyle.
thetumbled Jun 28, 2024
c2f6123
Update pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalan…
thetumbled Jul 5, 2024
00e5625
Update pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalan…
thetumbled Jul 5, 2024
1c5afe9
Update pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalan…
thetumbled Jul 5, 2024
4f4ae3e
improve.
thetumbled Jul 5, 2024
64de192
fix check style.
thetumbled Jul 5, 2024
bec01e6
throw exception.
thetumbled Jul 5, 2024
e438be4
fix checkstyle.
thetumbled Jul 6, 2024
4cfbbda
Update pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalan…
thetumbled Jul 8, 2024
5eac740
fix deprecated conf.
thetumbled Jul 8, 2024
eca4027
improve log.
thetumbled Jul 8, 2024
2fddb77
refactor to some sub methods.
thetumbled Jul 8, 2024
f731a25
refactor map.
thetumbled Jul 8, 2024
11d56c8
fix check style.
thetumbled Jul 10, 2024
0fc8bc5
avoid create list manually.
thetumbled Jul 11, 2024
65b8941
reduce one find operation.
thetumbled Jul 11, 2024
dfb8765
add test to cover getExpectedBroker.
thetumbled Jul 11, 2024
4a0f64c
avoid static Random object .
thetumbled Jul 11, 2024
26f5479
make selectedBundlesCache locally.
thetumbled Jul 11, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 19 additions & 0 deletions conf/broker.conf
Original file line number Diff line number Diff line change
Expand Up @@ -1414,6 +1414,25 @@ loadBalancerBundleUnloadMinThroughputThreshold=10
# Time to wait for the unloading of a namespace bundle
namespaceBundleUnloadingTimeoutMs=60000

# configuration for AvgShedder, a new shedding and placement strategy
# The low threshold for the difference between the highest and lowest loaded brokers.
loadBalancerAvgShedderLowThreshold = 15

# The high threshold for the difference between the highest and lowest loaded brokers.
loadBalancerAvgShedderHighThreshold = 40

# The number of times the low threshold is triggered before the bundle is unloaded.
loadBalancerAvgShedderHitCountLowThreshold = 8

# The number of times the high threshold is triggered before the bundle is unloaded.
loadBalancerAvgShedderHitCountHighThreshold = 2

# In the UniformLoadShedder and AvgShedder strategy, the maximum unload ratio.
# For AvgShedder, recommend to set to 0.5, so that it will distribute the load evenly
# between the highest and lowest brokers.
maxUnloadPercentage = 0.2


### --- Load balancer extension --- ###

# Option to enable the debug mode for the load balancer logics.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2395,21 +2395,51 @@ The max allowed delay for delayed delivery (in milliseconds). If the broker rece
@FieldContext(
dynamic = true,
category = CATEGORY_LOAD_BALANCER,
doc = "In the UniformLoadShedder strategy, the minimum message that triggers unload."
doc = "The low threshold for the difference between the highest and lowest loaded brokers."
)
private int loadBalancerAvgShedderLowThreshold = 15;

@FieldContext(
dynamic = true,
category = CATEGORY_LOAD_BALANCER,
doc = "The high threshold for the difference between the highest and lowest loaded brokers."
)
private int loadBalancerAvgShedderHighThreshold = 40;

@FieldContext(
dynamic = true,
category = CATEGORY_LOAD_BALANCER,
doc = "The number of times the low threshold is triggered before the bundle is unloaded."
)
private int loadBalancerAvgShedderHitCountLowThreshold = 8;

@FieldContext(
dynamic = true,
category = CATEGORY_LOAD_BALANCER,
doc = "The number of times the high threshold is triggered before the bundle is unloaded."
)
private int loadBalancerAvgShedderHitCountHighThreshold = 2;

@FieldContext(
dynamic = true,
category = CATEGORY_LOAD_BALANCER,
doc = "In the UniformLoadShedder and AvgShedder strategy, the minimum message that triggers unload."
)
private int minUnloadMessage = 1000;

@FieldContext(
dynamic = true,
category = CATEGORY_LOAD_BALANCER,
doc = "In the UniformLoadShedder strategy, the minimum throughput that triggers unload."
doc = "In the UniformLoadShedder and AvgShedder strategy, the minimum throughput that triggers unload."
)
private int minUnloadMessageThroughput = 1 * 1024 * 1024;

@FieldContext(
dynamic = true,
category = CATEGORY_LOAD_BALANCER,
doc = "In the UniformLoadShedder strategy, the maximum unload ratio."
doc = "In the UniformLoadShedder and AvgShedder strategy, the maximum unload ratio."
+ "For AvgShedder, recommend to set to 0.5, so that it will distribute the load "
+ "evenly between the highest and lowest brokers."
)
private double maxUnloadPercentage = 0.2;

Expand Down
Loading
Loading