[improve][broker] add switch for enable/disable distribute bundles evenly in LoadManager

[HQebupt]

Motivation

When we use ModualLoadManager as LoadManager in the pulsar cluster, and the load balancer shedding strategy is ThresholdShedder, we found that the unloaded bundles might be loaded by another broker which has resource usage above average load, and make the new broker be overloaded again, which frequently causes unloading bundles.

loadManagerClassName=org.apache.pulsar.broker.loadbalance.impl.ModularLoadManagerImpl
loadBalancerLoadSheddingStrategy=org.apache.pulsar.broker.loadbalance.impl.ThresholdShedder
loadBalancerBrokerThresholdShedderPercentage=10

We found that it tooks more than 10 hours and 400+ times to doing load shedding in the cluster. And the cluster continue to unload bundles when some of topics are lots of traffic in a short time. Below is the unload metric.

The key reason is that some brokers with lower load but more bundles can not be candidate due to distributing bundles evenly in LoadManager by force. Most of brokers are filtered out by the strategy, only 1 or 2 brokers can be candidate in the total 136 brokers as follows.

It could be much better to disable distribute bundles evenly in LoadManager, which can select the broker from those having resource usage below average load, so it can prevent the least loaded broker from quickly becoming heavily loaded.

Therefore, it recommend that enable distribute bundles evenly among all brokers by customers according to user scenarios .

After disabling distribute bundles evenly in LoadManager, the brokers with lower load but more bunldes can be candidate. It reduced the unload times and the cluster is stable with even load on each broker as follows.

Modifications

• Add a field property loadBalancerDistributeBundlesEvenlyEnabled for Load Balancer in ServiceConfiguration, keep it to be true as default.
• Add a switch to enable distribute bundles evenly in ModularLoadManager

Verifying this change

• Make sure that the change passes the CI checks.

This change is a trivial rework / code cleanup without any test coverage.

Does this pull request potentially affect one of the following parts:

If yes was chosen, please highlight the changes

• Dependencies (does it add or upgrade a dependency): (no)
• The public API: (no)
• The schema: (no)
• The default values of configurations: (no)
• The wire protocol: (no)
• The rest endpoints: (no)
• The admin cli options: (no)
• Anything that affects deployment: (no)

Documentation

Check the box below and label this PR (if you have committer privilege).

Need to update docs?

• no-need-doc

• doc-not-needed

[HQebupt]

/pulsarbot run-failure-checks

[gaozhangmin]

Good work, We encountered this issue early too.

[Jason918]

@hangc0276 PTAL

[hangc0276]

Good job.

[heesung-sn]

Discussing 2.10 cherry-pick here: https://lists.apache.org/thread/2mq3h9gpqv1b4zyyp2cddfltlqz3wtg0