Volume Replacement causes high latency/jitter to the clients due to excessive IO and CPU usage (#6069)

[ti-chi-bot]

This is an automated cherry-pick of #6069

What problem does this PR solve?

Volume Replacement causes high latency/jitter to the clients due to excessive IO and CPU usage because schedule.store-limit.xyz.remove-peer limit is being set to unlimited. Effectively the limit becomes num_available_tikv_pods * schedule.store-limit.xyz.add-peer. The problem is the TIKV store is still serving active leaders causing a high latency impact to clients.

This fix evicts the leaders from the TIKV pod/store before proceeding with volume replacement on the pod.

Ref: tikv/pd#4099

What is changed and how does it work?

Evict region leaders from the TIKV store before proceeding with volume replacement

Code changes

• [X ] Has Go code change
• Has CI related scripts change

Tests

• Unit test
• E2E test
• Manual test
• No code

Side effects

• Breaking backward compatibility
• Other side effects:

Related changes

• Need to cherry-pick to the release branch
• Need to update the documentation

Release Notes

Please refer to Release Notes Language Style Guide before writing the release note.

Fix high latency/jitter due to excessive IO and CPU usage during Volume Replacement.

[ti-chi-bot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign hanlins for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[sre-bot]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}