-
Notifications
You must be signed in to change notification settings - Fork 3.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
release-23.2: allocator: lower default IO overload threshold #115129
Conversation
cfdfad6
to
dd6920e
Compare
b534b49
to
7ef5660
Compare
Thanks for opening a backport. Please check the backport criteria before merging:
If your backport adds new functionality, please ensure that the following additional criteria are satisfied:
Also, please add a brief release justification to the body of your PR to justify this |
It looks like your PR touches production code but doesn't add or edit any test code. Did you consider adding tests to your PR? 🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Reviewable status: complete! 1 of 0 LGTMs obtained (waiting on @kvoli)
The IO overload threshold determines whether a store will be excluded as a target for rebalancing replicas, or leases onto it. The previous thresholds were `0.8` for replica rebalancing (`kv.allocator.replica_io_overload_threshold`) and `0.5` for lease transfers (`kv.allocator.lease_io_overload_threshold`). The previous settings were selected when Admission Control (AC) attempted to stabilize IO overload at a threshold of 1.0. AC will now typically stabilize a store to a threshold of 0.5. Lower the thresholds to `0.4` for replica rebalancing and `0.3` for lease transfers. Note that a store needs to both exceed the absolute threshold mentioned above, as well as be greater than `+10% * mean` of equivalent rebalancing candidates. Resolves: #112497 Release note: None
7ef5660
to
44a97a2
Compare
Rebased on |
TestSQLStatsIdleLatencies failed on extended stress race. Failure looks unrelated, merging. |
Backport 1/1 commits from #113667 on behalf of @kvoli.
/cc @cockroachdb/release
The IO overload threshold determines whether a store will be excluded as a target for rebalancing replicas, or leases onto it. The previous thresholds were
0.8
for replica rebalancing(
kv.allocator.replica_io_overload_threshold
) and0.5
for lease transfers (kv.allocator.lease_io_overload_threshold
).The previous settings were selected when Admission Control (AC) attempted to stabilize IO overload at a threshold of 1.0. AC will now typically stabilize a store to a threshold of 0.5.
Lower the thresholds to
0.4
for replica rebalancing and0.3
for lease transfers. Note that a store needs to both exceed the absolute threshold mentioned above, as well as be greater than+10% * mean
of equivalent rebalancing candidates.Resolves: #112497
Release note: None
Release justification: Brings thresholds in line with AC throttling in 23.2. Without this update, the default IO overload value will rarely/never be hit to more aggressive AC throttling.