storage: increase the load-based splitting QPS threshold from 250 to 2500 #39687
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
This change is |
…2500 Recent improvements in write efficiency and batching lead to a reevaluation of the reasons for load-based splitting. Intuitively fewer splits ought to offer increased batching opportunities while more splits ought to offer increased concurrency. Above a given number it is not obvious that increased concurrency will translate effectively to increased parallelism. Experimental evidence shows that the right threshold for load-based splitting is now closer to 2500 than 250. It also shows that over-splitting can have negative effects on latency and throughput. Load-based splitting remains important additionally for the opportunity it provides to balance load. Load-balancing however is not currently a part of the splitting heuristic. Release note (performance improvement): Adjust load-based splitting QPS threshold to avoid over-splitting.
Before this PR we always ran our KV benchmarks with 1000 manual splits. This is almost certainly too many. We preserve this value for the historical continuity of the tests but add new configurations which do no manual splitting. Release note: None
ajwerner
force-pushed
the
ajwerner/change-load-based-split-qps-threshold
branch
from
b5f8d1b
to
0099761
Compare
nvanbenschoten
approved these changes
Aug 16, 2019
|
TFTR! I ran a quick benchmark of kv0 with different QPS thresholds and here's what I got: Seems to support that we've picked a good number. bors r+ |
craig bot
pushed a commit
that referenced
this pull request
Aug 16, 2019
39687: storage: increase the load-based splitting QPS threshold from 250 to 2500 r=ajwerner a=ajwerner Recent improvements in write efficiency and batching lead to a re-evaluation of the reasons for load-based splitting. Intuitively fewer splits ought to offer increased batching opportunities while more splits ought to offer increased concurrency. Above a given number it is not obvious that increased concurrency will translate effectively to increased parallelism. Experimental evidence shows that the right threshold for load-based splitting is now closer to 2500 than 250. It also shows that over-splitting can have negative effects on latency and throughput. Load-based splitting remains important additionally for the opportunity it provides to balance load. Load-balancing however is not currently a part of the splitting heuristic. The second commit in the PR adds roachtests which do not perform any manual splits. Release note (performance improvement): Adjust load-based splitting QPS threshold to avoid over-splitting. Co-authored-by: Andrew Werner <ajwerner@cockroachlabs.com>
Build succeeded |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Recent improvements in write efficiency and batching lead to a re-evaluation
of the reasons for load-based splitting. Intuitively fewer splits ought to
offer increased batching opportunities while more splits ought to offer
increased concurrency. Above a given number it is not obvious that
increased concurrency will translate effectively to increased parallelism.
Experimental evidence shows that the right threshold for load-based splitting
is now closer to 2500 than 250. It also shows that over-splitting can have
negative effects on latency and throughput.
Load-based splitting remains important additionally for the opportunity it
provides to balance load. Load-balancing however is not currently a part of the
splitting heuristic.
The second commit in the PR adds roachtests which do not perform any manual splits.
Release note (performance improvement): Adjust load-based splitting QPS
threshold to avoid over-splitting.