Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

kvserver: load based splitting should be on a more granular metric #83519

Closed
kvoli opened this issue Jun 28, 2022 · 2 comments · Fixed by #96128
Closed

kvserver: load based splitting should be on a more granular metric #83519

kvoli opened this issue Jun 28, 2022 · 2 comments · Fixed by #96128
Labels
A-kv Anything in KV that doesn't belong in a more specific category. A-kv-distribution Relating to rebalancing and leasing. C-enhancement Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception) T-kv KV Team

Comments

@kvoli
Copy link
Collaborator

kvoli commented Jun 28, 2022 •
edited by exalate-issue-sync bot
Loading

Currently we load based split when the number of requests (requests in all batch requests) exceeds the qps split threshold (default 2500).

Requests are not uniform however and may be orders of magnitude more expensive than each other e.g. scan of 3k rows vs point read.

We should split on a more granular metric than requests per second. Such as the number of bytes read + written to a range per second.

The threshold should also be adjusted with testing to a new suitable default.

One alternative is to use the number of key accesses as opposed to the number of bytes. This method would not account for large values being read or written, however would set each access at a fixed cost.

Jira issue: CRDB-17122

Epic CRDB-18661
@kvoli kvoli added C-enhancement Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception) A-kv-distribution Relating to rebalancing and leasing. A-kv Anything in KV that doesn't belong in a more specific category. labels Jun 28, 2022
@blathers-crl blathers-crl bot added the T-kv KV Team label Jun 28, 2022
@tbg
Copy link
Member

tbg commented Jun 28, 2022

See also #55116 for forcing a load-based split manually.

@erikgrinaker
Copy link
Contributor

Related to #69364, possibly dupe?

@kvoli kvoli mentioned this issue Jan 20, 2023
Closed
@kvoli kvoli mentioned this issue Feb 1, 2023
Merged
@craig craig bot closed this as completed in 173e3d9 Feb 10, 2023
@cockroach-teamcity cockroach-teamcity mentioned this issue Feb 11, 2023
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
A-kv Anything in KV that doesn't belong in a more specific category. A-kv-distribution Relating to rebalancing and leasing. C-enhancement Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception) T-kv KV Team
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

kvserver: introduce cpu load based splits kvoli/cockroach
3 participants
@erikgrinaker @tbg @kvoli