Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

kvserver: include rebalance objective in store rebalancer logging #102812

Closed
kvoli opened this issue May 5, 2023 · 0 comments · Fixed by #102825
Closed

kvserver: include rebalance objective in store rebalancer logging #102812

kvoli opened this issue May 5, 2023 · 0 comments · Fixed by #102825
Assignees
@kvoli
Labels
A-kv-distribution Relating to rebalancing and leasing. C-enhancement Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception) T-kv KV Team

Comments

@kvoli
Copy link
Collaborator

kvoli commented May 5, 2023 •
edited
Loading

Is your feature request related to a problem? Please describe.
We enabled dual rebalancing objectives in #90582, where the store rebalancer will either look at QPS or CPU. The current objective can be found by checking the cluster settings and checking the cockroach.log if the value set by the cluster setting is unsupported:

cockroach/pkg/kv/kvserver/rebalance_objective.go

Line 251 in 9189237

log.Infof(ctx, "version doesn't support cpu objective, reverting to qps balance objective")

Or by inspecting which load dimension's max threshold is equal to the mean e.g. below shows cpu set:

[T1,n5,s9,store-rebalancer] 736  local load ‹(queries-per-second=1261.1 cpu-per-second=161ms)› is below max threshold ‹(queries-per-second=1515.4 cpu-per-second=234ms)› mean=‹(queries-per-second=1515.4 cpu-per-second=184ms)›; no rebalancing needed

Describe the solution you'd like
Log the current objective, as seen by the store rebalancer in the ambient ctx or somewhere otherwise visible.

Jira issue: CRDB-27665
@kvoli kvoli added C-enhancement Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception) A-kv-distribution Relating to rebalancing and leasing. labels May 5, 2023
@kvoli kvoli self-assigned this May 5, 2023
@blathers-crl blathers-crl bot added the T-kv KV Team label May 5, 2023
@kvoli kvoli mentioned this issue May 5, 2023
Merged
craig bot pushed a commit that referenced this issue May 9, 2023
@kvoli @stevendanna
Merge #98720 #101729 #102825
a961c9d 
98720: roachtest: enable kv/gracefuldraining/nodes=3 r=andrewbaptist a=kvoli

This commit re-enables the `kv/gracefuldraining/nodes=3` roachtest. The
test is still likely to fail occasionally however has produced
interesting findings just in testing to re-enable.

Informs: #59094

Release note: None

101729: streamingccl: don't require TLS certificates r=dt a=stevendanna

Users may want to use password auth to simplify their replication setup. While we may recommend TLS certificate auth, I don't see a strong reason to _require_ it.

Epic: none

Release note: None

102825: kvserver,storepool: misc rebalance logging improvements r=andrewbaptist a=kvoli

The store list string returned the mean leases, ranges and
queries-per-second float values without limiting the number of decimal
places. This led to log lines with needlessly long decimals:

`avg-ranges=40.66666666666667... avg-leases=10.166666666666666...`

This PR updates the store list string formatting to 2 decimal
places for float values.

Previously, the easiest method of determining the current rebalance
objective from logs was to view the cluster setting and check for
logging indicating a mixed version cluster - this was cumbersome.

This PR annotates the ctx in the store rebalancer loop with an
additional tag: `obj`. The `obj` tag indicates the current rebalance
objective, either `cpu` or `qps` currently.

resolves: #102812


Release note: None

Co-authored-by: Austen McClernon <austen@cockroachlabs.com>
Co-authored-by: Steven Danna <danna@cockroachlabs.com>
@craig craig bot closed this as completed in 8da1cea May 9, 2023
@blathers-crl blathers-crl bot mentioned this issue May 9, 2023
Merged
blathers-crl bot pushed a commit that referenced this issue May 9, 2023
@kvoli
kvserver: annotate rebalance ctx with objective
cff194d 
Previously, the easiest method of determining the current rebalance
objective from logs was to view the cluster setting and check for
logging indicating a mixed version cluster - this was cumbersome.

This commit annotates the ctx in the store rebalancer loop with an
additional tag: `obj`. The `obj` tag indicates the current rebalance
objective, either `cpu` or `qps` currently.

resolves: #102812

Release note: None
kvoli added a commit that referenced this issue May 30, 2023
@kvoli
kvserver: annotate rebalance ctx with objective
518fd7e 
Previously, the easiest method of determining the current rebalance
objective from logs was to view the cluster setting and check for
logging indicating a mixed version cluster - this was cumbersome.

This commit annotates the ctx in the store rebalancer loop with an
additional tag: `obj`. The `obj` tag indicates the current rebalance
objective, either `cpu` or `qps` currently.

resolves: #102812

Release note: None
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@kvoli kvoli

Labels
A-kv-distribution Relating to rebalancing and leasing. C-enhancement Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception) T-kv KV Team
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

kvserver,storepool: misc rebalance logging improvements kvoli/cockroach
1 participant
@kvoli