Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Added vectorized_raft_learners_gap_bytes metric #18691

Merged
merged 3 commits into from
Jun 6, 2024
Merged

Added vectorized_raft_learners_gap_bytes metric #18691

merged 3 commits into from
Jun 6, 2024

Conversation

mmaslankaprv
Copy link
Member

@mmaslankaprv mmaslankaprv commented May 29, 2024 •
edited
Loading

Added vectorized_raft_learners_gap_bytes metric providing additional insight into partition movement process. The metric informs about the total number of bytes that are left to be sent to a learner follower.

Backports Required

  • none - not a bug fix
  • none - this is a backport
  • none - issue does not exist in previous branches
  • none - papercut/not impactful enough to backport
  • v24.1.x
  • v23.3.x
  • v23.2.x

Release Notes

Improvements

  • new metric providing more insight into recovery process

@vbotbuildovich
Copy link
Collaborator

ducktape was retried in https://buildkite.com/redpanda/redpanda/builds/49628#018fc3f6-8adc-4c29-a294-a78f3129c407

bashtanov
bashtanov reviewed Jun 3, 2024
View reviewed changes
src/v/raft/group_manager.cc Outdated Show resolved Hide resolved
src/v/raft/group_manager.h Outdated Show resolved Hide resolved
src/v/raft/group_manager.cc Outdated Show resolved Hide resolved
src/v/raft/consensus.h Outdated Show resolved Hide resolved
tests/rptest/tests/nodes_decommissioning_test.py Outdated Show resolved Hide resolved
@mmaslankaprv mmaslankaprv force-pushed the recovery-metrics branch 2 times, most recently from c9cf0cf to 4dbad43 Compare June 3, 2024 16:21
bashtanov
bashtanov previously approved these changes Jun 3, 2024
View reviewed changes
Copy link
Contributor

@bashtanov bashtanov left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

in the tests, maybe could we also check that, either without decommissioning any nodes, or after a node has been re-added and has caught up, the gap is about zero?

LGTM anyway

tests/rptest/tests/nodes_decommissioning_test.py Show resolved Hide resolved
@mmaslankaprv mmaslankaprv requested a review from bashtanov June 4, 2024 05:52
bashtanov
bashtanov previously approved these changes Jun 4, 2024
View reviewed changes
Copy link
Contributor

@bashtanov bashtanov left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, only minor python style nitpicking

tests/rptest/tests/nodes_decommissioning_test.py Outdated Show resolved Hide resolved
tests/rptest/tests/nodes_decommissioning_test.py Outdated Show resolved Hide resolved
@mmaslankaprv mmaslankaprv requested a review from bashtanov June 4, 2024 10:15
bashtanov
bashtanov previously approved these changes Jun 4, 2024
View reviewed changes
tests/rptest/tests/nodes_decommissioning_test.py Outdated Show resolved Hide resolved
ztlpn
ztlpn reviewed Jun 4, 2024
View reviewed changes
src/v/raft/consensus.h Outdated Show resolved Hide resolved
src/v/raft/group_manager.cc Outdated Show resolved Hide resolved
src/v/raft/group_manager.cc Outdated Show resolved Hide resolved
@mmaslankaprv
raft: expose api to query bytes to deliver to followers
898f08b 
Exposed an API returning the total number of bytes to send to learner
followers.

Signed-off-by: Michał Maślanka <michal@redpanda.com>
@mmaslankaprv
raft: exposed metric representing bytes to sent to learners
00954f6 
Exposed a metric informing about the total number of bytes that are
there to deliver to learner follower.

Signed-off-by: Michał Maślanka <michal@redpanda.com>
@mmaslankaprv mmaslankaprv force-pushed the recovery-metrics branch 2 times, most recently from a7f6aa1 to 7acfd32 Compare June 5, 2024 17:24
@mmaslankaprv
tests: added a test validating correctness of learner gap metric
02286dc 
Signed-off-by: Michał Maślanka <michal@redpanda.com>
@mmaslankaprv
Copy link
Member Author

/ci-repeat 1

@mmaslankaprv mmaslankaprv merged commit bbd3000 into redpanda-data:dev Jun 6, 2024
19 checks passed
@vbotbuildovich
Copy link
Collaborator

/backport v24.1.x

@vbotbuildovich
Copy link
Collaborator

/backport v23.3.x

@vbotbuildovich
Copy link
Collaborator

Failed to create a backport PR to v23.3.x branch. I tried:

git remote add upstream https://github.com/redpanda-data/redpanda.git
git fetch --all
git checkout -b backport-pr-18691-v23.3.x-462 remotes/upstream/v23.3.x
git cherry-pick -x 898f08bc12952521821dd3ca6cb64a57a33bb803 00954f6589f2d87c144d7cb08316b6aeccef2bf2 02286dc13677eea7639c00c11ec04afe0fb95ad7

Workflow run logs.

This was referenced Jun 6, 2024
Open
Merged
@mmaslankaprv mmaslankaprv mentioned this pull request Jun 18, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ztlpn ztlpn ztlpn approved these changes

@bashtanov bashtanov bashtanov approved these changes

@bharathv bharathv Awaiting requested review from bharathv

Assignees
No one assigned
Labels
area/redpanda
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@mmaslankaprv @vbotbuildovich @bharathv @bashtanov @ztlpn