Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Partition balancer: integrate partition reclaimable size #13480

Merged
merged 4 commits into from
Sep 27, 2023
Merged

Partition balancer: integrate partition reclaimable size #13480

merged 4 commits into from
Sep 27, 2023

Conversation

ztlpn
Copy link
Contributor

@ztlpn ztlpn commented Sep 16, 2023

Backports Required

  • none - not a bug fix
  • none - this is a backport
  • none - issue does not exist in previous branches
  • none - papercut/not impactful enough to backport
  • v23.2.x
  • v23.1.x
  • v22.3.x

Release Notes

Improvements

  • Partition balancer now takes partition size reclaimable by space management into account.

@ztlpn
Copy link
Contributor Author

ztlpn commented Sep 18, 2023

/ci-repeat 2
skip-units
skip-redpanda-build
dt-repeat=20
tests/rptest/tests/partition_balancer_test.py

@ztlpn ztlpn force-pushed the partition-balancer-reclaimable-size branch 3 times, most recently from 5296199 to a7eb66d Compare September 23, 2023 23:11
@ztlpn
Copy link
Contributor Author

ztlpn commented Sep 23, 2023

/ci-repeat 2
skip-units
skip-redpanda-build
dt-repeat=25
tests/rptest/tests/partition_balancer_test.py

@ztlpn
Copy link
Contributor Author

ztlpn commented Sep 24, 2023

/ci-repeat 2
skip-units
dt-repeat=25
tests/rptest/tests/partition_balancer_test.py

@ztlpn
c/partition_balancer: deprecate movement_batch_size_bytes
e9721e2 
The reason is twofold:
1. We have another option for controlling the concurrency of the
   movement batch - partition_autobalancing_concurrent_moves which is
   easier to understand
2. Part of initial motivation is gone - in the past we waited for the
   whole batch to finish before scheduling the next movements so it was
   important for the batch to finish in a reasonable amount of time, but
   now we can schedule new movement while the old ones are still in
   progress.
@ztlpn
c/partition_balancer: take reclaimable partition size into account
3480ba8 
With the introduction of space management data eviction becomes lazy:
partitions can grow beyond their eviction limit if there is sufficient
space to do it and only if disk space pressure becomes high, this
"extra" data is evicted.

For partition balancing this means two things: 1) partitions sizes can
vary widely between replicas and 2) when predicting how much space the
partition will take on a node if it is moved there, we must take into
account the possibility of reclaim (otherwise we won't allow moves that
are perfectly possible if reclaim is applied).

So this commit we 1) store current sizes in a per-replica map instead of
a single value and 2) use non-reclaimable size when predicting the
partition size on the target replica.
@ztlpn
tests: add single-partition case to space-mgmt decom test
794d280
@ztlpn ztlpn force-pushed the partition-balancer-reclaimable-size branch from a7eb66d to 01493a3 Compare September 26, 2023 13:34
@ztlpn ztlpn force-pushed the partition-balancer-reclaimable-size branch from 01493a3 to dde8911 Compare September 26, 2023 13:40
@ztlpn ztlpn marked this pull request as ready for review September 26, 2023 14:32
@ztlpn
Copy link
Contributor Author

ztlpn commented Sep 26, 2023

I've seen some instances of #8226 but after I added trace logging, they disappeared :/ (several 100s green runs). I guess this patch doesn't make the situation worse and we can merge it with trace logging to understand possible future instances of 8226 better.

@ztlpn
Copy link
Contributor Author

ztlpn commented Sep 26, 2023

upgrade tests all failed because Connection to api.github.com timed out

@ztlpn
Copy link
Contributor Author

ztlpn commented Sep 27, 2023

/ci-repeat 2
skip-units
skip-redpanda-build
release
dt-repeat=25
tests/rptest/tests/partition_balancer_test.py

mmaslankaprv
mmaslankaprv approved these changes Sep 27, 2023
View reviewed changes
Copy link
Member

@mmaslankaprv mmaslankaprv left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The code lgtm, i am worried about the test failures

@ztlpn
Copy link
Contributor Author

ztlpn commented Sep 27, 2023

all test failures are

task: Failed to run task "rp:run-ducktape-tests": task: Failed to run task "rp:init-ducktape-containers": task: Failed to run task "rp:build-test-docker-image": exit status 1

retrying...

@vbotbuildovich
Copy link
Collaborator

ducktape was retried in job https://buildkite.com/redpanda/redpanda/builds/37703#018ad6a1-e48c-452b-ba8f-a80dab0a6c49

@ztlpn
Copy link
Contributor Author

ztlpn commented Sep 27, 2023

The retried failure is #9788
I guess we should turn off leader balancer in this test, because it leads to creation of new segments (and thus disk usage growth) at inopportune times.

@ztlpn ztlpn merged commit b5c983f into redpanda-data:dev Sep 27, 2023
9 checks passed
@vbotbuildovich
Copy link
Collaborator

/backport v23.2.x

@vbotbuildovich
Copy link
Collaborator

Failed to create a backport PR to v23.2.x branch. I tried:

git remote add upstream https://github.com/redpanda-data/redpanda.git
git fetch --all
git checkout -b backport-pr-13480-v23.2.x-454 remotes/upstream/v23.2.x
git cherry-pick -x e9721e29e0bedd1203802256591d74dc0f5bcf3b 3480ba84c892514ac4eed0e44b56d60e28670537 794d280f13bf534355b1f27504fac2b9cb140066 dde891176a737c3febccc553902ec3117d526b64

Workflow run logs.

@vbotbuildovich vbotbuildovich mentioned this pull request Sep 27, 2023
Closed
@ztlpn ztlpn mentioned this pull request Sep 28, 2023
Merged
ztlpn added a commit that referenced this pull request Oct 3, 2023
@ztlpn
Merge pull request #13784 from ztlpn/v23.2.x-bp
080ed02 
[v23.2.x] Backport of #12544 #12596 #12644 #12586 #12737 #12368 #13480
@ztlpn ztlpn deleted the partition-balancer-reclaimable-size branch November 27, 2023 14:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@mmaslankaprv mmaslankaprv mmaslankaprv approved these changes

@bharathv bharathv Awaiting requested review from bharathv

Assignees
No one assigned
Labels
area/redpanda
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@ztlpn @vbotbuildovich @mmaslankaprv