Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

storcon: track number of attached shards for each node #8011

Merged
merged 1 commit into from
Jun 11, 2024

Conversation

VladLazar
Copy link
Contributor

Problem

The storage controller does not track the number of shards attached to a given pageserver. This is a requirement for various scheduling operations (e.g. draining and filling will use this to figure out if the cluster is balanced)

Summary of Changes

Track the number of shards attached to each node.

Related #7387

Checklist before requesting a review

  • I have performed a self-review of my code.
  • If it is a core feature, I have added thorough tests.
  • Do we need to implement analytics? if so did you add the relevant metrics to the dashboard?
  • If this PR requires public announcement, mark it with /release-notes label and add several sentences in this section.

Checklist before merging

  • Do not forget to reformat commit message to not include the above checklist

Problem
The storage controller does not track the number of shards attached
to a given pageserver. This is a requirement for various scheduling
operations (e.g. draining and filling will use this to figure out
if the cluster is balanced)

Summary of Changes
Track the number of shards attached to each node.
@VladLazar VladLazar requested a review from a team as a code owner June 11, 2024 12:38
@VladLazar VladLazar requested review from skyzh and arpad-m June 11, 2024 12:38
Copy link

3198 tests run: 3056 passed, 0 failed, 142 skipped (full report)


Code coverage* (full report)

  • functions: 31.5% (6602 of 20948 functions)
  • lines: 48.5% (51128 of 105464 lines)

* collected from Rust tests only


The comment gets automatically updated with the latest test results
3243802 at 2024-06-11T13:25:20.572Z :recycle:

Copy link
Member

@arpad-m arpad-m left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving this, skipping verification that the functions which call update_node_ref_counts actually are only called once (because even though update_node_ref_counts is not idempotent, neither is the functions it replaces).

We don't rely on demote_attached idempotency anywhere, right?

@VladLazar
Copy link
Contributor Author

Approving this, skipping verification that the functions which call update_node_ref_counts actually are only called once (because even though update_node_ref_counts is not idempotent, neither is the functions it replaces).

We don't rely on demote_attached idempotency anywhere, right?

Right, demote_attached is not meant to be idempotent.

@VladLazar VladLazar merged commit 126bcc3 into main Jun 11, 2024
64 checks passed
@VladLazar VladLazar deleted the vlad/storcon-track-attached-shard-counts branch June 11, 2024 15:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants