Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

pageserver: aux keys go to all shards #9641

Closed
VladLazar opened this issue Nov 5, 2024 · 1 comment · Fixed by #9786
Closed

pageserver: aux keys go to all shards #9641

VladLazar opened this issue Nov 5, 2024 · 1 comment · Fixed by #9786
Assignees
@jcsp
Labels
c/storage/pageserver Component: storage: pageserver c/storage Component: storage

Comments

@VladLazar
Copy link
Contributor

While reading some unrelated code, I noticed that we write aux keys to all shards. Apparently, they should only be on shard 0.
I checked out the code prior to my sharded ingest work and believe that has always been the case.

We should:

  • make sure that's the current state of affairs (I'm pretty sure - but let's check)
  • only write aux keys to shard 0
@VladLazar VladLazar added c/storage Component: storage c/storage/pageserver Component: storage: pageserver labels Nov 5, 2024
@jcsp jcsp assigned jcsp and unassigned skyzh Nov 20, 2024
@jcsp jcsp mentioned this issue Nov 20, 2024
Merged
@jcsp
Copy link
Collaborator

jcsp commented Nov 20, 2024

I'm picking this up in #9786 where we're also changing SLRUs to only go on shard 0

github-merge-queue bot pushed a commit that referenced this issue Dec 3, 2024
@jcsp
pageserver: only store SLRUs & aux files on shard zero (#9786)
dcb6295 
## Problem

Since #9423 the non-zero shards
no longer need SLRU content in order to do GC. This data is now
redundant on shards >0.

One release cycle after merging that PR, we may merge this one, which
also stops writing those pages to shards > 0, reaping the efficiency
benefit.

Closes: #7512
Closes: #9641

## Summary of changes

- Avoid storing SLRUs on non-zero shards
- Bonus: avoid storing aux files on non-zero shards
@jcsp jcsp closed this as completed in #9786 Dec 3, 2024
awarus pushed a commit that referenced this issue Dec 5, 2024
@jcsp @awarus
pageserver: only store SLRUs & aux files on shard zero (#9786)
41bb9c5 
## Problem

Since #9423 the non-zero shards
no longer need SLRU content in order to do GC. This data is now
redundant on shards >0.

One release cycle after merging that PR, we may merge this one, which
also stops writing those pages to shards > 0, reaping the efficiency
benefit.

Closes: #7512
Closes: #9641

## Summary of changes

- Avoid storing SLRUs on non-zero shards
- Bonus: avoid storing aux files on non-zero shards
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@jcsp jcsp

Labels
c/storage/pageserver Component: storage: pageserver c/storage Component: storage
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

pageserver: only store SLRUs & aux files on shard zero neondatabase/neon
3 participants
@jcsp @skyzh @VladLazar