Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

gc-compaction: filter keys not in shard / not in keyspace #8920

Closed
Tracked by #8002
skyzh opened this issue Sep 4, 2024 · 1 comment · Fixed by #9004 or #9765
Closed
Tracked by #8002

gc-compaction: filter keys not in shard / not in keyspace #8920

skyzh opened this issue Sep 4, 2024 · 1 comment · Fixed by #9004 or #9765
Assignees
@skyzh
Labels
c/storage/pageserver Component: storage: pageserver

Comments

@skyzh
Copy link
Member

skyzh commented Sep 4, 2024 •
edited
Loading

So that we can reclaim spaces taken by #8623 and collect more garbages.
@skyzh skyzh self-assigned this Sep 4, 2024
@skyzh skyzh added the c/storage/pageserver Component: storage: pageserver label Sep 4, 2024
@skyzh skyzh mentioned this issue Sep 16, 2024
Merged
5 tasks
skyzh added a commit that referenced this issue Sep 23, 2024
@skyzh
feat(pageserver): filter keys with gc-compaction (#9004)
2969952 
Part of #8002

Close #8920

Legacy compaction (as well as gc-compaction) rely on the GC process to
remove unused layer files, but this relies on many factors (i.e., key
partition) to ensure data in a dropped table can be eventually removed.

In gc-compaction, we consider the keyspace information when doing the
compaction process. If a key is not in the keyspace, we will skip that
key and not include it in the final output.

However, this is not easy to implement because gc-compaction considers
branch points (i.e., retain_lsns) and the retained keyspaces could
change across different LSNs. Therefore, for now, we only remove aux v1
keys in the compaction process.

## Summary of changes

* Add `FilterIterator` to filter out keys.
* Integrate `FilterIterator` with gc-compaction.
* Add `collect_gc_compaction_keyspace` for a spec of keyspaces that can
be retained during the gc-compaction process.

---------

Signed-off-by: Alex Chi Z <chi@neon.tech>
@skyzh skyzh mentioned this issue Sep 23, 2024
Closed
33 tasks
@skyzh skyzh reopened this Sep 23, 2024
@skyzh
Copy link
Member Author

skyzh commented Sep 23, 2024

we still need to figure out a way to deal with keyspace changes in gc-compaction

@skyzh skyzh mentioned this issue Sep 23, 2024
Open
28 tasks
@skyzh skyzh mentioned this issue Nov 14, 2024
Merged
@skyzh skyzh closed this as completed in e5c89f3 Nov 18, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@skyzh skyzh

Labels
c/storage/pageserver Component: storage: pageserver
Projects
None yet
Milestone
No milestone
1 participant
@skyzh