Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

clean up resources after failed compaction #8362

Open
skyzh opened this issue Jul 11, 2024 · 2 comments
Open

clean up resources after failed compaction #8362

skyzh opened this issue Jul 11, 2024 · 2 comments
Assignees
@skyzh
Labels
c/storage/pageserver Component: storage: pageserver t/bug Issue Type: Bug triaged bugs that were already triaged

Comments

@skyzh
Copy link
Member

skyzh commented Jul 11, 2024

https://neondb.slack.com/archives/C033RQ5SPDH/p1720712394852919?thread_ts=1720702469.555749&cid=C033RQ5SPDH

failed compaction will keep the resources on the disk, and it will cause bloated space problem

related PR: #8359
@skyzh skyzh added t/bug Issue Type: Bug c/storage/pageserver Component: storage: pageserver labels Jul 11, 2024
@skyzh skyzh self-assigned this Jul 11, 2024
@skyzh skyzh mentioned this issue Jul 11, 2024
Merged
5 tasks
@jcsp jcsp added the triaged bugs that were already triaged label Jul 25, 2024
@jcsp
Copy link
Collaborator

jcsp commented Jul 25, 2024

Chi plans to do this as part of GC compaction work

@skyzh
Copy link
Member Author

skyzh commented Sep 18, 2024

implemented as part of #9048 and #8841, will integrate with the image creation path

@skyzh skyzh mentioned this issue Oct 23, 2024
Merged
5 tasks
skyzh added a commit that referenced this issue Oct 24, 2024
@skyzh
refactor(pageserver): refactor split writers using batch layer writer (…
fb0406e 
…#9493)

part of #9114,
#8836,
#8362

The split layer writer code can be used in a more general way: the
caller puts unfinished writers into the batch layer writer and let batch
layer writer to ensure the atomicity of the layer produces.

## Summary of changes

* Add batch layer writer, which atomically finishes the layers.
`BatchLayerWriter::finish` is simply a copy-paste from previous split
layer writers.
* Refactor split writers to use the batch layer writer.
* The current split writer tests cover all code path of batch layer
writer.

---------

Signed-off-by: Alex Chi Z <chi@neon.tech>
@skyzh skyzh mentioned this issue Oct 31, 2024
Open
5 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@skyzh skyzh

Labels
c/storage/pageserver Component: storage: pageserver t/bug Issue Type: Bug triaged bugs that were already triaged
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@jcsp @skyzh