Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Reclaim space for removed rows in in-memory column sources #5753

Open
rcaudy opened this issue Jul 10, 2024 · 0 comments
Open

Reclaim space for removed rows in in-memory column sources #5753

rcaudy opened this issue Jul 10, 2024 · 0 comments
Assignees
Labels
core Core development tasks feature request New feature or request query engine

Comments

@rcaudy
Copy link
Member

rcaudy commented Jul 10, 2024

We should be more space efficient in cases like rolling-window row key spaces.

This can be done with an update committer that iterates rows in reclaimed (= current - previous) and current block-wise:

  1. For the next row in reclaimed, find the impactedBlock. Advance to next block.
  2. For impactedBlock, calculate the block start and end. Determine if there are any rows between start and end present in current, advance to after end.

This should let us determine blocks to remove, possibly more complicated in the sparse (multi-level) case.

@rcaudy rcaudy added feature request New feature or request query engine core Core development tasks labels Jul 10, 2024
@rcaudy rcaudy added this to the 4. Unscheduled milestone Jul 10, 2024
@rcaudy rcaudy self-assigned this Jul 10, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
core Core development tasks feature request New feature or request query engine
Projects
None yet
Development

No branches or pull requests

1 participant