Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Rebatch aggressively while dumping the store to a stream of LogMsg #1894

Closed
Tracked by #1619
, #1898
...
teh-cmc opened this issue Apr 18, 2023 · 1 comment · Fixed by #6570
Closed
Tracked by #1619
, #1898
...

Rebatch aggressively while dumping the store to a stream of LogMsg #1894

teh-cmc opened this issue Apr 18, 2023 · 1 comment · Fixed by #6570
Assignees
@teh-cmc
Labels
🚀 performance Optimization, memory use, etc ⛃ re_datastore affects the datastore itself

Comments

@teh-cmc
Copy link
Member

teh-cmc commented Apr 18, 2023 •
edited by emilk
Loading

The save-feature walks the store and converts each bucket back into a DataTable that is then serialized and dumped as a LogMsg.

This is a great opportunity to batch things even further.
@teh-cmc teh-cmc mentioned this issue Apr 18, 2023
Closed
18 tasks
@teh-cmc teh-cmc changed the title Rebatch aggressively while dumping to disk Rebatch aggressively while dumping the store to a stream of LogMsg Apr 18, 2023
@teh-cmc teh-cmc added ⛃ re_datastore affects the datastore itself 🚀 performance Optimization, memory use, etc labels Apr 18, 2023
This was referenced Apr 18, 2023
Closed
Closed
@teh-cmc
Copy link
Member Author

teh-cmc commented Nov 7, 2023 •
edited
Loading

More importantly, we need the save process to be RowId-driven so that we can forbid re-use of RowIds once and for all, which will remove the need to count unique RowIds for downstream subscribers of store events, which is a blocker for the StoreView/StoreEvent stuff.

@teh-cmc teh-cmc mentioned this issue Nov 10, 2023
Closed
@teh-cmc teh-cmc mentioned this issue Dec 11, 2023
Closed
@teh-cmc teh-cmc mentioned this issue Dec 22, 2023
Closed
@teh-cmc teh-cmc self-assigned this May 16, 2024
@teh-cmc teh-cmc mentioned this issue Jun 18, 2024
Merged
5 tasks
@teh-cmc teh-cmc closed this as completed in ed20cba Jul 4, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@teh-cmc teh-cmc

Labels
🚀 performance Optimization, memory use, etc ⛃ re_datastore affects the datastore itself
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Implement ChunkStore and integrate it everywhere rerun-io/rerun
1 participant
@teh-cmc