Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Turn off store archiving to hotfix memory duplication issue #1397

Closed
emilk opened this issue Feb 24, 2023 · 3 comments · Fixed by #1535
Closed

Turn off store archiving to hotfix memory duplication issue #1397

emilk opened this issue Feb 24, 2023 · 3 comments · Fixed by #1535
Labels
🏹 arrow concerning arrow 🚀 performance Optimization, memory use, etc ⛃ re_datastore affects the datastore itself

Comments

@emilk
Copy link
Member

emilk commented Feb 24, 2023

Maybe we need both that and:

@emilk emilk added 🏹 arrow concerning arrow ⛃ re_datastore affects the datastore itself 🚀 performance Optimization, memory use, etc labels Feb 24, 2023
@emilk emilk self-assigned this Feb 27, 2023
@emilk
Copy link
Member Author

emilk commented Feb 27, 2023

The way to accomplish this is to set

DataStoreConfig {
    component_bucket_size_bytes: u64::MAX,
    component_bucket_nb_rows: u64::MAX,
    …
}

I've tested it, and it does indeed solve the memory duplication.

However, according to the docs, it also stops the GC from doing anything, which means memory pruning won't work.

However, considering our memory pruning is already crashing, maybe it doesn't matter for the time beeing:

@emilk emilk removed their assignment Feb 28, 2023
@emilk
Copy link
Member Author

emilk commented Mar 3, 2023

No - we want memory pruning

@emilk emilk closed this as not planned Won't fix, can't repro, duplicate, stale Mar 3, 2023
@teh-cmc teh-cmc changed the title Turn off store archiving to maybe solve memory duplication issue Turn off store archiving to hotfix memory duplication issue Mar 9, 2023
@teh-cmc
Copy link
Member

teh-cmc commented Mar 9, 2023

This turned out to be the right strategy in the end.

It need not break garbage collection: we can disable archiving without necessarily breaking the bucketing system.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
🏹 arrow concerning arrow 🚀 performance Optimization, memory use, etc ⛃ re_datastore affects the datastore itself
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants