Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implemented a memory-compressed storage #318

Open
HDembinski opened this issue Mar 24, 2021 · 1 comment
Open

Implemented a memory-compressed storage #318

HDembinski opened this issue Mar 24, 2021 · 1 comment

Comments

@HDembinski
Copy link
Collaborator

Sophisticated analyses use high-dimensional histograms, both disc and memory usage can become an issue. Disc usage is easy to fix by writing a compressed stream with Boost.Serialization and Boost.IOStreams. But this does not fix the memory issue.

It would be interesting to add an experimental memory-compressed storage and see how that performs. In-memory compression can be achieved in several ways.

  • Zero-suppression: Some high-dimensional histograms contain a lot of zeros. A simple compression would just replace all the zeros with a code that indicates how many zeros are in the omitted gap. This is comparably easy to implement. Frequent re-allocations of memory in the beginning will be an issue, but that could be optimized over time.
  • We could use an in-memory compressor like Blosc: https://www.blosc.org/pages/blosc-in-depth
@HDembinski
Copy link
Collaborator Author

BLOSC is BSD licensed, so compatible with BSL.

https://www.blosc.org/pages/blosc-in-depth/
https://github.com/Blosc/c-blosc2

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant