Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add lzma compressed bmsh file and benchmark #3

Merged
merged 1 commit into from
Mar 30, 2022

Conversation

fangq
Copy link
Contributor

@fangq fangq commented Mar 29, 2022

I want to add another data point - lzma compressed bmsh file - it has the smallest size (2.2 MB), but the decoding is as slow as obj.obj using pure-JS based lzma library (using a C-based lzma decoder in MATLAB, decoding lzma is only about 3x slower than zlib/zip, as opposed to 9x in JS)

on Intel i7-7700K

gz.gii	Size	4384750	Time	1905
gz.mz3	Size	3259141	Time	510
raw.mz3	Size	5898280	Time	21
obj.obj	Size	13307997	Time	5491
stl.stl	Size	16384084	Time	163
zlib.jmsh	Size	4405604	Time	593
zlib.bmsh	Size	3259049	Time	464
raw.min.json	Size	12325881	Time	1352
raw.bmsh	Size	5898902	Time	24
lzma.bmsh	Size	2295259	Time	5290

on AMD Ryzen 9 3950X

gz.gii	Size	4384750	Time	2177
gz.mz3	Size	3259141	Time	517
raw.mz3	Size	5898280	Time	20
obj.obj	Size	13307997	Time	5036
stl.stl	Size	16384084	Time	762
zlib.jmsh	Size	4405604	Time	642
zlib.bmsh	Size	3259049	Time	452
raw.min.json	Size	12325881	Time	1041
raw.bmsh	Size	5898902	Time	15
lzma.bmsh	Size	2295259	Time	5188

@neurolabusc
Copy link
Owner

@fangq I would advocate for zstd vs lzma - it has terrific decompression speeds. I have not used it with JavaScript, but several packages exist.

For files that will be only compressed once and decompressed/transferred/downloaded many times, one can choose a slow zstd compression level that creates very compact files. Likewise, one can use zopfli for making compressed mz3 files that will be smaller, but one hits diminishing returns with the older gz format:

$ pigz -11 -n -k raw.mz3 
$ mv raw.mz3.gz zopfli.mz3
$ ls -l *.mz3                     
-rw-r--r--@ 3259141 Mar 27 08:49 gz.mz3
-rw-r--r--  5898280 Mar 27 09:17 raw.mz3
-rw-r--r--  3151357 Mar 27 09:17 zopfli.mz3

@fangq
Copy link
Contributor Author

fangq commented Mar 30, 2022

@neurolabusc, just to get some idea on the smallest size zstd can offer, I tried zstd -22 raw.bmsh -o raw.bmsh.zst. and the output file is 3055172 byte, which is 33% bigger than lzma's output. I am sure zstd offers better-than-gzip decompression speed, although lzma still hold a place as the most-compact yet widely supported option.

@neurolabusc
Copy link
Owner

I am surprised that lzma does so much better than zstd. Previously, I have observed the zstd can benefits from swizzling data like the 32-bit indices. But empirically, lzma does do a great job with the compression.

@neurolabusc neurolabusc merged commit dc38e56 into neurolabusc:main Mar 30, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants