You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I searched open requests and couldn't find a duplicate
What is the idea?
When converting .tar.bz2 to .conda, conda-package-streaming uses... streaming. This is a neat trick, but unfortunately not so great for zstd decompression memory, which benefits when we can include the whole size at the beginning of the zstd stream. (If the whole decompressed file is smaller than the memory buffer it would otherwise allocate. Might be 128MB or so for -22 but much less for our standard level -19.)
Instead, we can write a temporary tar file, get its size, and then compress that back into the ZIP-format .conda archive.
We could do it without temporary files, by decompressing the .tar.bz2 twice, but bz2-decompression is a bit slow.
Probably the zstd compressor will dominate the runtime, and we won't notice any speed difference.
Old conda-package-handling converted by extracting the whole thing as individual files and then repacking.
Why is this needed?
No response
What should happen?
No response
Additional Context
No response
The text was updated successfully, but these errors were encountered:
Checklist
What is the idea?
When converting
.tar.bz2
to.conda
, conda-package-streaming uses... streaming. This is a neat trick, but unfortunately not so great for zstd decompression memory, which benefits when we can include the whole size at the beginning of the zstd stream. (If the whole decompressed file is smaller than the memory buffer it would otherwise allocate. Might be 128MB or so for -22 but much less for our standard level -19.)Instead, we can write a temporary tar file, get its size, and then compress that back into the ZIP-format
.conda
archive.We could do it without temporary files, by decompressing the .tar.bz2 twice, but bz2-decompression is a bit slow.
Probably the zstd compressor will dominate the runtime, and we won't notice any speed difference.
Old conda-package-handling converted by extracting the whole thing as individual files and then repacking.
Why is this needed?
No response
What should happen?
No response
Additional Context
No response
The text was updated successfully, but these errors were encountered: