Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update transmute to use SpooledTemporaryFile instead of streaming directly to zip, for zstd reasons #57

Closed
2 tasks done
dholth opened this issue Jan 31, 2023 · 0 comments · Fixed by #60
Closed
2 tasks done
Assignees

Comments

@dholth
Copy link
Contributor

dholth commented Jan 31, 2023

Checklist

  • I added a descriptive title
  • I searched open requests and couldn't find a duplicate

What is the idea?

When converting .tar.bz2 to .conda, conda-package-streaming uses... streaming. This is a neat trick, but unfortunately not so great for zstd decompression memory, which benefits when we can include the whole size at the beginning of the zstd stream. (If the whole decompressed file is smaller than the memory buffer it would otherwise allocate. Might be 128MB or so for -22 but much less for our standard level -19.)

Instead, we can write a temporary tar file, get its size, and then compress that back into the ZIP-format .conda archive.

We could do it without temporary files, by decompressing the .tar.bz2 twice, but bz2-decompression is a bit slow.

Probably the zstd compressor will dominate the runtime, and we won't notice any speed difference.

Old conda-package-handling converted by extracting the whole thing as individual files and then repacking.

Why is this needed?

No response

What should happen?

No response

Additional Context

No response

@dholth dholth self-assigned this Mar 30, 2023
@dholth dholth added the sprint label Mar 30, 2023
@dholth dholth closed this as completed in #60 May 8, 2023
@github-actions github-actions bot added the locked label May 8, 2024
@github-actions github-actions bot locked as resolved and limited conversation to collaborators May 8, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant