Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add some parallel compression type #19

Closed
luvarqpp opened this issue Sep 13, 2019 · 7 comments
Closed

add some parallel compression type #19

luvarqpp opened this issue Sep 13, 2019 · 7 comments

Comments

@luvarqpp
Copy link

I would like to use more cpu cores than single one.

Add possibility to use pigz, zstd or something similar.

PS: I like that your script is "auditable" in few minutes by simple reading. Consider, if feature like this one, does not make it "too long".

@loomchild
Copy link
Owner

I wasn't even aware these exist:)

I agree that I'd like to keep it simple. Would you like to propose a PR? If not, I will take a look at it a bit later.

@debuglevel
Copy link

That would be great! :-) To set a parallel compression method, additional tools (pigz, pbzip2, pxz) have to be installed and called by the -I pxz argument:

gz: tar -I pigz -cf tarball.tgz files
bz2: tar -I pbzip2 -cf tarball.tbz files
xz: tar -I pxz -cf tarball.txz files

@loomchild
Copy link
Owner

Thanks for the info!

I can try to integrate those if I find some time, or alternatively feel free to propose a PR.

@loomchild
Copy link
Owner

Zstd was just added by @ksurl #36. I hope it solves the ticket - if not, please reopen.

@ksurl
Copy link
Contributor

ksurl commented Mar 14, 2021

Zstd does support parallel compression with multithreading in the latest version but that version is not available in alpine yet. Pigz would be better if parallel is a must but zstd is pretty fast. Also since tar doesnt support passing flags to the compression program you'd have to use a pipe and change how the script calls it

@luvarqpp
Copy link
Author

Also since tar doesnt support passing flags to the compression program you'd have to use a pipe and change how the script calls it

IMHO it is possible to pass parameters. I use this approach:

tar --use-compress-program="pigz --best" -cf filesWithPrefixAsdf.tar.gz Asdf*

Note, that passed program have to support -d parameter (for decompression). If this is not the case (7za binary for example), you can have a look at simple wrapper approach on stack owerflow

@loomchild
Copy link
Owner

loomchild commented Mar 20, 2021

BTW, I have just added an option -x to pass any arguments to Tar (see #28). Perhaps this would be useful in such complex scenarios (unfortunately I can't verify as I am not using parallel compression)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

4 participants