Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DietPi-Imager | Move from 7z to xz for image compression #6591

Merged
merged 3 commits into from
Sep 14, 2023
Merged

DietPi-Imager | Move from 7z to xz for image compression #6591

merged 3 commits into from
Sep 14, 2023

Conversation

MichaIng
Copy link
Owner

No description provided.

@MichaIng MichaIng added the META Everything that is not code related, e.g. GitHub, Wiki, website, community label Aug 31, 2023
@MichaIng MichaIng added this to the v8.22 milestone Aug 31, 2023
@dirkhh
Copy link
Contributor

dirkhh commented Sep 1, 2023

Why? It's significantly slower. As in a LOT slower. For fairly marginal gain.
May I lobby you to consider moving to LZ4 instead? https://github.com/lz4/lz4

@Joulinar
Copy link
Collaborator

Joulinar commented Sep 1, 2023

using xz you could flash an image directly using the archive to SD card, while it was needed to extract on 7z first.

@dirkhh
Copy link
Contributor

dirkhh commented Sep 1, 2023

using xz you could flash an image directly using the archive to SD card, while it was needed to extract on 7z first.

So this is a comment about some SD card writing tools like Pi Imager being able to convert .xz images "on the fly" when writing an SD card? Well, if that's the goal, then please consider .zstd. Pi Imager supports that just as well, and it is once again dramatically faster than .xz

Put differently - the only time .xz should be the preferred choice is when size matters at all cost; whenever you a looking for something that provides a reasonable trade-off between time and compression, there are much better choices.
Just the impact on duration of GitHub Action runs (and, turnaround time when doing test builds) is huge. In a couple of quick tests here, compression took almost as long as the rest of the build using .xz.

@MichaIng
Copy link
Owner Author

MichaIng commented Sep 1, 2023

7z and xz both use LZMA2 (by default) and hence have similar (de)compression stats. Main reason indeed is the simple single file (de)compression (it automatically adds/removes the xz extension resp. by default removes the archive/original, like gzip, bzip2 and also zstd tools do) and pipe support.

Decompression speed is acceptable and usually not the bottleneck when flashing to a drive directly. I like to keep using LZMA2 as IMO download bandwidth and disk space usage is more a concern then (de)compression speed, and it still achieves a better compression ratio than zstd on highest level (which also reduces its performance benefit).

Basically we align it with Armbian, APT repos, Linux kernel sources etc, which all use xz nowadays, while e.g. zstd is not as widely used/known. A practical example is that 7zip (which I use on Windows as standard archiver) does not support zstd yet, so it is the only compression algorithm I need an extra tool for, annoyingly...

@dirkhh
Copy link
Contributor

dirkhh commented Sep 1, 2023

Thanks for the detailed response.
I understand your reasoning.

- DietPi-Imager | Move from 7z to xz for image compression and do not embed readme and hashes
- DietPi-Build | Move from 7z to xz for image compression and do not embed readme and hashes
- CHANGELOG | Images: All our images are now compressed via xz instead of 7z. These are a little easier to handle, especially on Linux hosts, and many flashing utilities allow to flash zx-compressed images directly to disk, without the need to manually decompress them first. As xz compresses files and no directories, the dedicated README.md and hash text files are not included anymore. The hashes for integrity checks within an archive have no real purpose, as the compression algorithms imply hashes internally (CRC64 in case of xz), which are checked and integrity of the content assed as part of the decompression.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
META Everything that is not code related, e.g. GitHub, Wiki, website, community
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants