Releases: fcorbelli/zpaqfranz
Windows 32/64 binaries, 64 bit-HW accelerated
New switch -tmp (now enabled by default for backups) that names archives with the .tmp extension during compression before renaming them to .zpaq. This helps handle cases of shutdown or zpaqfranz crash during archiving
One of the issues with the zpaq storage format is its fragility when dealing with the creation of corrupted archives. This typically occurs due to an unexpected system shutdown or a forced termination of the process. In these cases, an incomplete transaction is added, which generally causes a series of problems. In the worst-case scenario, it may become impossible to restore data from that point onward.
This issue can also affect multipart archives, which consist of a series of .zpaq files (e.g., 01, 02, 03...). Both zpaq and zpaqfranz, for reasons of historical compatibility, provide only very limited information when encountering incomplete transactions. Now, zpaqfranz displays a prominent warning to alert the user.
If the interrupted transaction is the last one, zpaqfranz now automatically (unless the -notrim switch is used) attempts to correct the archive. If successful, it continues; otherwise, it halts. In this more severe case, users can try using the consolidate command (for multipart backups) or trim (for single files). The purpose of these commands is to "cut off" corrupted parts, theoretically enabling data extraction and the creation of a new, correct archive.
Due to the seriousness of this issue, starting from version 60.10, the default behavior of the backup command has changed. Temporary files with a .tmp extension are now created and renamed to .zpaq only if everything is correct. This ensures that a potential interruption during the process will not corrupt the backup. On the next run, any existing .tmp files will be "parked," and the procedure will (hopefully) complete successfully.
For more details, refer to the GitHub issues section. If you have any questions or suggestions, feel free to share them.
New flag -notrim that disables the auto-correction mechanism for incomplete transactions when they are in the last transaction
Restoring zpaq's 7.15 behayviour
New switch -destination in the consolidate command to rename zpaq backup files
The consolidate command has two functions.
_With the -to switch, it merges multiple .zpaq files into a single file (labeled 01).
With the -destination switch, it renames a backup, for example, from pippo to pluto. _
In the future, a new switch will allow converting "normal" .zpaq archives directly into backups.
WARNING: When using the consolidate command, always use FULL paths. For example, c:\zpaqfranz\pippo.zpaq
is correct, while pluto.zpaq
is not.
The "i" (info) command, now shows totals
The "i" (info) command, there is a -n switch that shows the last few lines
Fixed a bug with -stdin that disabled the deduplicator
Restored support for Windows XP (for the 32-bit version)
Various minor source code fixes
New switch -nopid that disables the creation of .pid files for the backup command (to prevent multiple executions)
New feature for Windows that displays progress in the calling console during updates from my site
-big during backup shows the very last day (of the backup itself) very big (easier log-checking)
Windows 32/64 binaries, 64 bit-HW accelerated
-DNOJIT is now -nojit switch
Now the definition of compilation with -DNOJIT
should be obsolete. zpaqfranz automatically detects whether the system supports JIT execution, both at the CPU instruction level and the operating system level. Some systems (e.g., NetBSD) prevent code execution from allocated memory areas (PROT_EXEC). Essentially, the CPU may support JIT, but the operating system doesn't quite agree. It’s not yet fully tested, especially on virtual systems, where the JIT might be incorrectly disabled. Managing all virtualization systems with "fake" CPUs is challenging. I’ll probably add a "forcejit" switch in the next release. In summary: there’s no need to use -DNOJIT
during compilation. JIT, as a reminder, accelerates data extraction, while compression speed remains the same.
-tmp
The -tmp
switch causes multipart files to be named ".tmp" instead of ".zpaq," renaming them only when they are completed. This allows for running tests and updates in parallel, and also makes it compatible with systems (like Syncthing) that detect file changes and automatically trigger remote updates.
Better password handling
Improved password input handling (with the -key
switch), with support (at least theoretically!) for the delete key and cursor movement. When creating an archive, it now prompts for the password twice (which must obviously be the same). Example: zpaqfranz a z:\1.zpaq *.cpp -key
.
1980
During creation, the files are stamped with the date 1/1/1980. This makes it easy to identify if the file is incomplete (for example, because the compression was interrupted for any reason, such as a full disk, process kill, and so on). The date is set when the final header, jidac, is written
More placehoderls
On Windows, it's possible to use "placeholders" in file names, such as $pcname
, $computername
, and $username
, which will be automatically replaced. For example, zpaqfranz a z:\pippo_$username c:\nz
.
New ZETA hasher / -backupzeta
Now there are (actually, two) new ZETA hashes. They are used to calculate pseudo XXHASH64 hashes in conjunction with the -backupzeta
switch of the backup command. OK, it's a rather complicated thing, there's an explanation in the issues. Let me summarize.
When using the backup command, a text file is created (or updated) where MD5 hashes (default behavior) or XXH3 hashes (with the -backupxxh3
switch) are written for integrity checks (they actually also contain QUICK hashes). This requires a re-reading of the written data. That is, when the new chunk is created, for example, okane_00003456.zpaq
, zpaqfranz
will re-read the okane_00003456.zpaq
file (after creating it) to update the index TXT file. Normally, this isn’t a big issue (multipart files are usually small). However, they can become very large (e.g., virtual disks). In this case, the -backupzeta
switch will generate a pseudo XXHASH64 hash while generating the archive, WITHOUT needing to re-read the okane_00003456.zpaq
file after writing it (it actually also calculates the CRC-32).
As of now, it doesn’t work with encrypted multipart archives (but it will in the future). If anything is unclear, feel free to ask.
-nomore
The -nomore
switch disables the internal more
command, making external text processing faster. For example, zpaqfranz h h -nomore | less
. On 64-bit Windows systems, it also enables support for LargePages (experimental). In reality, it doesn’t improve anything, so I’ll probably remove it in the next release.
Minor fixes for very old compilers (slackware) and others
It’s difficult to maintain compatibility with very old compilers, 32-bit systems, and so on. However, I try to do so with a minimal number of differences compared to the more modern versions. I don’t always succeed, but after all, it’s a hobby project, so please be patient. On the other hand, if anyone has an unused Apple Mx machine, they can let me use it for testing on Apple Silicon
consolidatebackup command renamed to consolidate
List of common switches
You can quickly get with zpaqfranz h common
p7m
On Windows, there is the option to check the FEQ digital signature of hash files for updates. Here's the spiegone https://github.com/fcorbelli/zpaqfranz/wiki/Windows-update
Updated compatibility with HPPA CPUs
Fixed a potential issue with CRC-32 calculation on CPUs that require very strict memory alignment (slower computation, but should be fine for very old and unusual processors, such as 32-bit HP RISC).
IPv6 support (untested)
Using the compilation define -DIVP6
should, and I emphasize "should," make the upgrade command work on IPv6 systems as well. I don't have one available, so I cannot guarantee anything; it's impossible to test and debug. Maybe in the future.
HINT
You can often find small threads about the changes made within the issues, that is, at this link. You might find some "whys" about why certain things are done in a particular way.
https://github.com/fcorbelli/zpaqfranz/issues?q=is%3Aissue
Windows 32/64 binaries, 64 bit-HW accelerated
-backupzeta
This new switch, to be used in the backup command, generates checksums on the fly, without reading them at the end of the .zpaq file creation, saving a lot of time, especially on slow disk drives like HDDs. The generated checksums are quite robust ("almost" XXHASH64 and CRC-32). In the future, it will also support encrypted volumes #139
Improved build compatibility on BSD operating systems
- OpenBSD
- NetBSD
- DragonFly
Set the creation date of .zpaq archives to 1/1/1980
so it is easy to quickly identify those that were not written completely #138
New hash algorithms ZETA and ZETAENC.
Can be selected with -zeta and -zetaenc. You can find the explanation here #139 (comment)
-destination
With the new switch, it is possible to load a series of lines (from a text file) as if they were multiple -to options. The explanation is here #136 (comment)
-nodelete
This switch does not mark files as deleted if they are not found during the path scan. It is used for bulk manipulation of the file list. The explanation is here #136 (comment)
-salt
This switch forces an empty salt (i.e., 32 bytes of zeros). It is something you should NOT normally use. It serves as a development mechanism (i.e., it is something that is useful for ME, not for YOU).
-hdd
With this switch, you use the computer's memory (including virtual memory, i.e., the swap file) to sequentially write the extracted data. It is useful if you have a lot of RAM or an SSD system drive and want to extract to an HDD. In this case, everything will first be decompressed into RAM and then written to the HDD, without any seeks (head movement) in the output. It is not suitable for gigantic files, but for medium sizes, it can halve the extraction time.
#135
-ramdisk
Internal for -hdd
fix on -input
Minor bug fixed on Windows
warnings in yellows
Because sometimes colours... counts...
Everywhere
This release includes numerous new features; hopefully, not too many bugs 😄
- The license of one component has been changed to make it compatible with Fedora policies.
- It's now possible to create a file with a list of errors to reduce confusion in the logs (-errorlog).
- Captchas can be bypassed using -nocaptcha.
- By default, it now uses the number of physical CPUs (no Hyperthread); you can use -ht to bypass this and return to the previous method.
- On Solaris, it (should) now detect the number of CPUs.
- Error messages now appear in red by default.
- UTF8 file output on Windows has been completely redone. New work command added.
- zpaqfranz tracks memory usage (more or less) in the very last output line.
- It shows more understandable error messages (!) when running out of memory due to overly small fragments.
- Output lines have been renumbered.
- With the -input switch, you can load a list of files to be added.
- The -715 switch in the l (list) command restores an output that is nearly identical, binary-wise, to zpaq 7.15.
- In the sfx command (for Windows), it's possible to extract the zpaq from the exe using the -to switch.
- In the sum command, the -home switch replaces -checksum.
- On Windows, the -fixreserved switch removes the ":" character during extraction.
- Sometimes, it heuristically takes the name of an existing backup.
- more
I am, laboriously, updating the wiki, so I do not feel like making a very detailed list of changes
If you want to elaborate, or to report bugs, feel free to open an issue
https://github.com/fcorbelli/zpaqfranz/issues
Windows 32/64 binaries, 64 bit-HW accelerated
This release contains a large number of new features and therefore, as usual, may contain a new bugs.
So be careful, test on your own that the archived files are truly archived correctly.
Some features are introduced to facilitate my work, that is it's possible to obtain the same result with a combination of various other tools (awk, grep etc), but they are complicated to use and fine-tune.
While the *nix logic is do one thing, and do it well, that of zpaqfranz is the opposite
Do everything I need. Because, often, you don't have the tools you would like (e.g. ESXi, NAS and so on)
Better -stdin management
Now files added with -stdin should be deduplicated as well as those added manually (up to the previous version the deduplicator was less efficient, for a whole series of reasons long to explain)
New comparehex command
This command performs an operation that happens very frequently, namely comparing two lines (typically HASH codes) present in two files. If the codes are the same it exits with OK. You can also impose the expected length of the hash.
Attention, all non-HEX characters are discarded
Example:
zpaqfranz comparehex z:\1.txt z:\2.txt "GLOBAL SHA256:" 64
New count command
Counts within more than one file the occurrences of a certain string, and returns OK if the number is the expected one In essence, within a log file, it counts how many positive results there are.
If you don't select a string to search for, it will use the default one for OKs with the -big switch
Example:
zpaqfranz count z:\*.txt 3 "all OK"
zpaqfranz count z:\*.txt 10
Better Control-C handling
During program termination with Control-C press, a better management of the deletion of files created with -chunk is performed.
It might also perform rollback of .zpaq files, maybe in the future.
Apparently it should work in a portable way across various platforms, or at least I hope so.
The problem of terminating a multithreaded system based on a condition is not trivial, even from a performance perspective. I used "dirty tricks", we'll see in the future.
gettempdirectory
Improvement in the creation of temporary files, for the use of multiple zpaqfranz processes simultaneously
Temporary files are created inside different subfolders (marked with mtime) in order to reduce the risk of "collision" if
multiple zpaqfranz are launched simultaneously.
new work command with many verbs
Used to automate many convenient functions for backup logs.
Maybe for most users they are useless, but they make my work a lot easier to prepare easily readable log files.
Write something big work big "count the ok"
Write something big work big "COUNT THE OK"
Write something big work big forza inter
Pad number 123 to 00000123 work pad 123
Pad number 123 to 0123 work pad 123 -n 4
Extract path work filepathnotrailing z:\doc\2\3.eml
Write datetime work date -terse
Write year_month_day work date "%year_%month_%day" -terse
Write ----- work printbar -terse
Write !!!!! work printbar "!" -terse
Newer -crc32 switch for t (test)
Run a triple CRC-32 check (!) against the filesystem
Use -find/replace to fix path (if needed); -ssd for M/T
Translation
The t command (without a path) do NOT read something from filesystem
You have to use the t command WITH a path (or use the -verify switch)
The brand new -crc32 switch is makes a triple CRC-32 comparing, READING from the filesystem.
In this case, the -find and -replace switches can be useful to transform the stored paths into physical paths (if they are different)
What's so exciting about it?
The first CRC-32 is the one calculated by the filesystem during the file addition phase
The second CRC-32 is the one RE-calculated from the blocks of the compressed file.
In other words, it's the CRC-32 of the file once extracted and written to the filesystem
The third CRC-32 is the one that is re-read from the filesystem
If these three CRC-32s match, it means that the file (during processing) has not been modified, and that the compressed file once decompressed will have the same CRC-32 that the file now has in the filesystem.
In short, that by DECOMPRESSING it will be correct
Obviously it is NOT equivalent to the p command (paranoid), to t -paranoid and not even to w, where files are really extracted.
But it's enormously faster, and can process in multithread with the -ssd switch
Triple check CRC-32: t z:\\1.zpaq -crc32");
3 check w/path manipulation: t z:\\1.zpaq -crc32 -find \"x:/memme/\" -replace \"c:/nz/\"");
New -terse switch
Almost everywhere it transforms the output into a version without additional information
That is, it shows the lines typically for redirection to other programs.
If you think about the difference in output between the ls command on Linux and dir on Windows, it will be immediate.
New -csv and -csvhf switches for l (list)
Using these switches you can get an output more easily processable with other programs
In other words, if you want to extract the list of files in a zpaqfranz archive, for example to load it into an Excel file or wherever you want, you can use these switches, using \t instead of TAB.
Keep in mind that -csvhf stands for "csv header footer", that is any string indicated at the beginning and end
For example pippo pluto paperino can become "pippo","pluto","paperino"
Otherwise it would remain pippo","pluto","paperino
TAB-delimited CSV output l z:\1.zpaq -terse -csv "\t"
Enclose fields between double quote l z:\1.zpaq -terse -csv "\",\"" -csvhf "\""
-external in add
It is now possible to specify a command that is executed BEFORE the add function, save its output to a text file, and automatically add it to the archive.
It can be used for activities such as creating .zfs snapshots, but also - and above all - to perform an independent check, for truly paranoid people
Translation
Let's say you want to have a... squared certainty that your files are truly identical to the original ones (we are in the hypothesis of using some remote backup system, so you CANNOT compare the files with the local filesystem, because they are not there)
But zpaqfranz can have bugs, right?
You can use zpaqfranz for a long time with the -hashdeep switch, which will generate a file in the
hashdeep format
But it's still using zpaqfranz: any bugs could still affect its reliability
For delicate situations we use an external program, which therefore doesn't have the possible bugs of zpaqfranz, such as hashdeep (in the Windows version hashdeep64 is faster)
Now we will store in a virtual file (of zpaqfranz) VFILE-l-external.txt the list of sha256 hashes generated by hashdeep (or any other similar program).
In this case the string %files and $files is replaced with the files specified in zpaqfranz (they are different for Windows and Linux).
In this example, the command will be executed
c:\nz\hashdeep64 -r -c sha256 c:\nz >temporaryfile.txt
The temporary file will be compressed as usual and added to the version
zpaqfranz a z:\2.zpaq c:\nz -external "c:\nz\hashdeep64 -r -c sha256 %files"
Now let's send the .zpaq file to another computer "somehow".
Let's suppose now we extract the c:\nz folder (object of the example) into the z:\memme folder
zpaqfranz x z:\2.zpaq -to z:\memme
We will now find in the z:\memme folder a folder c (containing the content of the files) AND also the VFILE-l-external.txt file, which contains the output of hashdeep
Now we can use hashdeep "by hand" to test the identity of the stored hashes with those of the folder
If we are not so paranoid there is the zpaqfranz versum command that understands the hashdeep format and allows to manipulate paths
So pay attention to -find and -replace
zpaqfranz versum z:\memme\VFILE-l-external.txt -hashdeep -find "c:\nz" -replace "z:\memme\c\nz" -ssd
-external in the x extract command
You can directly extract the virtual external file, to have an output to redirect
Show external VFILE x z:\2.zpaq -external -silent
-silent, in this example, disables the output of zpaqfranz. You can then, if needed, easily redirect for example >mygoodoutput.txt
Various minor fixes
Increased the maximum number of versions shown by the i command
-symlink in add (on Windows)
Ignores symlinks completely. The magical world of NTFS is so complex that every now and then
a more or less unknown functionality pops up. Anyway for one of my clients I need to ignore them, so I do it. It will probably be evolved in the future.
Always show output if there's the -big switch
By limiting the output with various switches the final OK output (gigantic) is still written
It's for the recognition (with the count command) of logs that went well
In essence, it reduces the verbosity of the output, but maintaining the final result
Execution dates are stored in backups
And the testbackup command shows the last one (if it exists)
This way you can understand if an automated backup procedure works
I had to use very dirty tricks to maintain backwards compatibility with older zpaqfranz, but it should work
newer franzomips CPU
Added some new CPUs to the franzomips list
It's possible to set file dates with -touch X
During the a add phase, you can force the timestamp of added files with the X switch
where X represents a date or a date and time
It also works for files added with -stdin
Windows + ARM NAS + Intel NAS + ESXi
Faster Windows 64-bit binary
At least on AMD runs 5% (average) to 20% faster (best case)
C:\zpaqfranz>release\60_1\zpaqfranz a "" c:\dropbox -noeta
zpaqfranz v60.1k-JIT-GUI-L,HW BLAKE3,SHA1/2,SFX64 v55.1,(2024-07-01)
(...)
Files added +17.218
21.938 seconds (00:00:21) (all OK)
C:\zpaqfranz>zpaqfranz a "" c:\dropbox -noeta
zpaqfranz v60.5d-JIT-GUI-L,HW BLAKE3,SHA1/2,SFX64 v55.1,(2024-07-20)
(...)
Files added +17.218
18.875 seconds (00:00:18) (all OK)
New switch -stat in add
Show files added/removed/updated
zpaqfranz a z:\test.zpaq c:\zpaqfranz\*.cpp -stat
New switch -stat in i (info)
Show uncompressed data size (slower)
zpaqfranz i z:\test.zpaq -stat
New switch -quick in test with paths (do not hash, only size/date)
zpaqfranz t z:\test.zpaq c:\zpaqfranz\*.cpp -quick
The testbackup command show global SHA256 (of hashes) in verify
Quick look comparing local and cloud backups
Local Synology drive
ash-4.4# /volume1/copie/bin/zpaqfranz_nas testbackup /volume1/copie/cloud/nas_test
zpaqfranz v60.5e-NAS-L(2024-07-20)
franz:testbackup _ - command
============================================================================
part0 /volume1/copie/cloud/nas_test_00000000.zpaq i_filename /volume1/copie/cloud/nas_test_????????.zpaq
Multipart backup looks good
Loading backupfile... /volume1/copie/cloud/nas_test_00000000_backup.txt
Rows in backup 00000026 from 00000001 to 00000026
Enabling XXH3 (in reading) hasher
Quick hash check
GLOBAL SHA256: EDBEE1D3D826F07D476B1332E21E0147C0CBE7AE9142E443497BF4D7A8C9DBA5
Chunks checked OK: 26 (233.636.698.925 @ 77.878.899.641.666/s)
0.003 seconds (00:00:00) (all OK)
Remote FreeBSD server.
root@franco:/tmp/zp # /usr/local/bin/zpaqfranz testbackup /home/moz1/cloud/nas_test
zpaqfranz v60.5e-JIT-L,HW SHA1/2,(2024-07-20)
franz:testbackup _ - command
*** WARNING: original backup path <</volume1/copie/cloud/>> != current index path <</home/moz1/cloud/>>
*** I highly recommend using -paranoid! ***
*** I highly recommend using -paranoid! ***
*** I highly recommend using -paranoid! ***
============================================================================
part0 /home/moz1/cloud/nas_test_00000000.zpaq i_filename /home/moz1/cloud/nas_test_????????.zpaq
Multipart backup looks good
Loading backupfile... /home/moz1/cloud/nas_test_00000000_backup.txt
Rows in backup 00000026 from 00000001 to 00000026
Enabling XXH3 (in reading) hasher
Quick hash check
GLOBAL SHA256: EDBEE1D3D826F07D476B1332E21E0147C0CBE7AE9142E443497BF4D7A8C9DBA5
Chunks checked OK: 26 (233.636.698.925 @ 38.939.449.820.833/s)
0.006 seconds (00:00:00) (all OK)
GLOBAL SHA256 is the very same (this is GOOD!)
For paranoid people a -verify will ask for a full-sized rehashing of everything
de-overloaded fclose()
Helpful for debugging
minor fix in chunked add
A possible double close file
Windows 32/64 binary, 64 bit-HW accelerated
-nosynology
Switch to disable hidden-system Synology's folders during a
*/@recycle/*
*/#recycle/*
*/#snapshot/*
*/@ActiveBackup/*
*/@sharebin/*
*/@appconf/*
*/@appdata/*
*/@apphome/*
*/@appshare/*
*/@apptemp/*
*/@clamav/*
*/@database/*
*/@docker/*
*/@eaDir/*
*/@maillog/*
*/@MailScanner/*
*/@postfix/*
*/@S2S/*
*/@sharesnap/*
*/@synoconfd/*
*/@SynologyDrive/*
*/@SynoFinder-etc-volume/*
*/@SynoFinder-log/*
*/@tmp/*
*/@USBCopy/*
*/@userpreference/*
Fixes for "strange" things
Should compile and run on Haiku and Macintosh and Solaris
Celeron J4125 bench
Added franzomips for Synology DS224+ NAS
isjitable
Inside the b (benchmark) command a WARNING is issued if the source code is compiled without -DNOJIT, but the CPU does non seems Intel/AMD. Will be evolved in future
Some examples
root@theserverone:~# zpaqfranz b
zpaqfranz v60.4b-JIT-L,HW SHA1/2,(2024-07-12)
This seems Intel/AMD 'normal' CPU
uname x86_64
root@franco:~ # zpaqfranz b
zpaqfranz v60.4b-JIT-L,HW SHA1/2,(2024-07-12)
I am not sure this is Intel/AMD CPU (virtual? arm?)
WARNING: non Intel/AMD CPUs should be compiled with -DNOJIT
uname amd64
With -debug you can get some more infos (if available)
root@franco:~ # zpaqfranz b -debug
38761: error on realpath of zpaqfranz
41554: Pretend to be FreeBSD
43948: This seems a LITTLE ENDIAN CPU (aka:'normal')
zpaqfranz v60.4b-JIT-L,HW SHA1/2,(2024-07-12)
46184: FULL exename <</usr/local/bin/zpaqfranz>>
42993: The chosen algo 3 SHA-1
04058: SSSE3 :NO
04064: SSE41 :NO
04070: SHA :NO
46776: franz:-debug -verbose
98567: Multiple not-intel vendor_id: Common KVM processor
68456: I am not sure this is Intel/AMD CPU (virtual? arm?)
68467: WARNING: non Intel/AMD CPUs should be compiled with -DNOJIT
In this example the vendor_id (Common KVM processor) is "strange" => warning
There is no easy, portable way across platforms to figure out exactly what type of CPU.
As mentioned, over tim, I will improve
Binary for everywhere
Version 60.3, with a few microfixes for compilability on various systems
Sorry, no Mac today 😄
- zpaqfranz.exe Windows 64 bit
- zpaqfranz32.exe Windows 32 bit
- zpaqfranzhw.exe Windows 64 bit with SHA-1 asm (AMD)
- zpaqfranz_armv8 ARM-NAS compatible build (QNAP, Synology)
- zpaqfranz_esxi ESXi build
- zpaqfranz_freebsd FreeBSD 64 bit
- zpaqfranz_linux32 Generic Linux 32 bit
- zpaqfranz_linux64 Generic Linux 64 bit
- zpaqfranz_nas Generic Linux 64 bit Intel-powered NAS
- zpaqfranz_openbsd OpenBSD 64 bit
- zpaqfranz_haiku Haiku 64 bit
Windows 32/64 binary, 64 bit-HW accelerated
Improved dump command
Show the technical information of the archives in a more readable way (!) and, in particular, the type of archive format
Mind you, it is not meant to work on giant files; it is just a debugging tool. It can therefore crash for large archives (multigigabyte), this is normal, it get a very naive approach. Maybe in the future I will write a better implementation, for now it is enough for me.
There are 3 switches
- : -summary Brief
- : -verbose Show useful infos
- : -all A bit deeper
BTW will show archive compatibility level
60+ = zpaqfranz from v60 and more ~jul 2024
<60 = zpaqfranz up to v59.x
715 = standard zpaq 7.15 (or zpaqfranz -715)
I think it is pretty clear
C:\zpaqfranz\release\60_2>zpaqfranz dump z:\715.zpaq -summary
zpaqfranz v60.2b-JIT-GUI-L,HW BLAKE3,SHA1/2,SFX64 v55.1,(2024-07-03)
(...)
This seems a 715 archive (original zpaq format)
In this example there are older FRANZOBLOCKs with XXHASH64 (older hash, not "b")
C:\zpaqfranz\release\60_2>zpaqfranz dump z:\v59.zpaq -summary
zpaqfranz v60.2b-JIT-GUI-L,HW BLAKE3,SHA1/2,SFX64 v55.1,(2024-07-03)
(...)
FRANZOBLOCK 050 24
HASHTYPE XXHASH64 24
This seems <60 archive (older zpaqfranz v59)
This is a newer HASH64B ("binary") => a mark 60 zpaqfranz
C:\zpaqfranz\release\60_2>zpaqfranz dump z:\v60.zpaq -summary
(...)
FRANZOBLOCK 050 24
HASHTYPE XXHASH64B 24
This seems 60+ archive (newer zpaqfranz v60)
This is a more interesting example. It is a mix of small (050) and large (190) blocks , composed of two different hash types, binary XXHASH64 and old whirlpool
It means that by using the -frugal command to append data with a short block format (i.e., with the default hash, i.e., without any particular indication) the program will CRASH . Which is normal (in this case)
C:\zpaqfranz\release\60_2>zpaqfranz dump z:\v60_mixed.zpaq -summary
(...)
FRANZOBLOCK 050 24
FRANZOBLOCK 190 88
HASHTYPE WHIRLPOOL 88
HASHTYPE XXHASH64B 24
This seems 60+ archive (newer zpaqfranz v60)
*** WARNING DO NOT USE -frugal adding data to this archive!
Perhaps, in the future, I will put an automatic control. Short version: use the -frugal switch if you know what you are doing, otherwise avoid. It's for advanced users
Minor bug fixing
Could create unnecessarily large backup command files if there were no files to add
Windows 32/64 binary, 64 bit-HW accelerated, Intel/arm based
This branch introduces several new features, also new bugs, so it is up to you to use it with awareness
I understand that the "spiegone" is heavy, however, the alternative is to write nothing at all
The biggest improvement is the use of a new binary format for storing FRANZBLOCKs.
These are (or rather can) be smaller, reducing (though not by much) the overall size of the archive file.
The side effect is the possibility of "breaking" something.
To maintain some degree of safety, I kept all the old switches and added new ones that end in b (as binary)
So the new blake3 storage format is called -blake3b, while the "historical" one (up to version 59) is called -blake3 (as usual)
Warning: if you do not specify any switch the new default binary format -xxhash64b will be used. If you want to use an old format you MUST specify it explicitly
-frugal
There is one additional significant modification, which can lead to program crashes if misused.
It is the -frugal switch, which reduces memory consumption in storing internal structures (significantly)
This makes sense for storing archives with large amounts of files (10 or 100 MILLION) where the memory occupied can become very large, even exceeding the physical memory of the computer (yes, some people use zpaqfranz to store tens of millions of files)
There are no particular problems in using it IF you do not mix different types (in size of FRANZOblocks)
Translation
Suppose you store within an archive files always with the same hash (the default is xxhash64b).
It doesn't matter which one it is, small (e.g., blake3) or large (e.g., whirlpool)
The problem arises when you create an archive with a small hash (blake3 for example) then add data
with a larger hash (ex whirlpool) and then you want to add data with a smaller one (suppose sha3)
This "mix" of formats will crash zpaqfranz if you use the optional -frugal switch.
So if you always use the same hash you have no problem, you can use -frugal (should you have problems with giant numbers of files)
Or don't use it and live happily
Short version: if you use -frugal DO NOT mix different types of hashes.
If you don't have tens of millions of files to add, -frugal is not needed.
Backward compatibility
Barring bugs the new format can be listed, extracted and added by both zpaq and zpaqfranz up to version 59
Archives created with version 60 (and the new binary format) cannot be tested by earlier versions.
Basically everything works, but the t (test) command loses the ability to check CRC32 codes and, in general, all commands cannot read stored hashes (v command and so on).
Therefore if for some reason you want to maintain 100% backward compatibility with version 59 you have to specify the older versions of the switches (e.g. -xxhash64 or -blake3 or maybe -sha3). Those without the final binary "b".
There is a new -date switch that shows the date the files were created (stored in the new format)
The new format, in addition to being smaller when compressed, also stores the date the files were created. Always (on Windows, by default) and optionally for *nix systems (with the -date switch).
This difference is given by the slower processing of this information (modest on Windows, heavy on *nix)
Also - as is well known - there is no "true" method to determine the creation date on non-Windows systems (the so-called birth date) so zpaqfranz uses a heuristic for the various operating systems, with no guarantee of perfect operation (not my fault)
The new format also stores the number of files added, modified, and deleted for each version
Command l list
The l command has been heavily rewritten, result now slower (!) though more beautiful.
Introduces additional information, such as an estimate of file compression ratio, number of files added (+), modified (#) or deleted (-)
Auto-resizes column size width to maximize space for file names
Uses colors by default (disabled during redirection to files either with system variable NO_COLOR or switch -nocolor to disable)
Changes way of showing up with switches -attr, -checksum (e.g., -crc32), -date, -terse)
A lot of operations are performed that slow down, maybe in the future I will put an uglier but faster version.
More informations here
New list and ads: no way
ADS management has not yet been aligned, so, in case you want to force the recalculation, remember that you can force it to
Backup command with -index
I had a request to make the indexes created by the backup command "mobile" (with the -index switch somewhere). That is, to detach the index files from the archives containing the .zpaq data. The purpose is to make it compatible with Worm systems, writing the data inside a "protected" folder and the indexes in a "normal" one
This, however, makes the backup more fragile, as there is nothing to prevent pairing the WRONG indexes with the .zpaq data
It is up to you to be aware of this and operate accordingly. zpaqfranz cannot do miracles
More informations here
t command with parameters and replace/find/to
A user requested the possibility of using the -test switch, with substitution of strings operated by, for example, -find/-replace, during a test command with parameter (!)
It is quite a complex and specific topic, I leave a link with detailed explanations
More informations here
There is a new compilation switch, -DNAS, which is used to make a mixture of reducing
memory usage and maintaining functionality (in short, it is a relaxed ANCIENT)
It serves, typically, to create executables compatible with the popular Synology and QNAP NAS
autotest
The self-test -all -to something command has been extended considerably.
The generated dotest file operates many more tests, in more areas, some of them deliberately to expected failure,
in short to get a picture I hope fairly reliable
I do not exclude that under certain circumstances it may give false positives, that is, detect errors that are not actually there.
To date it is tested on Windows, Debian, FreeBSD, and QNAP-arm, I have not carried out tests on "weird" things like Macintosh, Solaris, and so on.
The completion time is longer (depends on the speed of the system, of course) even a few tens of minutes or hours on cheap NASes and the space needed is about 15GB (maybe it is too big for small VPS, I will see maybe later)
If you have come this far, and you are very confused, patience
If you have problems open an issue on github or send me a direct email
Remember: the continuous improvement (kaizen) process of zpaqfranz depends on YOU
The more reports and suggestions, the better the program will become