Releases: fcorbelli/zpaqfranz
Windows 32/64 binary
New commands
password
Set, change or remove password from single archive (NO multipart)
Please note the new -key2 switch; others are -space (do not check writeability), -force (overwrite output)
Change pass from X to Y: password z:\x.zpaq z:\y.zpaq -key X -key2 Y
Add password X to NON encrypted: password z:\nocrypt.zpaq z:\yescrypt.zpaq -key2 X
Remove password X from encrypted: password z:\crypt.zpaq z:\nocrypt.zpaq -key X
To remove a password (last example) you should enter, when asked, . (a single dot) as destination password
Please be aware: changing or removing password of an encrypted archive, inserting the WRONG password, will generate SILENTLY corrupted output
Why do not check before? To make it less immediate use brute force scripts
This type of encryption is designed to operate on all blocks (read everything from the file, encrypt, write out); the -repack function of the x command remains active if you want to operate on the referenced blocks
trim
Trim incomplete .zpaq file (NO multipart, only single), removing incomplete transaction
Thanks to JaitinPrakash #16
Dry run
zpaqfranz trim z:\1.zpaq
Suggested usage (safer: make a backup, trim the backup)
zpaqfranz trim z:\1.zpaq -to d:\tiny.zpaq
In place trim (risky!)
zpaqfranz trim z:\1.zpaq -kill
dirsize
show cumulative folder(s) size
Search (by default case insensitive) the rightmost portion of a path
H:\rambo\cloud>zpaqfranz dirsize rambo2.zpaq marcus vanessa
zpaqfranz v54.15-experimental (HW BLAKE3), SFX64 v52.15, compiled May 20 2022
Block 8 K 15.503 (block/s)
31038: case iNSeNsITIVE right search, because NO -force switch
7.895.331.940 [2022-05-05 19:47:56] <</tank/d/documenti/MARCUS/>>
851.602.330 [2019-04-03 17:19:40] <</tank/d/documenti/vanessa/>>
0.984 seconds (00:00:00) (all OK)
With -force do a case sensitive find
In this example vanessa do not match
H:\rambo\cloud>zpaqfranz dirsize rambo2.zpaq /tank/d/documenti/MARCUS vanessa -force
zpaqfranz v54.15-experimental (HW BLAKE3), SFX64 v52.15, compiled May 20 2022
Block 8 K 15.037 (block/s)
31019: case sensitive == search, because of -force switch
7.895.331.940 [2022-05-05 19:47:56] <</tank/d/documenti/MARCUS/>>
31094: folder searched 2 != founded 1
0.953 seconds (00:00:00) (with warnings)
dir mode
On *nix systems: if the executable name is dir... emulate Windows' dir
The dir command (shows cumulative size, free space) can be "compressed" by placing a copy of zpaqfranz, named dir, in the path
Usually changing the Makefile, adding the last line
install: zpaqfranz
install -m 0755 -d $(DESTDIR)$(BINDIR)
install -m 0755 zpaqfranz $(DESTDIR)$(BINDIR)
install -m 0755 zpaqfranz $(DESTDIR)$(BINDIR)/dir
I really, really do NOT like ls behavior
root@aserver:/tmp/zp # dir /root /os
==== Scanning dir <</root/>> /os
2022-05-20 10:30:06 <DIR> /root/
2017-08-11 18:04:33 <DIR> /root/.cache/
2018-10-17 15:05:45 <DIR> /root/.config/
(...)
2018-10-17 15:34:03 <DIR> /root/VirtualBox VMs/
2022-05-11 16:15:45 0 /root/expr
2022-05-18 17:13:04 <DIR> /root/script/
2021-02-17 08:25:41 28 /root/.lesshst
2020-09-09 17:24:57 96 /root/.detoxrc
(...)
2022-05-03 12:39:19 139.374 /root/mbox
14 File 202.081 byte
10 directory 141.117.390.848 bytes ( 131.43 GB) free
0.001 seconds (00:00:00) (all OK)
minor fixes
Hidden SFX module (for 32-bit Windows) is visible
Some refactoring in help
Windows 32/64 binary
Debug switches to add/extract files filled with zero
Much easier to send problematic archives for debug, respecting privacy
In command x (extract) new combo switch -zero -debug
Extract all files but... filled with zeros
In command a (add) new combo switch -zero -debug
Add all files but... filled with zeros
In command r (robocopy) new switch -zero
Fill the output with zeros
In command f (fill) switch -verify (for -zero)
Turn on aftercheck, debuggin for compressed fs like zfs
Command dir
Now support all hashers
-md5 -xxh3 -sha3 -blake3 ... for the duplicate finder
*On nix transform symlinks to "real"
takes care of zfs' sometime strange relative links for command dir
Minor changes
Help of command p (paranoid)
Directly kill VIRTUAL files in paranoid test
Kludge for getting correct size of NTFS's symlinks
Support for "strange" NTFS symlinked files (folders already supported)
Searching for -comment is now case INsEnsiTive
Windows 32/64 binary
NO differences on 54_12 (final)
First try for an archlinux automatic package-builder (the related package manager is quite fussy)
Windows 32/64 binary
Untested features have been added, hoping to get some feedback from users
Initial support, on Windows, for long filenames (>255)
switch -longpath
Add the -longpath and... should work (it is a feature, afterall)
zpaqfranz a z:\1.zpaq z:\myfolder -longpath
zpaqfranz a z:\2.zpaq \\franzk\z\myfolder -longpath
Extracting to a local folder (maintaining >255 filenames) should work, by using -to something and -longpath
zpaqfranz x z:\1.zpaq -to z:\here -longpath
Extracting from UNC (Uniform Naming Convention \host-name\share-name[\object-name], aka: network shares) is a mess: a lot of "kludges" inside the code, Microsoft's documentation is not... very good
The "easier" is extract files stored from UNC to a LOCAL drive, with a SINGLE -to
zpaqfranz x z:\1.zpaq -to z:\here -longpath
A special function is triggered: the "\\server\share\something" will be converted to "__server_share\something"
Harder mode: use -find and -replace to do "manual" adjustments
Restore to local drive (suppose backup of the \\franzk\z\NS folder)
zpaqfranz x z:\1.zpaq -find //franzk/z/NS -replace \\?\z:\here -longpath
Restore a longpath UNC to longpath UNC (network share to network share, like Linux-based NAS etc) is doable with -find and - replace
zpaqfranz x z:\1.zpaq -find //franzk/z/NS -replace \\?\UNC\nas\share\somewhere -longpath
zpaqfranz now show warning if "long" files are elaborated on Windows
On Windows, by default, shows time in local and not UTC
Disable by new -utc switch
Fix: do not "autocheck" password on multivolume archives
Very experimental (in fact "invisible") extremely paranoid check
The possibility of testing large archives (more than the free space available) is under development, if the largest file contained therein is smaller in size
Too immature, its use is not recommended
-search and -replace switch in extract
It is possible to search and replace the header of filenames' path
Example, files originally stored in
/tank/dati/condivisioni/(...)
zpaqfranz x 1.zpaq -find /tank/dati/ -replace z:\pippo\
or
zpaqfranz x 1.zpaq -replace z:\pippo\
new -entropy switch on sum()
Calc the Shannon's entropy of some files. Quick-and-dirty check to find hard-to-compress file (8.0000)
zpaqfranz sum d:\*.jpg -entropy
Windows 32/64 binary
New command cp
cp and copy commands (*nix and Win) does not have any progression feedback
the cp command copy one (or more) file/s (wildcards allowed) into a destination folder -to
-verify does a XXH3 check of written data
-paranoid does NOT overwrite existing files, but create unique filename (ex a.txt => a_00001.txt)
-force do not check if destination is writeable
-append (future extension for .zpaq files)
C:\zpaqfranz>zpaqfranz cp b*.cpp *.txt -to z:\prova
zpaqfranz v54.11-experimental (HW BLAKE3), SFX64 v52.15, compiled Dec 28 2021
Your friendly neighborhood cp/copy
Space needed Space free
1.186.658 14.748.295.168 ben.cpp
1.239.466 14.747.107.328 bk1.cpp
1.239.466 14.745.862.144 bk2.cpp
1.239.292 14.744.621.056 bk4.cpp
1.236.885 14.743.379.968 bk5.cpp
1.236.885 14.742.142.976 bk6.cpp
181 14.740.905.984 1.txt
6 14.740.905.984 2.txt
20.962 14.740.905.984 3.txt
1.186.658 14.740.905.984 ben.cpp
1.239.466 14.740.905.984 bk1.cpp
1.239.466 14.740.905.984 bk2.cpp
1.239.292 14.740.905.984 bk4.cpp
1.236.885 14.740.905.984 bk5.cpp
1.236.885 14.740.905.984 bk6.cpp
2.104 14.740.905.984 news.txt
5.865 14.740.905.984 quickstart_bsd.txt
---------------------
14.786.422 bytes written
14.786.422 bytes expected
17 files OK
17 files expected
0.125 seconds (00:00:00) (all OK)
Check writeability / free space in x (extraction)
A long-awaited test: is extraction path writable ? is there enough space?
zpaqfranz x pippo.zpaq -to mk:\
zpaqfranz v54.11-experimental (HW BLAKE3), SFX64 v52.15, compiled Dec 28 2021
Cannot write on <<-to mk:/>>
Aborting. Use -space to bypass and enforcing.
0.000 seconds (00:00:00) (with errors)
use -space to bypass and try
minor fix in zfs function (better create and delete of temporary script)
Windows 32/64 binary
Media "fullness" check
zpaq 7.15 does not throw errors when the free output space become too low
In this example the uncompressible file fill the available space, giving a corrupted zpaq archive but ending with "all OK". This is not good
zpaq64 a z:\ugo k:\2014.mp4 -summary 1
zpaq v7.15 journaling archiver, compiled Aug 17 2016
Creating z:/ugo.zpaq at offset 0 + 0
Adding 2256.846154 MB in 1 files -method 14 -threads 32 at 2021-12-20 16:43:19.
1 +added, 0 -removed.
0.000000 + (2256.846154 -> 2256.846154 -> 1415.975911) = 1415.975911 MB
8.875 seconds (all OK)
zpaqfranz 54.10 now compares bytes to be "theoretically" written with bytes that are reported as written
zpaqfranz a z:\ugo k:\2014.mp4
zpaqfranz v54.10d-experimental (HW BLAKE3), SFX64 v52.15, compiled Dec 20 2021
Integrity check type: XXHASH64+CRC-32 + CRC-32 by fragments
Creating z:/ugo.zpaq at offset 0 + 0
Adding 2.256.846.154 (2.10 GB) in 1 files at 2021-12-20 16:42:18
90.36% 00:00:00 ( 1.90 GB) -> ( 1.88 GB) of ( 2.10 GB) 243.09 MB/sec
1 +added, 0 -removed.
0 + (2.256.846.154 -> 2.256.846.154 -> 1.415.975.939) = 1.415.975.939
***********************************************************************************************
Something STRANGE happened. Archive seems corrupt. Media full?
WRITTEN BYTES 1.415.976.043
EXPECTED 2.257.392.747
***********************************************************************************************
10.250 seconds (000:00:10) (with errors)
The -debug switch show those errors
Here x (extract) says "all OK", but it is not
c:\nz\zpaq64 x macc.zpaq -to z:\muk -summary 1
zpaq v7.15 journaling archiver, compiled Aug 17 2016
macc_xxh3_G643p.zpaq: 115 versions, 1150 files, 10613081 fragments, 165389.322343 MB
Extracting 401005.451965 MB in 10 files -threads 32
435.360 seconds (all OK)
zpaqfranz 54.10+
zpaqfranz x macc.zpaq -to z:\kjn
zpaqfranz v54.10-experimental (HW BLAKE3), SFX64 v52.15, compiled Dec 20 2021
Archive seems encrypted (or corrupted)
Enter password :**************
macc.zpaq:
115 versions, 1.150 files, 10.613.081 fragments, 165.389.322.343 bytes (154.03 GB)
Extracting 401.005.451.965 bytes (373.46 GB) in 10 files -threads 32
99.66% 00:00:01 ( 372.21 GB) of ( 373.46 GB) 650.41 MB/sec
***********************************************************************************************
Something STRANGE happened. Archive seems corrupt. Media full?
WRITTEN BYTES 1.411.810.403
EXPECTED 400.082.097.403
***********************************************************************************************
588.141 seconds (000:09:48) (with errors)
New switch -checksum in command t (test)
Run hashtest after chunked-SHA1 against folders
The most reliable until p (paranoid) command and -paranoid switch
zpaqfranz a z:\1.zpaq c:\nz
zpaqfranz t z:\1.zpaq c:\nz -checksum
franzomips in command b (benchmark)
quick and dirty CPU speed comparison, either for single or multithread performances
zpaqfranz b
zpaqfranz b -all
franzomips are even worse than bogomips :)
A rating typically used for VPS and rented machines to get a rough idea of CPU performance (60s to not consume too much time). In the case of -all (multithreaded) run SHA-3 for all logical cores. You can use -tX to limit the number of tasks
BEWARE: no space, ex. -t4 for 4 threads
franzomips single thread index 1.093 (quick CPU check, raw 1.093)
Atom N2800 (phy) 4 268.60 %
Xeon E3 1245 V2 (vir) 4 45.27 %
i5-6200U (phy) 2 57.45 %
Xeon E5 2620 V4 (phy) 8 59.16 %
Xeon E5 2630 V4 (phy) 10 70.44 %
Xeon D-1541 (vir) 8 53.83 %
i5-3570 (phy) 4 36.98 %
i7-4790K (phy) 4 33.48 %
i7-8700K (phy) 6 32.56 %
i9-10900 (phy) 10 29.51 %
AMD-5950X (phy) 16 22.78 %
60.047 seconds (000:01:00) (all OK)
In the above example the single-core is ~ half speed of low-end Xeons
Windows 32/64 binary
New "hasher"
-nilsimsa
This is a similarity algo, experimental for different sorting method
https://en.wikipedia.org/wiki/Nilsimsa_Hash
Command a (add)
Custom sort of files (study reasons)
- -orderby X
ext;size;name;hash;date;data;nilsimsa;
zpaqfranz a z:\1.zpaq c:\nz -orderby ext;size;hash;
- -desc
descending sort - -nodedup
Do not deduplicate before compress archive. NOT faster (for backward compatibility), but to compare zpaq's compression against other software - -tar
Disable deduplication and compression, something like 'tar'. Can work with optional sorting -ordeby
Command r (robocopy)
- -flagappend
When robocoping a .zpaq do heuristic check of equality (first 64K, mid 64K, last 64K), then append data instead of overwrite from scratch
Scenario: after making an internal backup with zpaqfranz
(ie writing a zpaq archive containing the data to be saved on a different fast internal disk)
you want to make a second copy to a slow device (eg external USB HDD, NAS etc)
to keep it in a different place.
The -append switch, in case the destination files exist and are smaller than the original ones,
will copy only the final, additional portion of the files.
Typically reduces the time needed to update a copy by a factor of 100 or more,
compared to transferring (ex. robocopy) the entire (giant) files each time.
An heuristic check of equality is done (just to be sure): first 64K, mid 64K and last 64K of the .zpaq
Not perfect, but much better than rsync --append.
Note: enabled ONLY on .zpaq files
zpaqfranz a d:\myzpaqfolder\myverybackup.zpaq c:\datafoldertobesaved
zpaqfranz r d:\myzpaqfolder z:\mynasusbdrive -kill -append
instead of something like
zpaqfranz a d:\myzpaqfolder\myverybackup.zpaq c:\datafoldertobesaved
robocopy d:\myzpaqfolder z:\mynasusbdrive /mir
Command s (size)
- -minsize X
Show a warning if freespace less than X. Typically for backup script log, check NAS space etc
C:\zpaqfranz>zpaqfranz s z:\ -minsize 30000000000
zpaqfranz v54.9-experimental (HW BLAKE3), SFX64 v52.15, compiled Nov 4 2021
franz:minsize 30.000.000.000
Get directory size ignoring .zfs and :$DATA
04/11/2021 17:00:21 Scan dir <<z:/>>
z:/System Volume Information/*: error access denied
=====================================================================================================
Free 0 17.661.624.320 ( 16.45 GB) <<z:/>>
***************************************
|=|=|=|=|=|=|=|=|=|=|=|=|=|=|=|=|=|=|=|
WARN: free space < of minsize 17.661.624.320 30.000.000.000
|=|=|=|=|=|=|=|=|=|=|=|=|=|=|=|=|=|=|=|
***************************************
Everywhere
In archive and files substitute $hour $min $sec $weekday $year $month $day $week $date $time $datetime
Normally used on filesystems with deduplication (like zfs) to keep "versioned" copies with the r command (robocopy) in many different folders
zpaqfranz a z:\mycopy_$weekday.zpaq c:\datatobesaved
zpaqfranz r c:\datatobesaved \\nas\mybackup\$month -kill
root@prod113:~ # zpool status -D dedup
pool: dedup
state: ONLINE
scan: none requested
config:
NAME STATE READ WRITE CKSUM
dedup ONLINE 0 0 0
da2 ONLINE 0 0 0
errors: No known data errors
dedup: DDT entries 1628935, size 420 on disk, 315 in core
bucket allocated referenced
______ ______________________________ ______________________________
refcnt blocks LSIZE PSIZE DSIZE blocks LSIZE PSIZE DSIZE
------ ------ ----- ----- ----- ------ ----- ----- -----
1 2.57K 316M 254M 254M 2.57K 316M 254M 254M
2 3.86K 472M 409M 409M 7.78K 950M 821M 821M
4 107 12.5M 10.2M 10.2M 449 52.3M 42.2M 42.2M
8 33 3.90M 2.11M 2.11M 466 55.4M 30.6M 30.6M
16 1.48M 185G 170G 170G 32.1M 3.92T 3.61T 3.61T
32 57.0K 6.63G 5.95G 5.95G 2.45M 291G 261G 261G
64 11.4K 1.36G 1.18G 1.18G 848K 101G 86.6G 86.6G
128 2.80K 308M 193M 193M 479K 51.2G 32.6G 32.6G
256 1.14K 129M 66.6M 66.6M 442K 48.5G 23.1G 23.1G
512 853 86.1M 48.5M 48.5M 599K 61.2G 34.4G 34.4G
1K 353 38.0M 31.7M 31.7M 431K 46.6G 39.4G 39.4G
2K 12 643K 459K 459K 35.1K 1.70G 1.21G 1.21G
4K 2 128K 1.50K 1.50K 10.8K 602M 7.72M 7.72M
8K 1 512 512 512 8.92K 4.46M 4.46M 4.46M
Total 1.55M 194G 178G 178G 37.3M 4.51T 4.08T 4.08T
About 200GB of "used data", and ~4TB "robocopied"
Windows 32/64 binary
-
Always add .ppt (as .xls) by default
-
Some fixes
-
Strongest checks for "strange things" when adding (read the wiki)
-
Added -exec_warn to execute some script
zpaqfranz a d:\testme.zpaq *.cpp -exec_warn "c:\script\callmeifwarnraised.bat"
-
New rsync command (purge rsync orphaned files)
zpaqfranz rsync \\nas\thebackup
-
Changed -kill to -zero in x extract for only 0-bytes-long extraction
Windows 32/64 binary
- Support (maybe!) for ReFS-deduplicated VHDX
- Verify now optionally supports multithread (-all), just like summa() (the hasher)
On SOHO hardware (NVMe) in "real world" can run at 1-2GB/s even for "heavy checksums" (cryptohashes) like SHA-3 on modern CPUs
zpaqfranz v z:\1.zpaq -all
Please note: running multithreaded I/O task is OK for non-spinning media (SSD, NVMe, RAMdisk). On HDD can cause heavy "thrashing" by the high head latency, in fact running slower
Windows 32/64 binary
- Refactoring of source code in progress
- Some fixes
- Test (disabled for now) HW acceleration for SHA-1 on AMD Ryzen