Cannot allocate memory on M1 using higher compression levels

[demhademha]

The command: zpaqfranz a backup.zpaq backup fails with

Integrity check type: XXHASH64+CRC-32
all.zpaq: 
0 versions, 0 files, 0 fragments, 0 blocks, 0 bytes (0.00 B)
Updating all.zpaq at offset 0 + 0                                           
Adding 19.857.884.160 (18.49 GB) in 1 files (0 dirs), 8 threads -method 56 @ 2022-11-27 13:05:16 
(001%)   0.34% 00:01:41 (  64.04 MB)->(    0.00 B) of (  18.49 GB)   64.04 MB/sejob 1: allocx failed

[demhademha]

% zpaqfranz a a.zpaq ../all.tar -m5 -debug
zpaqfranz v56.1j-JIT-L archiver, (15 Nov 2022)
franz:DEBUG very verbose (-debug)
21026: check_if_password of a.zpaq
Default sort
k3
000 first ../all.tar
000 date 20.221.105.215.334
000 size 19.857.884.160
000 attr 8.496.245
000 data 8.386.109.325.481.705.858
000 writ -1
000 ptrsi 0
000 dtv 0
Creating a.zpaq at offset 0 + 0
Adding 19.857.884.160 (18.49 GB) in 1 files (0 dirs), 8 threads
20006: converting to localtime 2022-11-27 13:30:51
@ 2022-11-27 13:30:51
Real write
39481: g_cdatasize 0
39482: g_htsize 0
39431 writeJidacheader -1
(001%) 0.34% 00:01:42 ( 64.04 MB)->( 0.00 B) of ( 18.49 GB) 64.04 MB/sejob 1: allocx failed

[fcorbelli]

I do not have a M1, but we can make some tests
First try: please compile with -DANCIENT
Thank you

[fcorbelli]

56_2c.zip
Please run the pre-release build
Should show how much RAM is allocating
Thank you

[demhademha]

Here is the updated output:

% zpaqfranz a all.zpaq ../all.tar -m5 -debug   
zpaqfranz v56.2c-JIT-L archiver,  (27 Nov 2022)
franz:DEBUG very verbose (-debug) 
21026: check_if_password of all.zpaq
Default sort                                                                
k3
000  first ../all.tar
000  date  20.221.105.215.334
000  size  19.857.884.160
000  attr  8.496.245
000  data  8.386.109.325.481.705.858
000  writ  -1
000  ptrsi 0
000  dtv   0
Creating all.zpaq at offset 0 + 0
Adding 19.857.884.160 (18.49 GB) in 1 files (0 dirs), 8 threads 
20006: converting to localtime 2022-11-29 07:47:37
@ 2022-11-29 07:47:37 
Real write
39481: g_cdatasize  0
39482: g_htsize     0
39431 writeJidacheader -1
(001%)   0.34% 00:01:37 (  64.04 MB)->(    0.00 B) of (  18.49 GB)   64.04 MB/sejob 1:  2249: allocx failed for 12.288
 

[hugmouse]

Same for me. MacOS v13.0.1, M1, 8GB:

% ./cmake-build-debug/56_2c a .test-m5.zpaq ~/Documents/Archive.zip -m5 -debug    
zpaqfranz v56.2c-JIT-L archiver,  (27 Nov 2022)
franz:DEBUG very verbose (-debug) 
21026: check_if_password of .test-m5.zpaq
.test-m5.zpaq: 
0 versions, 0 files, 0 fragments, 0 blocks, 0 bytes (0.00 B)
Default sort                                                                                                                                                                                          
k3
000  first /Users/mysh/Documents/Archive.zip
000  date  20.221.129.082.534
000  size  5.737.869.867
000  attr  8.496.245
000  data  8.820.704.490.500.564.988
000  writ  -1
000  ptrsi 0
000  dtv   0
Updating .test-m5.zpaq at offset 0 + 0
Adding 5.737.869.867 (5.34 GB) in 1 files (0 dirs), 8 threads 
20006: converting to localtime 2022-11-29 08:25:47
@ 2022-11-29 08:25:47 
Real write
39481: g_cdatasize  0
39482: g_htsize     0
39431 writeJidacheader -1
job 1:  2249: allocx failed for 12.288(    0.00 B) of (   5.34 GB)   54.67 MB/sec

Same error even on very small files 👀

[hugmouse]

By compiling with -DNOJIT it works with small files. Im going to test my 5.34GB archive now and will update this comment later.

zpaqfranz v56.2c-NOJIT-L archiver,  (27 Nov 2022)
franz:DEBUG very verbose (-debug) 
21026: check_if_password of .test-m5.zpaq
Default sort                                                                                                                                                                                          
k3
000  first /Users/mysh/Documents/Screen.png
000  date  20.220.512.135.535
000  size  264.308
000  attr  8.496.245
000  data  8.101.526.029.400.539.119
000  writ  -1
000  ptrsi 0
000  dtv   0
Creating .test-m5.zpaq at offset 0 + 0
Adding 264.308 (258.11 KB) in 1 files (0 dirs), 8 threads 
20006: converting to localtime 2022-11-29 08:39:28
@ 2022-11-29 08:39:28 
Real write
39481: g_cdatasize  0
39482: g_htsize     0
39431 writeJidacheader -1
39864: out.tell   232.333
39865: header_end 104
Model2: XXHASH64 55CB919D3D1A82E9 /Users/mysh/Documents/Screen.png
Mode2: XXHASH64: <<55CB919D3D1A82E9>> CRC32 <<5A75D097>> /Users/mysh/Documents/Screen.png
1 +added, 0 -removed.                                                                                                                                                                                 
**** 40036 calculated **** 232.229 1
40045 writeJidacHeader 232.229 1

0 + (264.308 -> 264.308 -> 233.016) = 233.016 @ 102.71 KB/s

2.513 seconds (000:00:02) (all OK)
33040: call xcommand with a different errorcode (not 1, not 2) 0
33047: g_archive not null, setting g_exec_text to .test-m5.zpaq

It's working very-very slow, so Im using 10MB archive instead:

zpaqfranz v56.2c-NOJIT-L archiver,  (27 Nov 2022)
franz:DEBUG very verbose (-debug) 
21026: check_if_password of .test-m5.zpaq
.test-m5.zpaq: 
0 versions, 0 files, 0 fragments, 0 blocks, 0 bytes (0.00 B)
Default sort                                                                                                                                                                                          
k3
000  first /Users/mysh/Documents/10mb_archive.zip
000  date  20.221.129.090.500
000  size  10.315.204
000  attr  8.496.245
000  data  8.820.704.490.500.914.570
000  writ  -1
000  ptrsi 0
000  dtv   0
Updating .test-m5.zpaq at offset 0 + 0
Adding 10.315.204 (9.84 MB) in 1 files (0 dirs), 8 threads 
20006: converting to localtime 2022-11-29 09:05:20
@ 2022-11-29 09:05:20 
Real write
39481: g_cdatasize  0
39482: g_htsize     0
39431 writeJidacheader -1
39864: out.tell   10.144.262
39865: header_end 104
Model2: XXHASH64 6C8C71839D33EF37 /Users/mysh/Documents/10mb_archive.zip
Mode2: XXHASH64: <<6C8C71839D33EF37>> CRC32 <<3DD84D9C>> /Users/mysh/Documents/10mb_archive.zip
1 +added, 0 -removed.                                                                                                                                                                                 
**** 40036 calculated **** 10.144.158 1
40045 writeJidacHeader 10.144.158 1

0 + (10.315.204 -> 10.251.076 -> 10.148.680) = 10.148.680 @ 100.49 KB/s

100.244 seconds (000:01:40) (all OK)
33040: call xcommand with a different errorcode (not 1, not 2) 0
33047: g_archive not null, setting g_exec_text to .test-m5.zpaq

[fcorbelli]

On non Intel CPUs (better on anything not amd64 and x86 with SSE2) -DNOJIT is mandatory

The embedded jit translate zpaql code into intel machine code, just like a java jit compiler

On M1 or PPC or whatever else this cannot work, and a full software interpretatio (NOJIT) is needed

Turning back to high memory allocation it is possible to use parameters that need too much ram. The deployed release throw an error. The pre release should say how much ram is trying to allocate (then abort)

Please note that m5 is about a placebo level compression, not good for big file
I can explain how it works, or you can read the documention

For normal backups (aka: backup of home or virtual machine or whatever) the default (m 1) is good. For a bit more compression use m 2 (just as fast in extraction)
-m 3 for compress once extract almost never
-m 4 high compression very slow, OK for example for small mysqldump (1GB)
-m 5 top ram usage, very slow very high compression

Short version: you must see NOJIT and not JIT near version info (the very first line of the oputput)

With autotest you can check almost everything, except ram too small

[hugmouse]

Also I want to ask not really related to this issue question.

-m 5 top ram usage, very slow very high compression

With -m5 it eats up almost 100% of the size of the archive that I tried to compress into the RAM (5.74GB archive, 5.6GB usage in RAM), is this an expected behavior?

[fcorbelli]

Can eat up much more RAM :)

RAM usage is made by two parameters (in fact it is much more,say the two main)
One is fragment size
The other is 'kind of algo'
Plus algo specific parameters

You should really read the source (!)
Or the Mahoney's web page
http://mattmahoney.net/dc/zpaq.html

Short version: yes, it is normal to need huge amount of RAM in this case.
My home PC runs with 128GB, the office one with 768GB

[fcorbelli]

If you want please leave a review on
https://sourceforge.net/projects/zpaqfranz/
and put a star on github (if you haven't already).
Any comment or suggestion is welcome
Thanks for collaboration