compress files in virtual file system (#885)

[erossignon]

Fixes #885

[hipstersmoothie]

Would be cool to see some before and after data on this

[erossignon]

This PR enables compression of Virtual File System table and Zip/Brotli compression of each individual file content.
This allows a gain of up to 60% of the embedded file system.

node.exe size  = 71917203

Virtual File System size: 
No compression =  175703
        Δ GZip =  -100540 ( -57 %)
      Δ Brotli =  -109057 ( -62 %)

[jesec]

If possible, IMO we can try to make it possible to compress the entire binary, not only the virtual file system.

[robertsLando]

@erossignon Nice work here! BTW I have linked the pr that was asking for zip executables, what do you think about them? I know https://www.npmjs.com/package/caxa works in this way for example

[erossignon]

If possible, IMO we can try to make it possible to compress the entire binary, not only the virtual file system.

Yes, The executable that we use from pkg_fetch is far too big, It should be compiled with minimum ICU to significantly reduce its size. then we could had some options in pkg to reference or add the ICU as a external module ( optional)

I have no idea how to compress the entire executable further... if you have a idea, let me know.

[robertsLando]

@erossignon caxa does this, more info in the repo: https://github.com/leafac/caxa#readme

https://github.com/leafac/caxa#references-on-creating-self-extracting-archives-in-nodejs
https://github.com/vk-twiconnect/sfx-creator-service

[robertsLando]

LGTM.

cc @leerob @jesec

[erossignon]

@leerob @jesec , do you have any objection that would prevent merging ?

[leerob]

I'd like to wait until we cut a release of pkg that's fetching from pkg-fetch properly.

[erossignon]

I'd like to wait until we cut a release of pkg that's fetching from pkg-fetch properly.

OK, I'll keep rebasing and keep the code up to date until new version of pkg-fetch is fully integrated

[Hypfer]

I have no idea how to compress the entire executable further... if you have a idea, let me know.

UPX might be an option

See:
#50 (comment)

Relevant quotes

https://sourceforge.net/p/upx/discussion/6806/thread/79e2a6b8/

As it is now, the readPrelude function patched into the nodejs binary will open /proc/self/exe and try reading the overlay from the offset replaced into the binary at packaging time. However since the binary found at /proc/self/exe is the compressed version, all those offsets are of course just wrong and even if they weren't they would still read compressed data.

To get pkg to work with upx, one approach could be to update the readPrelude function so that it uses relative offsets from the end of the file and extend the pkg process so that it first bakes in the commandline options, then packs that binary and finally appends the rest of the code.

[erossignon]

@leerob can we incorporated to a 5.1 release ?

[leerob]

5.1 is out, but we can get this in 5.2! 😁

[alitas]

On an unrelated note, there seems to be a typo on a few lines in the commits, both in code and documentation, using Brolti instead of Brotli. It might cause some confusion among users.

[robertsLando]

@erossignon Could you fix the typos?

[erossignon]

let's go

[Hypfer]

When testing this + upx, my webserver spontaneously started to deliver compressed assets. It did work fine initially and only stopped working after a few minutes of runtime

This may or may not be a side-effect of the upx compression though, although considering that that never touched the assets and that the server still serves files via http just fine, I'd say that that's unlikely

[robertsLando]

@jesec @leerob Could we merge this?

[erossignon]

@Hypfer

It did work fine initially and only stopped working after a few minutes of runtime

This could mean that the size of the executable could change at runtime somehow? Strange! (your technique for relative offset, reads the executable size each time to find the start of the prelude by subtracting the payload size. )

This could also mean that one asset is read by chunks randomly (direct file access, set position then read ) rather than as a whole.

This behavior could be studied in a subsequent Pull Request, and should not prevent this PR to merged to main branch. At the end the compress options is not on by default.