cat, head, tail

[guest271314]

We can do

cat node example.js > node_example
tail --bytes=2003 node_example > example.js
head --bytes=86483184 node_example > node
chmod +x node example.js

We just need a means to create symbolic links to parts of a file and modify exec https://man7.org/linux/man-pages/man3/exec.3.html to accept symbolic link to parts of a single file rather than pathname or execute raw string.

[guest271314]

I used -e option to execute the plain text script concatenated to node executable

#!/bin/sh
js=`tail -c+86483184 node_example`
./node_example -e "$js"

We can keep track of offsets in a JSON array to support extraction of multiple files concatenated to the executable, without recompiling.

[jviotti]

Yep, this is in the right direction, and the base concept that we are thinking about implementing. The main problems with that are:

• Concatenating the JS code at the end of the binary works OK for Linux (PKG does pretty much what you are suggesting here), but it will not play well with code signatures on macOS and Windows. The reason for that is that by cating the JS program at the end of the file, the contents are by definition outside of the sections/segments of the binary format (like ELF in this case). Because code signatures operate on the binary format, any data section outside of the boundaries of the format is either not protected by the signature at all, or the OS will simply not run it (in the case of macOS)

Our starting point was not too different from what you are suggesting, but we ended up pivoting the approach to https://github.com/postmanlabs/postject because of those security concerns. You can think of Postject as a cat tool that operates within ELF, Mach-O and PE :)

• Your example concatenates a single JS file into the executable. While we started with a similar approach, we quickly realized that we couldn't concatenate full programs that include data files dynamically loaded with i.e. the fs module, programs including native add-ons, etc. We are introducing the idea of a virtual file system that serves as the generic "bundler" for a Node.js app for this specific reason.

I think @RaisinTen can expand more on the second point, as he faced these limitations and fixed them himself.

[guest271314]

Your example concatenates a single JS file into the executable. While we started with a similar approach, we quickly realized that we couldn't concatenate full programs that include data files dynamically loaded with i.e. the fs module, programs including native add-ons, etc. We are introducing the idea of a virtual file system that serves as the generic "bundler" for a Node.js app for this specific reason.

I think that approach will work for *nix. That is what I do here https://github.com/guest271314/WebCodecsOpusRecorder, serializing, desrializing and playing back WebCodecs opus packets to and from a single file. Borrowing from Native Messaging protocol

Writing

1. Create a Blob
2. Create JSON configuration, if necessary, else just keep offsets of arbitrary data in an array
3. Write Uint32Array with value set to length of proceeding JSON array that will be written next
4. Write offsets of data written to JSON array

Reading

1. Get Uint32Array length of proceeding JSON array of configuration or just offsets
2. Apply configuration to reader, or just iterate data per offsets

[jviotti]

A format that consists of sequence of data files prefixed by a JSON file with offset information is exactly what ASAR is (https://github.com/electron/asar).

In any case, I agree with the approach. My only suggestion would be that if we want to implement a data format like that, we rather use something that is a bit more of an industry standard like ASAR rather than re-inventing the whole thing.

[RaisinTen]

Simply appending stuff to the end of the binary makes strip throw an error at least on macOS:

$ strip node_example
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/strip: fatal error: the __LINKEDIT segment does not cover the end of the file (can't be processed) in: /some/arbitrary/path/node_example
$ echo $?
1

Haven't tested on other platforms yet but I believe this is something we should keep in mind. Folks might want the binaries to be compliant with what strip accepts.

And I wrote about how this doesn't work well with codesigning on macOS in nodejs/node#42334 (comment).

Fwiw, pkg does some additional bit fiddling after the naive append on macOS - vercel/pkg#1164.