Some corner use cases

[zcbenz]

On Windows, extracting DLLs and executing them is a red alarm to anti-virus softwares, so when supporting native addons with VFS there should be a way to leave the DLLs on the real filesystem, like the "unpack" feature of ASAR.

Another usage of the "unpack" feature is to feed assets to APIs outside JS world. Most system APIs and third party C libraries only read files on the real filesystem, for example when creating a tray icon you have to pass the path of the icon on the real filesystem. You can of course extract the file before passing its path, but that would require modifications to existing code to support VFS.

---

On macOS executables are usually signed and the single executable implementation should be compatible with signing. Many single executable implementations are concatenating the packages to the tail of the executables, which would be broken after signing, but it can be fixed by including payload in str table.

---

While this repo is about single executable, it would be nice supporting reading from multiple bundled packages in one process, so it would be possible to distribute Node.js apps as single files. For example users can execute downloaded apps with something like node downloaded_app.nodeapp or by double clicking the package file, and app developers can do fork('/path/to/another/app.nodeapp').

---

There should be a way to manipulate the bundled package file itself, which is important for creating toolings. Electron does it with original-fs module.

---

When supporting VFS in Node's builtin modules, instead of implementing it in JS, I think it is better implementing in C and make the code reusable for other libuv users. So it would be easier for other runtimes like deno and bun to adopt the support.

[cspotcode]

...there should be a way to leave the DLLs on the real filesystem

Should this be included in a formal list of design requirements, for the VFS or for the SPA bootstrapper?

• built-in support for a manifest describing which files to extract
• SPA will automatically extract files when it starts as needed
  • can use a temp cache if already exists? not sure
• JS APIs allow the SPA to get the paths to these extracted assets
  • These paths will be different from the VFS paths; will not share a common base path
  • Can/should the VFS have virtual "symlinks" to the extracted paths on the real FS? For example, a symlink from /foo/spa-exe/__vfs__/extracted.dll to /tmp/where-ever/it/extracts/extracted.dll?

distribute Node.js apps as single files.

Are you describing something like #27? Or different?

[jviotti]

It is done implicit as part of their fs monkey-patching logic. @RaisinTen might have links to the relevant parts of the source code

[cspotcode]

This was initially sent as an email reply, which made it appear in the wrong place. I'm copying it here

What about APIs that accept a string that is a path that are not the FS
API? For example, worker_thread, spawn, fork, etc? Since we're talking
about things like dlls, where the whole point is that those paths cannot be
VFS paths because of native code / OS requirements.

[zcbenz]

How Electron does monkey patching can be found at https://github.com/electron/electron/blob/main/lib/asar/fs-wrapper.ts#L236.

• For APIs like fs.access and fs.exists, fake information would be returned based on the manifest of ASAR file.
• For fs.readFile, it is reimplemented to read from ASAR file with fs.read.
• For child_process.execXXX, the file would be extracted first and then its path would be passed.
• If the passed path is an "unpacked" file, then the path to the real file would be passed instead of doing the above preprocessings.

[jviotti]

@zcbenz

For child_process.fork, the file would be extracted first and then its path would be passed.

Have you witnessed this being problematic (like with the DLLs) too?

[zcbenz]

Sorry I just re-read the code and found fork does not extract files since the child Node.js process created by Electron can recognize files in ASAR archives. Only child_process.execXXX extracts files.

[jviotti]

All very good point @zcbenz ! Thanks a lot for chiming in. We would love to have you in our calls if you can find the time (see #19)

On Windows, extracting DLLs and executing them is a red alarm to anti-virus softwares, so when supporting native addons with VFS there should be a way to leave the DLLs on the real filesystem, like the "unpack" feature of ASAR.

Indeed! We have been extensibly discussing this on today's SEA call. Apart from AV concerns, this seems to be a good solution for making native add-ons that interact with I/O also play nicely with the VFS. See #29 for some related discussions.

On macOS executables are usually signed and the single executable implementation should be compatible with signing. Many single executable implementations are concatenating the packages to the tail of the executables, which would be broken after signing, but it can be fixed by vercel/pkg#1164.

Yeah, this is a top concern. To solve this problem, we have created https://github.com/postmanlabs/postject, a tool to inject arbitrary data to pre-compiled executables as proper Mach-O/ELF/PE sections.

While this repo is about single executable, it would be nice supporting reading from multiple bundled packages in one process, so it would be possible to distribute Node.js apps as single files. For example users can execute downloaded apps with something like node downloaded_app.nodeapp or by double clicking the package file, and app developers can do fork('/path/to/another/app.nodeapp').

This is a very interesting idea. We are talking about providing clear interfaces for extending require(), the fs module and others to avoid monkey-patching (like what Electron does with ASARs at the moment), which could in theory be used to implement something like this too? @RaisinTen @addaleax @arcanis

[arcanis]

While this repo is about single executable, it would be nice supporting reading from multiple bundled packages in one process, so it would be possible to distribute Node.js apps as single files. For example users can execute downloaded apps with something like node downloaded_app.nodeapp or by double clicking the package file, and app developers can do fork('/path/to/another/app.nodeapp').

Yep, I think it's in line with my suggestion at #27 (edit: Ah, I see someone mentioned it already 😆)

[cspotcode]

_EDIT This was meant to be a reply to a thread above, but Github's email integration doesn't do that. I've copied it manually: #30 (reply in thread) What about APIs that accept a string that is a path that are not the FS API? For example, worker_thread, spawn, fork, etc? Since we're talking about things like dlls, where the whole point is that those paths cannot be VFS paths because of native code / OS requirements.

…

On Fri, Sep 9, 2022, 11:41 AM Juan Cruz Viotti ***@***.***> wrote: It is done implicit as part of their fs monkey-patching logic. @RaisinTen <https://github.com/RaisinTen> might have links to the relevant parts of the source code — Reply to this email directly, view it on GitHub <#30 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAC35OFDXSS7GWL477MEQPTV5NLDJANCNFSM6AAAAAAQC3LR34> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[jviotti]

Yeah, these have to be taken care of. Projects like PKG and Electron monkey patch all of them. However, it is very tricky to maintain all these monkey-patches, which is the reason I was suggesting to experiment with providing a lower-level hooking capability to Node.js here: https://github.com/nodejs/single-executable/pull/31/files#diff-52c3ed092f4b8e8c123a7369c422f8846e288fe8abf3752b1d7eed4ec7808f83R33

[cspotcode]

I copied this to the place where I originally intended to post it: #30 (reply in thread)