Split "package multiple files together" and "Distribute a script w/ Node.js"

[arcanis]

I understand from the discussions I see here that the main use case currently talked about is shipping a completely standalone executable application. It's a reasonable one, but I have another: execute packaged scripts from an existing Node.js installation (and also to be to able to require the entry point from such packaged scripts, from existing Node.js processes) - similar to how Java knows how to run JAR files as if they were regular scripts, but without having to distribute the Java interpreter inside them, or how Python can natively import modules from zip files.

Fortunately, I don't think those two use cases are conflicting. Solving both would simply mean splitting the question statement:

• How can we distribute an application, resources and all, inside a Node.js binary?

Into two:

• How can we make Node.js understand the concept of packaged application (just like it understands json, wasm, etc)?
• How can we distribute a entry point inside a Node.js binary (whether this entry point is js, wasm, or packaged app)?

Then, by answering those two questions, both use cases will be addressed.

"Can't we just postpone this concept of packaged application to a follow-up proposal, once SEA are solved?"

While, in a sense, the original question statement is a subset of the split one (because it allows fewer things), I believe the split question is a better start for a design discussion. Indeed, if we decide to focus the design on the single initial question, then whatever format we pick for storing the SEA resources will either:

• constrain the packaged application design down the road (because we'll have to use the same format for consistency, even if it doesn't match all the requirements), or
• bring technical debt on arrival (because we'll have one format for SEA resources and a different one for packaged application, where a single one would have been enough)

[ljharb]

What’s the use case of distributing the equivalent of a jar that requires node, when you can do npx whatever or the equivalent?

[arcanis]

The same as for any other dependency from any Node.js project. To avoid having to reify the packages' node_modules directory on disk, by keeping them as independent entities.

While this isn't the approach npm/npx picked, this is the one that Yarn implements with its Plug'n'Play installs: each Node.js dependency is stored as a separate zip archive. When the CJS/ESM loader detects an access to a module stored within these dependencies, it loads the files directly from those places. This approach works very well for us, since each package is easy to cache, copying them is always atomic and can even take a single I/O operation for supporting systems, they can easily be stored within repositories, they are trivial to checksum for corruption checks, they can be signed, etc.

The drawback is that in order to make that work we have to apply a VFS layer on top of Node.js to make it understand how to access the zip archives files (for the same reason we are discussing one in this repository: some packages need to access their resources via fs operations). We do this (via ZipFS), but it's currently implemented in wasm-land (which makes the perfs worse than they could). If there was a way to use native capabilities instead, it'd be very beneficial.

[ovflowd]

The same as for any other dependency from any Node.js project. To avoid having to reify the packages' node_modules directory on disk, by keeping them as independent entities.

That's a very valid point, but not sure if it fits what SEA intends to be (My personal view of the scenario). Yet, I find the idea interesting, but not sure if there are enough people out there that are also interested on a "jar" approach for node (Haven't honestly given a thought about this)

[arcanis]

Also keep in mind that SEA binaries won't be portable, by design. Packaged application, however, have no problem being used on different platforms.

Taking the Yarn case again, we distribute our binary as a single bundled JS file, so that people can just download it and use it immediately. It kinda works, but it makes the initialization slower (the whole file has to be parsed, even if a subset is used) and the file size bigger (we can't store binary data efficiently).

It wouldn't be difficult for me to imagine that packaged application would have a variety of usage for cloud providers, where the Node.js version is defined in the workflow configuration rather than the build itself.

[RaisinTen]

Packaged application, however, have no problem being used on different platforms.

The packaged app might contain executables that might run on specific platforms though. Projects like GitHub Desktop uses / used to use dugite which contains platform-specific git executables AFAIK.

[arcanis]

In Yarn's case, we simply unpack those packages on disk; they make installs less efficient, but there are generally few enough such packages in any single project (swc, esbuild, fsevents, a few others) that the effect isn't too significant. The same is true for packages with postinstall scripts.

[ovflowd]

Also keep in mind that SEA binaries won't be portable, by design. Packaged application, however, have no problem being used on different platforms.

That is true, true. SEA applications will very probably (per design) be platform-specific.

Taking the Yarn case again, we distribute our binary as a single bundled JS file, so that people can just download it and use it immediately. It kinda works, but it makes the initialization slower (the whole file has to be parsed, even if a subset is used) and the file size bigger (we can't store binary data efficiently).

That is super cool!

It wouldn't be difficult for me to imagine that packaged application would have a variety of usage for cloud providers, where the Node.js version is defined in the workflow configuration rather than the build itself.

I see your point! Well, my 2 cents are for what I see right now:

• SEA might become part of the node-binary itself, allowing to bundle of SEA applications
• We could find a mid-term (extra functionality) that, instead of outputting a platform-specific binary file, could output the bundled package.
  • This could then be used as node my-bundled-package.extension (Not sure which extension we would use)
  • It would go through all the bundling steps but output a packed file

I'm just not sure how much it would differentiate from npm pack (I do understand both are different things). Would love to have some more concrete ideas here :D

[jviotti]

This is a great point @arcanis , and I think it will work very well for making incremental progress on the SEAs approach.

Referring my initial blog post on SEAs (https://github.com/nodejs/single-executable/blob/main/blog/2022-08-05-an-overview-of-the-current-state.md), what you are suggesting is essentially that we can implement only the Bootstrapper and the VFS ingredients, have something running and then do the Resource Injection.

If the Bootstrapper is made in a way that it is agnostic on WHERE it loads the bundle from, then this should work by default.

@RaisinTen @robertgzr This is a requirement for the Boostrapper component we should document: make it read the bundle from anywhere, including from within the binary or as a standalone file.

Can we make sure we address this idea on the first call agenda?

[jviotti]

BTW, I'm migrating this into a GitHub Discussion, as it is an interesting idea worth exploring in more depth. We can extract specific action items to GitHub Issues out of this.

[jviotti]

Thinking more about this idea, I have an example of this being a use case in production already: this is EXACTLY what Electron does. Electron ships a "pre-compiled" application using ASAR (https://github.com/electron/asar) as an archive format. However, the ASAR is not embedded on the binaries themselves, but distributed as part of the application resources directory. Still, Electron understands ASARs being a valid entry point and can run the entire thing from it.

[jviotti]

@arcanis Can you elaborate more (as much as you can! :P) on how/why you use this concept on Yarn? I'd love to understand more about it and try to create a new blog entry on this repo explaining this interesting use case, how Electron uses it, and how we could support it.

[arcanis]

I'll think about it - it'd be interesting to make it a general tech post on the Yarn website (that we can copy here) 🤔

[jviotti]

Yeah, that would be great. I'm up for co-authoring the post if you want.