Use a Rust wrapper over the Node.js CLI

[zkochan]

pnpm install is fast but not all the pnpm commands are fast. The pnpm run commands are slow #3409, volta-cli/volta#984. Although we can come up with some workarounds, a Node.js CLIs startup will be always slower than a native program's startup.

Furthermore, if we would like to implement a builtin version switcher #1383, pnpm will load itself twice, causing even more delays.

On the other hand, if we create a wrapper written in Rust, we could use Rust for some actions, and pass thru to Node.js for the rest. The Rust wrapper could:

1. load the correct version of pnpm (maybe we won't need Volta in that case Support for pnpm volta-cli/volta#737)
2. implement a much faster pnpm run command

Unfortunately, pnpm run has currently a lot of features, so this new CLI wrapper will have to be experimental for a long time. We can name it pn and keep it as a separate project for now.

PoC: https://github.com/pnpm/pn

[qballer]

FWIW, I think that an infra tool with such performance needs would benefit from a full re-write in rust.

[zkochan]

Maybe, but I do not have the experience to do that.

Also, I am not sure the performance gains would be that significant. The most time consuming operations are the filesystem operations (which are already executed with native code of Node.js) and HTTP requests. So we would get a boost in startup but unlikely a huge difference in overall time of installation.

There is a package manager written in Reason called esy. I don't know if it implements all the features of pnpm/npm/yarn but might be a good idea to check how it performs.

[qballer]

I'm unsure if the reason implementation would be faster as it compiles to javascript as well(Unless I'm missing something as I don't know much about reason).
perhaps it's worthwhile to benchmark Nodejs vs rust in that aspect.

While taking the discussion a bit off-topic replacing node-fetch with https://github.com/nodejs/undici in @pnpm/fetch might boost performance greatly

[zkochan]

No, Reason compiles to native code. It can compile to JavaScript as well, maybe that causes the confusion.

[qballer]

Good to know, I didn't know that it stands to "reason" that it will work better.

[vjpr]

The most time consuming operations are the filesystem operations (which are already executed with native code of Node.js)

Are there any benchmarks on this? The context-switching overhead between JS and native could still be significant.

[KSXGitHub]

Currently, pnpm's tab completion is slow enough that I find typing out the whole command more productive. Re-implementing tab completion in Rust would certainly boost its performance.

[zkochan]

Well, there are at least 3 different shells with autocompletion support. We cannot implement autocompletion for all three. What we do instead is we call pnpm to return the list of options to the shell

[KSXGitHub]

@safinsingh The shell does not parse package.json to extract script names and dependency names, otherwise, you'd end up with even slower and less reliable completions.

[Schniz]

Tab completion can be autogenerated using clap, although it is not dynamic (AFAIK) so there's no way of listing custom scripts etc. Works pretty fast and very intuitive when combined with structopt. This is how fnm drives its tab completion

[KSXGitHub]

It would be cool if clap provides a way to customize completion for certain subcommands and flags, but I don't find it in the documentation.

[vjpr]

I'd love speedy autocompletion for workspace package listings for use in filtering...and also other monorepo cli tools.

pnpm exec start --filter <tab>

Currently its takes my monorepo 1 sec to list all its packages (and to run an empty pnpm i). I analyzed this issue here: #3346

[Brooooooklyn]

Cross compile will be a problem since Github actions is lack of support for BSD systems

[Schniz]

Native addons are very easy with https://neon-bindings.com, but if it is not prebuilt, people will have to have Rust on their computer to compile it, which is similar to producing a native binary without Node

[KSXGitHub]

I was mistaken that native addons somehow cross-platform.

Anyway, cross-compiling isn't that difficult, there is a qemu action that would help.

[Brooooooklyn]

napi-rs has supported many projects contains prebuilt binary in production:

• https://github.com/napi-rs/node-rs
• swc
• https://github.com/Brooooooklyn/canvas
• https://github.com/Brooooooklyn/pinyin
• parcel
• prisma

All of these packages support Linux arm platforms, but non of them support BSD platforms. Support BSD is possible, but too complex for now.

[Brooooooklyn]

I'm building the BSD binary for swc, and toolchains seems to be better than 1 year ago, swc-project/swc#1758

[vjpr]

Actually you can't use WebAssembly because pn is highly IO-related while WebAssembly doesn't provide any IO interfaces.

NOTE: WASI provides IO for WASM. FS is possible. Sockets coming.

[vjpr]

I get a lot of value from the way pnpm is decomposed into many small packages, so I wouldn't want to lose this to a monolithic Rust dependency doing various things.

A good strategy could be to create optional native Rust bindings for some packages containing hot code paths or anything impacting cli startup performance.

I wonder though if some inter-package dependencies would not benefit from perf improvements if they are in separate packages. I wonder if you could have native bindings per package, and have them communicate natively, instead of marshalling objects through v8...

We should do some benchmarking on various things to figure out where the best wins could be.

I started benchmarking some cli stuff here last month - #3346.

It would be great if find-workspace-packages ran natively and was optimized. fast-glob is a great candidate for Rust bindings I think.

Also, on a large repo, pnpm m ls hangs the terminal and takes forever to finish. Things like this could be good candidates.

Should not that some perf improvements could also come from persistent file watchers, and daemons, in some cases. Although the current pnpm server didn't yield benefits to date.

[vjpr]

Leading the way: https://github.com/napi-rs/node-rs

[vjpr]

Related: #3906

[zkochan]

Someone is experimenting with writing a Node.js package manager in Rust
https://twitter.com/XtremeDevX/status/1399984973403336705
https://github.com/voltpkg/volt

[Cyberhan123]

Does anyone use go to do this？

[zkochan]

@Jarred-Sumner recommended to try Zig which might be easier to learn than Rust.

[IWANABETHATGUY]

Maybe it's not a good choice use a language which is not reach 1.0

[hyf0]

@zkochan Hi.

I have had the same ideas before and developed some tools to achieve that goal. It's very happy to share them with you.

There are some portability problems in rust's standard path module. You could use rusty_nodejs_path to deal with it.

I also think you could use js_workspace_tools and graph_executor to supports features about monorepo in pnpm run.

pnpm is a wonderful tool. I'll see what I could do rather than create a new one.

[zkochan]

An update regarding this. I have resumed work on the pnpm wrapper but I am reducing scope for now. It will implement the pnpm run and pnpm exec commands. If you have experience with Rust, you can help reviewing the changes: https://github.com/pnpm/pn

I am also planning to implement a tarball fetcher written in Rust. So Rust will download the tarball, extract its contents, calculate the checksums of each file and write them to the content-addressable store. I hope that this will make pnpm a couple times faster.

A full rewrite is unrealistic for now. We cannot switch all our efforts to it. The goal is to make pnpm fast enough for our users to stick with us instead of switching to less mature but faster alternatives written in Rust/Zig/etc.

[zkochan]

oh, by the way, let me know if you want to review my changes. I am a rust beginner, so help is needed. @KSXGitHub helped a lot the last couple of days! Thank you!

I can create a new team of Rust collaborators who will be included in code reviews.