Parcel 2: Rust transformer

[devongovett]

Create the @parcel/transformer-rust package in packages/transforms/rust. This package is a Parcel 2 transform plugin for Rust to WASM. It should be based on the RustAsset from Parcel 1.

The @parcel/transformer-rust package is responsible for the following things:

• Installing the rust toolchain if needed
• Resolving Cargo.toml and running cargo if available.
• Otherwise, running rustc
• Collecting dependencies from the rust compiler and adding them as connected files so Parcel watches them
• Generating WASM

It is possible that the current approach in Parcel 1 is out of date. Additional research on the current Rust/WASM ecosystem is needed here. Some related issues:

• Importing JS from WASM - WASM loader does not expose importObject #647
• ESM integration - Better WebAssembly integration #1325
• HMR - WASM hot module reload #2688

There are also some plugins that have been working on some related stuff:

• https://github.com/mysterycommand/parcel-plugin-wasm-pack
• https://github.com/catsigma/parcel-plugin-wasm.rs
• https://github.com/rustwasm/rust-parcel-template

If someone from the community would like to own this feature in Parcel 2 core, that would be very helpful and appreciated! 😍

cc. @xtuc @linclark @littledan @alexcrichton - what's the current state of WASM + Rust + ESM? What should we be aiming for in Parcel 2?

[talentlessguy]

Would be cool to see Golang transformer as well. It has good support for WebAssembly. Also Go now has modules which makes development easier (and I guess making a transformer will be easier)

Related issues:

#2903

[devongovett]

@talentlessguy let's keep this issue about Rust. #2903 seems to cover Go well.

[alexcrichton]

Thanks for the cc @devongovett! The current state of Rust/WASM/ESM is that tooling like wasm-bindgen and wasm-pack have a "bundler" target which is intended to be consumed by Parcel/Webpack/etc. This "bundler" target assumes that the output wasm file is interpreted as an ES module and follows the wasm ESM integration spec.

From our perspective having Parcel act as a polyfill for the wasm ESM integration spec would cover all our use cases!

[devongovett]

Thanks for the info @alexcrichton. I'll take a look through that spec.

[kellytk]

FWIW #2973 is another issue related to Rust and wasm-pack.

[justinas]

Would it be possible to also make the Rust transformer respect the rust-toolchain file or the override precedence in general? Currently RustAsset downloads nightly unconditionally.

[mischnic]

The generated WASM binaries from Rust are often large and can be optimised.

I was hoping to call something like wasm-snip to optimise the build WASM outputs.

I took a quick look around to see if this can be done with Cargo configuration so Parcel doesn't need to be configured, but couldn't come up with a useful solution.

(@ChetanBhasin)

[hgzimmerman]

In a similar vein to concerns about the default toolchain, would it be possible to allow for the configuration of the optimization level?

By default, currently RustAsset will compile using --release for cargo projects or -O (which is equivalent to -C opt-level=2, which differs from what --release stipulates by default - opt-level=3) for plain rustc builds.

Having the ability to optimize for compile time or output size over runtime speed would be nice. You can currently set the opt-level in Cargo.toml, but it would be nice to be able to have some control over this from Parcel.

Might it be possible to only run in release mode (or rustc equivalent) for parcel build, and default to running in debug mode otherwise?

[buckle2000]

IMO, the default toolchain should be stable because it's more stable. Recently nightly regressed, making some projects impossible to build.

[Pauan]

I would just like to point out that in Rust, compilation happens at the crate level, not the file level.

That also means that all extern functions in Rust are exported from the crate, not individual files.

So it doesn't make any sense to import an individual .rs file, instead you should import the Cargo.toml file (which represents the entire crate).

This is something to keep in mind for the new Rust transformer.

[qwerty2501]

Here is shrinking wasm size when rust build wasm.
https://rustwasm.github.io/docs/book/game-of-life/code-size.html
The points are below.

• set opt-level "z"
• enable lto
• run wasm-opt -Oz

[Pauan]

We have created a Rollup plugin for Rust, it may be useful to look at its implementation to get ideas for the Parcel transformer.

[mysterycommand]

Hi there. I wrote parcel-plugin-wasm-pack and last week @D1plo1d brought to my attention the lack of support for building wasm modules in Parcel 2. I'd like to see if I can help. Reading the comments here it sounds like a good starting place (please correct me if I'm wrong) would be:

1. Detect if an asset is Cargo.toml (ignore lib.rs or main.rs per Parcel 2: Rust transformer #3365 (comment))
2. Respect the current toolchain, per Parcel 2: Rust transformer #3365 (comment)
3. Specify the opt-level per Parcel 2: Rust transformer #3365 (comment) and Parcel 2: Rust transformer #3365 (comment) … would Cargo's default profiles be sufficient fall backs here?
   • the comment makes it sound like more control than "Cargo.toml-level" is desired. What's the best way for me to get up to speed on Parcel 2's configuration options?
4. I'll def take a look at the Rollup plugin
5. … and the ESM integration spec

Iirc (it's been a while since I worked on this), the way I did this before was to create 3 assets per imported wasm asset. One "bundle-loader" to actually fetch the *.wasm file, an in-bundle module to be the "imports object" passed to the wasm module when it arrives (is compiled? instantiated?), and then the wasm module itself.

Does this still seem like a reasonable approach? Where could I look for any docs, or tests, or just like instructive source code for Parcel 2's way of doing things? Anything I should look out for, common gotchas when creating/porting asset types? Anything I missed?

[Pauan]

@mysterycommand Nice work!

To be clear, it should ignore all .rs files, it should only be possible to import a Cargo.toml file.

Some small things I noticed:

• This should be configurable (and it should probably default to false). It should also specify '--log-level', 'info',

• This should be removed, instead it should just let wasm-pack manage wasm-bindgen (this is important because wasm-pack must synchronize the version of wasm-bindgen).

• This is not needed, because wasm-pack already does that check.

• This should be removed, since wasm-pack manages the wasm-bindgen version.

• This and this don't seem needed, since you're using the bundler target.

In general wasm-pack handles almost everything, so the plugin should just be a thin wrapper around wasm-pack, right now it's duplicating a lot of things that wasm-pack does.

[mysterycommand]

@Pauan thanks for the code review! I'll try to keep all this in mind when I get a chance to work on this … hopefully this weekend. 🙏

[qwerty2501]

@Pauan

To be clear, it should ignore all .rs files, it should only be possible to import a Cargo.toml file.

I agree.
Almost Rustaceans won't build without Cargo.toml.

@mysterycommand Awesome!
However, my comment seems not correct. So I think you should ignore my comment.

Because I tried build following that procedure, but generated Wasm file size still remained larger than what was built with wasm-pack.

The wasm-pack looks good. It seems perform shrinking Wasm file size automatically when release build.
So you should still use wasm-pack as well as your parcel-plugin-wasm-pack.

I will keep looking for better way, but probably not find that way.
Because Wasm optimization is so complex...

[qwerty2501]

Fix my opinion. I was able to get the effects of lto and opt-level=z with wasm-pack.
I'm sorry to have changed over and over.
To shrinking the Wasm file, do one of the following:

The generally way.
Set lto and opt-level via Cargo.toml.

[profile.release]
opt-level = "z"
lto = true

Then run wasm-pack build.

$ wasm-pack build --release

Also, you can set lto and opt-level via environment variable(without setting Cargo.toml) as below.

$ RUSTFLAGS="-Clto=yes  -Copt-level=z" wasm-pack build --release

[qwerty2501]

I found additional method 'cargo rustc -- [rustc flags]'.
That can control optimization flags from out of cargo as below.

cargo rustc --target wasm32-unknown-unknown --release -- -C lto=yes -C opt-level=z

This method pros is to keep user environment as clean.
However, it seems not provided from wasm-pack.

[qwerty2501]

@mysterycommand
I forgot reply to your comment.

Specify the opt-level per #3365 (comment) and #3365 (comment) … would Cargo's default profiles be sufficient fall backs here?

It is probably not.
So you should explicitly set each flags when you want provide optimization feature from your plugin.

the comment makes it sound like more control than "Cargo.toml-level" is desired.

Yes. Need run wasm-opt and some optimization tools.
But if you using wasm-pack, only need control "Cargo.toml-level".
Because wasm-pack will automatically run each optimization tools when release build.

What's the best way for me to get up to speed on Parcel 2's configuration options?

I guess there are two situations for users.

1. If user want shrinking Wasm size like my comment, there are not in default profiles.
   It is nessesary to explicitly set opt-level=z and lto=true.

2. If user want more performance than shrinking Wasm size in runtime, set opt-level=3 (opt-level=3 is default in cargo release build. So you don't need set opt-level=3.).
   Then, It is nessesary to explicitly set lto=true when user want keep shrinking Wasm as small as possible and get more performance.

*Note: If set lto=true, the build time will be very long.

However, I do not recommend to control Cargo.toml from Parcel 2's configuration options.
Because the introduced two methods in my comment will pollute the user environment.

Alternatively, use 'cargo rustc -- [rustc flags]' method. But this method seems too hard because it cannot be used from wasm-pack.
If you use this method, you will have to run each optimization tool yourself.

Conclusion

In the first release plugin, I think it is better to just run wasm-pack build --release without control "Cargo.toml" from Parcel 2's configuration options.
That means taking the approach of parcel-plugin-wasm-pack.

[samvv]

I'm really looking forward to seeing this working. In fact, I can use it right now on the project I'm working on. I'll try to give it a go tonight given that I can't sleep anyways 😅 Will give an update as soon as I have something (or nothing).

[samvv]

Ok so here's my experience with trying to implement this feature: it's almost working (doing the install and all) but am I correct to say that Pacel 2 does not currently support WASM assets? Even if it did, Parcel would probably use its own loader (I found one in runtimes/js/src/loaders/browser/wasm-loader.js) while wasm-pack generates a loader that contains some magic. The two are incompatible, I think. How would we proceed? I'm not an expert in the Parcel API so I might have missed something obvious.

[mischnic]

Yeah, there is no plan yet for more flexible WASM (#647, #1325).

We'd need to get that sorted out first (together with loaders that aren't based solely on filetype)

[samvv]

@mischnic All right then, I guess it's too early to fully implement this. Will push the code to a repo later today and take a look at the issues you mentioned.

[mischnic]

(Compared to the Parcel 1 implementation, #4673 should be applied)

[cdbattags]

Any updates on this? I know we've been quiet for a while but I'm taking a stab at https://medium.com/@cwervo/parc3l-combining-three-js-rust-and-webassembly-c1e643ef7681 and I'd love to assist with this issue.

[DeMoorJasper]

@cdbattags feel free to implement this, there's some information about how to do it in this issue. If you have any particular questions you can open a draft PR or just ask questions in here

[mysterycommand]

Hi hello again, thank you for the nudge. I keep having to drop this for work, and when I pick it back up there's always this kind of lengthy "where was I" phase where I track down my various comments and conversations. It looks like the last time I dug in on this I had a sort of rocky idea of what to do, and I've got a small window now so I'm going to see if I can make some progress in the next couple days.

[mysterycommand]

Hey again, I've got some progress on this here, but I've run into an issue with my initial implementation that I'm not sure how to handle properly.

Looking for help/advice on how to inspect the state of the bundle before it throws the No transformers found error. I'd love to get this done this week, and maybe someone has some insight that'll save me a few hours of digging around. Thanks in advance!

[mysterycommand]

In the spirit of "make it work, make it good, make it fast" … this works: #4970 (I make no guarantees about it being good or fast at this time).

[zbraniecki]

What's the current state of this issue? Is it expected that I cannot get Parcel 2 to work with Rust? When trying to replicate a sum example with path pointing at Cargo.toml instead of lib.rs the bundling works, but at runtime it throws an error that libCargoToml_sum is not a function.

Is that expected and blocked on this bug?

[mysterycommand]

Hey, hi, sorry I have not had (and do not currently have) the time to keep working on this. If anyone wanted to take it over I'd be glad to see it happen! That error is new to me, but since my initial implementation here a lot has changed. Sorry I can't be of more help right at the moment.

[ZanderBrown]

Huh, must say after having a good experience with the original parcel + rust I was surprised to discover that parcel 2 doesn't seem to support it — especially when parcel itself seems to make use of rust these days.

Alas I lack the time/knowledge to get this done myself

[ultrasaurus]

@devongovett would be nice to add the "rust" label on this... took a bit of digging to find it and figure out that Parcel 2 doesn't support Rust (yet)

[Tails]

What's the current state of this issue? Is it expected that I cannot get Parcel 2 to work with Rust? When trying to replicate a sum example with path pointing at Cargo.toml instead of lib.rs the bundling works, but at runtime it throws an error that libCargoToml_sum is not a function.

Is that expected and blocked on this bug?

Parcel (2) imports the TOML as json/text after auto-installing a toml transformer, instead of including the WASM blob. Then, of course the TOML json does not have the functions you expect to be exported from the actual WASM file.

[Roba1993]

Any update on this? This is the only reason I can't use Parcel 2 at the moment. Also it's a shame to not have rust support on a project written in rust IMHO...

[chinoto]

@Roba1993 I gave up on waiting and moved to Vite. My Rust dependency isn't automatically compiled by the plugin I'm using, but it's better than the nothing that Parcel 2 provides.
This diff from Webpack to Vite may be useful in getting started: chinoto/wasm_game_of_life@990f8e7

[feynon]

Any updates on this?

[devongovett]

Going to close this. I don't have the bandwidth to implement and maintain this. It would be great as a third party plugin if someone wants to build it!