wasm-npm-packager: what to write it in

[ashleygwilliams]

i'm currently tasked with writing a tool to take compiled WASM and package it up for publishing to npm, the requirements for which are documented in #34. in thinking about this workflow, i realize that the question of what to write this tool in is an interesting one.

current candidates:

• rust
• node
• shell? (let's not, for maintainability/contributors' sake?)

questions to think about:

• who will use this tool?
• what langs/toolchains do they already have?
• what would they be willing to use?
• should we write this tool in a few langs? not just one?
• is a goal to have this get built into current pkg managers, cargo? npm? or should it remain independent? what are the factors in making this decision?

would love to hear ya'lls thoughts on the matter.

[aturon]

To clarify: is the tool specific to packaging Rust projects, or does it work with arbitrary .wasm files?

[linclark]

In planning, @lukewagner and I have been thinking this could work with C++ modules as well. I believe the info in the relevant custom section could be completely decoupled from Rust.

Because of this, I think writing in JS makes the most sense.

[mgattozzi]

I really like the idea of using Rust for CLI tools but if it's for arbitrary wasm files then I think JS is probably the better choice. Mostly because if someone is publishing to npm they already have node. It would be easier to integrate it into npm possibly.

I would think decoupling from Rust is good. Yes I want Rust to succeed but a useful tool shouldn't be tied to any one lang in my opinion.

I guess I have a few questions:

• Like @aturon said is the is for any type of project that compiles to wasm?
• Are you thinking a platform independent tool that does one thing well and is then integrated into other build systems?

Knowing what and for whom we're trying to solve this problem for with the tool changes what we do. Was there some group or user base you had in mind as part of this @ashleygwilliams?

[aturon]

One thing we are hoping to push for in general is using Rust to write pure wasm tooling, which is already happening with the tools you can see at that issue. This is part of a bid to set up Rust as the predominant "systems language" for both writing in and around wasm. If we're successful on that front, we can probably expect a Rust toolchain to be available.

re: integration, I imagine there eventually being a Cargo subcommand that invokes this tool, but I don't think the tool itself needs to be coupled.

To me the high-order bits here are:

• What language seems technically best suited to this problem?
• What base of contributors do you want to draw from?

[ashleygwilliams]

so, given the workflow i currently have in my head, i believe that writing it in Rust makes the most sense. here's why?

• i am a Rust developer
• i write a Rust -> WASM project
• i want to publish the WASM to npm
• i dont have node on my machine
• i use tool X to publish to npm

i think the interesting idea is the ability to also allow C++ devs and other WASM backends to use the same tool. because Rust is distributed via a binary that seems ideal for multiple envs as we dont have to assume a runtime (which we would with node). i dont think we can assume that people compiling to WASM necessarily have a node env, and i dont think we should (though if we had to i dont think it would be the end of the world).

anyways. i'm leaning toward Rust, unless that really blocks C++ and other WASM backend devs, based on the ability to not depend on a runtime being present.

[aidanhs]

As a user, I wouldn't have a preference between rust, go or C++ because I can just download binaries. When I see arbitrary tools I want to use requiring ruby or node I tend to wince because I don't have that installed (shell and python would also work for me personally, but think of the windows users!).

Then again, if you're publishing to npm it follows that you're likely pulling from npm, implying that you do have npm installed...unless https://github.com/rust-lang-nursery/rust-wasm/issues/36 ends up with NPM integration in cargo.

So for me the answer hinges around how you consume the packages after publishing. If the only way is with a tool that requires node then...requiring node to publish isn't such a big deal.

[raphaelcohn]

OK, my two cents from running secure, productive environments:-

• When running a farm of continuous deployment or build servers, we want to keep to a minimum what's needed on each machine so it's easier to throw it away and rebuild it;
• We need to make sure that it's possible to a build a project reproducibly from scratch, based solely on what's in source control (even if that is a list of build binary dependencies and their versions)
• We, wherever possible, don't want a package manager (npm) running on internet or intranet facing web servers (node); each server has a purpose and should have as small an attack surface as possible
• We want tools that have a minimum of dependencies and are easy to install

So, that means such a tool should be either Rust (already has to be present to compile) or Shell (easy to install and always supported). Personally, I find shell hard to maintain and manage dependencies in (and I wrote shellfire and even cross-platform packagers in it), but it is the lowest common denominator for everything bar Windows - and even Windows these days with the new Linux subsystem or Cygwin, cough. However, shell sucks for anything which needs to work with binary data.

[sendilkumarn]

I also feel like writing in JS would have a broader coverage and wider audience.

[Pauan]

@sendilkumarn Do you mean for people making improvements to the tool itself? You're right, that is an advantage, since many programmers know JavaScript.

But if you're talking about people simply using the tool, I think Rust has the same coverage as Node/JavaScript: Rust is cross-platform, so a compiled Rust binary will work everywhere that Node/JavaScript works. And a Rust binary has the advantage of not requiring a runtime, unlike JavaScript which requires Node.

[sendilkumarn]

That is a valid point no runtime for rust.

But this compiled packages will always have wasm and (may or may not) js bundled in it.

IMO (just an opinion) having this library in js makes it easier to adapt when something in js package manager | world changes.

Do you think, rust projects will use these compiled binaries as dependencies? Is that possible?

[Pauan]

@sendilkumarn Do you think, rust projects will use these compiled binaries as dependencies? Is that possible?

They probably wouldn't include it as a direct dependency in Cargo, but they would be using the binary during the compilation process.

This is how the tool works:

1. The programmer writes a Rust crate. This Rust crate can use other Rust crates (through Cargo). Thus far, this is all normal Rust stuff, nothing unusual.

2. Each Rust crate might have some npm dependencies that it uses, and it might also have some JavaScript code that it uses.

3. When the programmer compiles their Rust crate, the Rust compiler does three things:

   1. It bundles up all the Rust code into a single .wasm file.

   2. Within that .wasm file it contains the npm dependencies for all the Rust crates (including transitive crates).

      Just to be clear, when I say "it puts the npm dependencies into the .wasm file", I don't mean the node_modules folder, I mean the dependencies section of the package.json file, so it's a very small amount of information.

   3. Within that .wasm file it contains the JavaScript code for all the Rust crates (including transitive crates).

   It contains all of that information within custom sections within the .wasm file.

4. Thus far the entire process I just described is handled entirely by Rust. Now is when the special wasm-npm-packager tool is used. The wasm-npm-packager tool will:

   1. Create a folder.

   2. Then it will extract the npm dependencies out of the .wasm file and put those npm dependencies into a package.json file inside of the folder.

   3. Then it will extract the JavaScript code out of the .wasm file and place them into the folder.

   4. Lastly it removes the custom sections from the .wasm file.

The end result is that you have a folder which contains a package.json, and also a mixture of .wasm and .js files. This folder can then be published to npm (using npm publish) and it can be used by other npm packages. Or you can use a JavaScript bundler like Webpack or Parcel to bundle everything together.

The only thing the tool does is extract npm dependencies and JavaScript code out of the .wasm file, nothing else. It's up to the programmer if they want to do anything else (such as publishing to npm, using Webpack/Parcel, etc.)

As you can see, the tool is very simple and it doesn't do much, therefore it doesn't really need to keep up with the changing JavaScript ecosystem.

Given the above, do you still think writing it in JavaScript would help? If so, would you mind explaining your viewpoint in more detail?

---

You might be wondering why it does it in two steps (combining the information into the .wasm, and then extracting the information out of the .wasm). There's two reasons:

1. This way the Rust compiler doesn't need to know anything about npm or JavaScript or Webpack or whatever: it just inserts npm dependencies and JS files into the .wasm file, nothing more.

2. The wasm-npm-packager tool is useful for any WebAssembly language, not just Rust. For example, a C++ program could use npm dependencies and JavaScript code, then compile to .wasm, and then use wasm-npm-packager to extract that information out of the .wasm

[sendilkumarn]

Thanks for the detailed explanation. It makes sense.

This raises another interesting question. Do we have any example of converting an entire lib into one wasm file. For example, converting a converter in binaryen / emscripten into wasm. How big will that single wasm be?

So many use case so lil time 😛

[Pauan]

@sendilkumarn Do we have any example of converting an entire lib into one wasm file.

There is a Rust WebAssembly book which contains some simple examples. There is work being done on expanding and improving the book. WebAssembly support for Rust is still in the early stages: there is a lot of discussion happening but it will take time for things to be implemented and stabilized.

If you want examples, right now the best way is to write them yourself. I have written a ~4,000 line Rust program and then used cargo-web and stdweb to compile it to WebAssembly (with emscripten).

It works great, though there are a lot of intricacies with the JavaScript interop. In general I've found that the .wasm file is roughly the same size as the equivalent x86_64 binary, though it's quite a bit larger than the equivalent JavaScript code (this is due to various complicated reasons).

[mgattozzi]

@Pauan if you have examples could we get that into the book? As part of #41 we want to add examples for people to reference. In particular an external section that isn't the book but a list of places people can look at code wise.

[sendilkumarn]

@Pauan Thanks for the information again. I will start exploring cargo-web and std-web.

well I did an attempt to convert the binaryen wat to wasm converter into wasm module. Since for wasm it is hard (impossible) to link (send) the file (as input parameter to the generated wasm). I have changed it into asm.js. But the output was a huge one. I feel the same will happen for wasm.

Is there any discussion splitting the wasm ?

[Pauan]

@sendilkumarn But the output was a huge one. I feel the same will happen for wasm.

In general the wasm output is dramatically smaller than asm.js, even for the exact same program.

But I think this is straying pretty far off-topic, so let's continue this conversation here.

Is there any discussion splitting the wasm ?

I don't see any discussion about splitting the .wasm, could you file a new issue about it?

[alexcrichton]

wasm-pack now exists, so I'm gonna close this