Should this repository's CLI be merged with wasm-tools?

[alexcrichton]

This came up as a question during the BA summit last week and we didn't get to talk about it properly (alas!) so I wanted to raise the issue here. @thomastaylor312 originally expressed interest in possibly merging with wasm-tools and @macovedj I believe was also interested (please cc more folks if y'all know of more too!)

Historically I believe I was personally of the stance "let's have the tools be separate" to have a clear line between "this thing needs the network" and "this thing doesn't". Given our proliferation of tools I'm at least personally reversing stance and thinking "well that's not obviously a good idea". In that sense, I'd like to open this up to discuss more about whether this is a good idea, why so, and if so how to do it. From a wasm-tools perspective the most difficult part I think is that there's a pretty diverse group of folks interested in it (or at least I sort of assume so, I'm doing my best to sort of personify possible personas interested in the tool moreso than channeling folks I actually know about).

In the abstract "monorepo good" so I'm at least not here initially going to try to enumerate why the two repos should merge. For me visibility of the tool, coordinated release schedules, and ease of learning for newcomes are huge benefits to earn. Instead here I'm going to focus on the possible downsides of a merge. I want to be clear though in that I don't want to try to create a laundry list of "these must all be fixed" but rather make sure that these downsides are considered. I wouldn't consider it necessary or required to solve every last one.

Broadly speaking though bringing networking functionality to a CLI tool is, in my opinion, no trivial thing to take on. That's a huge step up from what wasm-tools does today. All networking nowadays typically comes with TLS/authentication as well, which raises questions like which TLS library is used? How does that affect release binaries? How does that affect local developers. With that as sort of background, here's what I would imagine are possible downsides:

• Loss of portability. Right now wasm-tools doesn't have really any system dependencies except std::process sorta, so it's pretty easy to compile and run on most systems so long as Rust is supported. Pulling in a dependency on TLS tends to not be quite so portable. For example cross-compiles tend to be harder once TLS is in the mix.
• Bigger builds for developers. In a monorepo-style situation developers that want to "run all the tests" tend to have to build quite a lot. This means that anyone developing wasm-tools would have to figure out TLS dependencies, make sure they're all available, and satisfy this new build dependency.
• Bigger builds for end-users. The CLI tool itself will inevitably get larger with new subcomands. Not a huge issue but, for example, static linking OpenSSL for something you never actually use (e.g. if you just use wasm-tools print or validate) is a tougher sell.
• More build dependencies for distros/packagers/etc. I know of a Homebrew recipe for wasm-tools but I'm not sure if wasm-tools is packaged anywhere else. Bringing in TLS though inevitably brings about the problem of someone needing to set up the dependencies and make sure everything works well.
• Bumpy transition period. Currently wasm-tools and users don't deal with networking. Adding that in raises the risk of bugs that prevent upgrades or cause issues. While likely all solvable problems it's something to be aware of.

My initial preference for the current state of the world (no networking in wasm-tools) was me thinking of wasm-tools as a "binutils" of sort which don't do any networking, and then other tools are responsible for networking. A major counterpoint to this though is that tools like Cargo which have networking built-in can be a great success and benefit a lot from bundling everything together. That's what makes me start to lean more towards "maybe it's better to merge everything together".

---

Well in any case I mostly wanted to file an issue to get the conversation going. I'm curious what wasm-pkg-tools maintainers think about this and whether they think it's an obvious thing to do, if I'm way off the mark about TLS, or if it's a terrible idea and should be shelved for awhile before reconsidering (if even).

[diversable]

From the perspective of the user - and to aid in the ongoing adoption of wasi & the component model + its surrounding ecosystem - I wholeheartedly endorse the "single tool" approach.

Even within the Rust ecosystem (which seems to have the best wasi+component support), the proliferation of wasi- and component-related tooling can be hard to grok for persons trying to get into the ecosystem.

As @alexcrichton mentions above, tooling such as Cargo have been very succesful, and is loved by developers. Now, contrast Cargo tooling with the dogs-breakfast of tools one needs to set up to get a new Node.js/Typescript project going, or to start a new Python project, and you'll understand where I'm coming from - there is real value in having a single primary tool for users to interact with. Establishing a user-friendly CLI interface will, I believe, help new users & wasi + components adoption in general.

How that singular tool is architected under the hood is another matter entirely, however. Personally, I would support exploring a plug-in architecture (cf. Cargo plug-ins) for the wasm-tools CLI (to help keep project build times down for developers); wasm-pkg-tools - and further wasm/wasi & component tooling - could then be installed as plug-ins which would be installed by default for new users when they install wasm-tools, and still allow advanced users more options and flexibility for installing plug-ins/tooling individually or with certain options enabled/disabled.

The plug-in architecture would also allow individual tools (ie. plug-ins) to iterate & release on their own schedules and enable a more-rapid pace of tooling innovation.

Importantly, a plug-in style architecture would also still enable the usage of separate git/github repos for each tool individually during ongoing development: this would mean that tracking issues & other github project mgmt features would still be available for each tool/plug-in individually.

Even if the plug-in style system proposed above is not adopted, I still believe that considering the perspective of the tool users is still important in choosing a longer-term direction. Of course, taking into account the preferences & perspective of tooling authors, as well as the technical realities of the various projects, is critical, but I think it's important that the DX for end users not get relegated to the position of afterthought.

That's my two bits, anyhow.

Thank you for your time & consideration.

[thomastaylor312]

FWIW, I'm all on board with this. I know we have one more major refactor coming though to remove warg and refactor the configuration file, so I was thinking it might be better to get through that and then merge it together! Currently it is just a lack of time/juggling things