New Tier-3 target: wasm32-wasi-preview2

[alexcrichton]

Proposal

Note: This proposal is intended to augment a previous MCP for the same target string and update its definition. This MCP is written from the perspective of the compiler today where wasm32-wasi-preview2 is not implemented. This will reiterate some design points of the prior MCP (also because I did not find that MCP and forgot about it before I wrote this all up). More information about the relation to that MCP is available in a section at the end.

This is a proposal to add a new Tier 3 target to the Rust compiler named
wasm32-wasi-preview2. This new target will be unlike preexisting WebAssembly
targets in two major ways:

• The default binary output of this target would be a WebAssembly component,
  not a core wasm module.

• The standard library would not use functions from the wasi_snapshot_preview1
  namespace, commonly referred to as "WASI Preview1". Instead it would use
  functions from "[WASI Preview2]". These functions are defined at the component
  abstraction layer, not core wasm.

This target is not to be confused with wasm32-wasi, the currently existing
WASI target in the Rust compiler. That target is in the process of being renamed
to wasm32-wasi-preview1
to precisely make way for this target. The wasm32-wasi-preview1 target, and
current wasm32-wasi target, are effectively at a dead-end in the design space
of WASI itself, and the preview2 iteration represents the next phase of
development of WASI itself.

Note that this target is also not to be confused with the
wasm32-wasi-preview1-threads
target. The Component Model does not have support for threads at this time and
that's not expected to be supported until the time scale of years in the future.
Eventually there will likely be a wasm32-wasi-preview2-threads target but
that's not being proposed at this time.

Background: Component Model

To help better understand this MCP and what it means for the Rust compiler I
wanted to also provide some background on the motivations for what's going on
here. The first bit to clarify is the component model
itself. The Component Model is a WebAssembly
proposal currently at "Phase 1"
which is the introduction phase of features into WebAssembly. Despite this early
phase of the Component Model it's been in development for nearly 5 years now and
has had quite a bit of progress in the meantime. The wasm-bindgen project in
Rust, for example, is a major source of design for the component model
historically.

The Component Model is a specification that is layered on top of "core
WebAssembly" which is what "wasm" typically refers to today. The component model
defines a "component" which internally contains core wasm modules. One major
feature of a component is that describes imports and exports with rich types
such as string or tuple<u32, char, string>. This is in contrast to core
WebAssembly where all functions take integral/floating point types. One purpose
of the component model is to provide easier integration of WebAssembly
both on the web and off the web. For example wasm-bindgen also
supports richer types than i32 but it requires JS and the web, whereas
components are runnable off-the-web without supporting JS (and on the web too
with a polyfill).

The component model is reaching a milestone of stability at the end of this year
with the release of WASI Preview 2. The component model's binary format has been
stable for a number of months now and the ecosystem around components is
expected to provide longer-term stability for the current binary format
specification. This is not an ironclad guarantee of stability and it is
guaranteed to change in the future, but that timescale is in years.

Background: WASI

The second major feature of this proposed wasm32-wasi-preview2 target is that
it will use the next iteration of WASI for the implementation of the standard
library. Currently the wasm32-wasi target uses APIs from the
wasi_snapshot_preview1 module. This namespace is defined by the WASI
subgroup of the WebAssembly Community
Group, the main standardization body of the WebAssembly specification. The
wasi_snapshot_preview1 namespace is the original iteration of design for WASI
and was intended to be a "get the foot in the door" sort of phase of design
where it was the best that could be done at the time.

Since its introduction the wasi_snapshot_preview1 has served the WASI subgroup
well in providing feedback for future iteration and design. Namely there are
fundamental design aspects of the wasi_snapshot_preview1 module which required
changing to satisfy design requirements moving forward. One major design
requirement is that WASI is today now defined in terms of the component model,
not core WebAssembly. This means that wasi_snapshot_preview1 is incompatible
and has no "easy" translation into the component model since its whole purpose
was to be defined at the core wasm layer.

This redefinition in terms of the component model means that APIs from
wasi_snapshot_preview1 have largely all changed. All functionality is
preserved "in spirit" in the preview2 snapshot of WASI, however. For example
programs can still get environment variables, print to stdout, open files, etc.
The precise way that all these APIs works now is defined via two pieces:

1. All WASI APIs are defined with WIT, an IDL suitable for code generation in
   many languages and environments. WIT functions are all defined in terms of
   components so there is a clear mapping to the component model.

2. Component model functions can be "lowered" into core WebAssembly functions
   using the Canonical ABI of the component model. This provides a clear
   definition for what it means to communicate a string to a core wasm module
   for example.

This means that WASI is a collection of *.wit files
(this
is the current snapshot in Wasmtime for example). These WIT files describe
interfaces, functions, and types which can all be imported into WebAssembly core
wasm modules. From these WIT files a bindings generator such as
wit-bindgen is used to
generate Rust functions to interface with these WIT interfaces.

Background: Why Now?

The Component Model and WASI itself are in development. Neither provides an
ironclad guarantee of stability, and in fact both are guaranteed to change over
time instead. Given this nature, why would Rust want to add a
wasm32-wasi-preview2 target at this time?

In some sense this is a chicken-and-egg problem. For the further development of
the Component Model and WASI itself developer feedback is required. We would be
nowhere near where we are today without the Rust wasm32-wasi for example.
If stability is required before adding a target to the Rust
compiler, however, then there's no great way to break this cycle and bootstrap
somehow. This is more-or-less a way of saying that a new target in Rust is
desired as a means to develop these proposals and move them towards stability
since without a Rust target stability is unlikely to be achieved due to lack of
developer feedback.

Despite this though we (those working on the component model) didn't try to get
a Rust target from day 1 for the component model because it was understood that
the churn would be too high. The feeling though is that now is where the
component model and WASI have reached a level of stability that it's now
suitable to have a Rust target. While the Rust target isn't be guaranteed to
be present and unchanging until the end of time there are still "more stable"
foundations on which to build this target:

• The binary format of components has been stable for some time now and is
  expected to be supported at this version for quite some time going forward. No
  known major changes are left and any future iterations to the binary format
  will be required to support everything in the binary format today, meaning
  that any changes should in theory be cosmetic. This means that the base level
  of the output should be stable enough in theory for a Rust target.

• WASI itself is expected to enter a period of stability. Lots of churn has
  happened up to the release of preview2, but the upcoming version release of
  0.2.0 is expected to be a milestone in the development of WASI. APIs will not
  break or change throughout the development of 0.2.x which is expected to have
  a lifetime in the scale of years. Iteration, however, is expected. It is
  planned to have an 0.2.1 release with added functions/interfaces for example
  in the future (no concrete timeline at this time).

These combined together mean that the wasm32-wasi-preview2 target is expected
to have a good degree, albeit not ironclad, sense of stability moving forward. Major
breaking changes are expected to be evaluated as necessary against the entire
component ecosystem and a suitable proposal would be brought forward for both
Rust and other languages. An "easy way out" would be deprecation of
wasm32-wasi-preview2 and the introduction of wasm32-wasi-preview3 for
example. This is not guaranteed to happen, though, and it depends on how the
development of preview2 goes.

To reiterate, the trajectory of the wasm32-wasi-preview2 target is uncertain
at this time. While it's expected to have stability on the time scale of a year
or so further out than that is unknown. It highly depends on how the component
ecosystem shapes up, developer feedback from integrating this target with Rust,
etc. The introduction of the target, however, is seen as crucial to providing
this feedback.

What's in the Target: libstd

Ok with all that written down my hope is that the existence of this target is
sufficiently motivated for inclusion into the Rust compiler and standard
library. Here I'd like to discuss some more details about what this change might
concretely look like.

The first, and most desirable, change would be to the standard library itself.
Currently the standard library has a std::sys::wasi module which implements
Rust's platform-specific APIs in terms of WASI APIs. The standard library
additionally interacts with
wasi-libc to ensure that Rust and C
sources can be linked into the same WebAssembly binary with interoperability
going through the wasi-libc library. In this new target the std::sys::wasi
submodule will be renamed as std::sys::wasi_preview1 and then reimplemented as
std::sys::wasi_preview2. This could be configured with target_vendor = "preview2" or target_os = "wasi_preview2" for example (TBD).

The new sys submodule would not use the wasi
crate from crates.io as is done today with the
wasm32-wasi target because that crate uses wasi_snapshot_preview1. Instead
functions will be generated through
wit-bindgen. To have the
least impact on libstd's build process the generated files will be checked into
the source of the standard library. Hand-written documentation will indicate how
the files can be regenerated if need be.

Note, however, that not all APIs may be implemented in terms of the raw WASI
interfaces. The wasm32-wasi-preview2 target will still use wasi-libc and may
use some APIs from there. For example WASI preview2 has support for sockets but
they don't look like POSIX sockets. The wasi-libc library will implement a
POSIX-like API in terms of WASI preview2. This means that sockets in Rust can
either use wasi-libc's APIs or WASI preview2 APIs directly. Precisely how the Rust standard library chooses to implement primitives is not something this proposal wants to set in stone. Some primitives might go one way while some may go the other way. It's intended that the best means of implementation is iterated on over time internally in the standard library. A major factor in deciding this will be figuring out what the AsRaw* traits will look like for this target. WASI has no concept of a "file descriptor" but at the same time the representation of sockets is quite different than POSIX for example, meaning that there's not an obvious 1:1 mapping from standard library primitives to WASI primitives.

Overall the exposed abstraction layer is something that this proposal does not intend to specify at this time but wants to highlight will be a point of development for this new target. Integration with ecosystem crates doing async I/O, for example, is expected to help guide this design and inform what the best choices are here.

What's in the Target: rustc

The final piece I'd like to talk about in this MCP are the changes to rustc
required for this target. The easy part of these changes is the introduction of
a wasm32-wasi-preview2 target specification in rustc_target. This won't be
all that different than the other WebAssembly targets with one major exception.
Currently all WebAssembly targets that rustc supports emit core WebAssembly
modules but I've indicated that this new target would emit a WebAssembly
component instead. Here I'd like to explain how that is envisioned to work.

As background all emission of core WebAssembly modules is done through wasm-ld
today. The wasm-ld binary is provided by LLVM as a flavor of LLD, LLVM's
linker. The wasm-ld binary does not have support for components and it is not
planned to add support to emit components at this time. The development of the
component model to-date has worked with the wasm32-wasi target for example
which emits a core WebAssembly binary using wasi_snapshot_preview1. This core
wasm binary is then transformed into a component using two pieces:

• The wit-component
  project
  is primarily used to transform a core wasm module into a component. This uses
  type information embedded by wit-bindgen into wasm custom sections to
  generate the final component. The custom sections are stripped from the input
  module and the final component is created.

• The wasi_snapshot_preview1 imports are translated to preview2 through the
  use of an
  "adapter".
  This adapter exports symbols matching those in wasi_snapshot_preview1 and
  then internally translates to preview2. This adapter is itself a core wasm
  module and is present in the final output component.

Given all this background, the purpose of wasm32-wasi-preview2 is to bundle
all this together for users. The compiler will effectively run wit-component
for each compilation in addition to using an adapter if necessary. The current
vision for this is to decouple all of this from rustc itself by using a custom
linker binary distributed with the compiler.

Today the rust-lld binary is built by the Rust project's CI and distributed
for all major targets. This new linker, I'll call it wasm-component-ld for
now, would also be built by the Rust project's CI and distributed for all
platforms. This means that the wasm32-wasi-preview2 target would be composed
of both this linker in addition to the standard library itself. This separation
achieves a few goals:

1. Primarily the stability of the wasm-component-ld tool isn't tied to rustc
   itself. All the code will live in its own binary off to the side for
   exclusive use for this one target.
2. This additionally provides the flexibility to skip this process altogether.
   For example a linker of wasm-ld could be specified to emit a core wasm
   module instead of a component in a non-default fashion. This can be useful
   for some component building processes.
3. This eventually provides the opportunity for this project to be developed
   externally to rustc itself for use with other languages too.
4. Coupling the distribution of the linker and the standard library together
   enables customizing the adapter used for libstd's purposes. For example if
   the standard library nor wasi-libc use wasi_snapshot_preview1 then
   there's no need for an adapter. This is expected to take some time to develop
   though so for the early lifetime of the wasm32-wasi-preview2 target it's
   expected an adapter will be required. As the target evolves the linker can
   transparently evolve as well.

Despite the intention to eventually bundle this binary my intention would be,
for the initial implementation, to not bundle anything. This would mean that the
wasm32-wasi-preview2 target would, by default, not actually work for anyone
since it would invoke a nonexistent linker. This should help provide a bit of an
early speedbump to enable changes if necessary to this linker scheme. The
intention is that this linker would be developed externally to get to a "proof
of concept" stage and then if and when wasm32-wasi-preview2 approaches Tier 2
in Rust it would move to being distributed at that time.

What's in the target: Changes from Today

Adding a new target is a good opportunity to fix mistakes and/or issues with the wasm32-wasi target that this is in the long-term going to replace. While these issues can theoretically be fixed on the existing target it typically requires a good deal of effort to keep things working with smoother transition periods. Some examples of issues which can be fixed/changed with this new target are:

• All code will be compiled with an -fPIC equivalent by default. This enables much more easily producing a "shared everything" library in wasm for Emscripten-style linking against other libraries in the same component. This is already being done for a Rust extension to Python compiled to wasm but requires -Zbuild-std for now. The impact of this is expected to be minimal if at all measurable.
• If needed crate-type outputs could be adjusted. Currently the "bin" crate type generates a "command" with a main function which is generally agreed to be ok. The cdylib crate type, however, is a bit odder in wasm where currently it generates a wasm file without an entry point (sometimes referred to as a "reactor"). There's also a use case for *.so equivalents for the Emscripten-style dynamic linking above. How exactly this is mapped to Rust crate types is a bit unclear, but this new target provides an opportunity to adjust these as appropriate to get things working.

If others have grievances or things they'd like to adjust in WASI Rust targets this would be a great time to consider them as well (especially before Tier 2 is applied for)

Benefits - Future and Immediate

I realize that this is a nontrivially-sized proposal for the Rust project. Here
I'd like to explain some benefits which will result from this work to help
further motivate why this should be done.

Perhaps the largest reason is that the current existing wasm32-wasi (to
be renamed to wasm32-wasi-preview1) is at a "dead end" from a spec-design
perspective. It's expected that this target will still be widely used for quite
some time while components fully mature, but from a standards perspective no new
functions are being added. For example std::net::UdpSocket doesn't work on
WASI as there is no specification of APIs to use, nor will there ever be from
the WASI subgroup. By switching the design of the Rust target to the preview2
snapshot of WASI it moves the target onto a trajectory to follow the future
development of the WASI standard. For example UDP APIs are specified in the
preview2 snapshot of WASI, but the standard library cannot make use of them on
the wasm32-wasi target, only on this new wasm32-wasi-preview2 target which
targets components.

This new target additionally assists in getting out of the "rut" of adding new
features to the standard library for newer WASI APIs. Currently it's not clear
how to integrate the standard library with preview2 on the preexisting target.
Additionally there's quite a lot of work to do switching everything over to
preview2 from preview1. This target is designed to provide a ramp by which
incremental steps can be made toward the eventual goal of implementing
everything in terms of preview2. Initially the standard library will look
exactly the same for preview1 and preview2, but with a new target that uses
components as output it's possible to implement new APIs in terms of preview2
(e.g. UDP sockets). The high-level specification of the output (a component as
opposed to a core wasm module) provides a good deal of flexibility for the
standard library to evolve over time as opposed to being required to be perfect
and final from day 1.

Finally this is going to be a crucial step for the evolution and standardization
of the component model itself. This greatly lowers the barrier to entry to
writing Rust programs to generate components and provide feedback for
implementations. This will enable seeding preview2 support in the ecosystem as
well, for example one day tokio could work on preview2 in addition to
reqwest using APIs from preview2. None of that can easily happen until a Rust
target is added first.

Relation to the prior MCP for wasm32-wasi-preview2

The wasm32-wasi-preview2 target has had a prior MCP to add it as a Tier 2 target. I unfortunately forgot about that and additionally didn't search well enough to find it, so here I'd like to address the differences and why I think a new MCP is necessary.

This proposal has two major changes relative to the prior MCP, both of which build on each other:

1. The binary format is proposed here to be a component, not a core wasm module. The primary motivation for this is to best align with the Component Model itself and how WASI is specified. This additionally, however, enables a more incremental approach to standing up this target rather than requiring it to be "finished" from day one. For example this won't require transitioning away from using preview1 as an internal implementation detail via the usage of adapters mentioned above.
2. This change in the binary format is novel for rustc and additionally requires a new linker. Given this substantial change relative to the prior target it's expected that some iteration will be required to "get this right", meaning that this proposal starts at Tier 3 instead of Tier 2. This is hoped to be able to buy some runway in terms of breaking changes in the linker and figuring out and fixing bugs before everything is finalized before Tier 2.

Otherwise though the spirit of the prior FCP is preserved here, and it likely turns out that much of the words in this MCP aren't necessary to be restated as it was agreed upon for the first MCP.

Next Steps

If this proposal is accepted then I wanted to also outline a few steps of what would happen next with this target. The end goal would be to promote this target to Tier 2 to enable rustup target add wasm32-wasi-preview2 to be the primary end-user experience. Before this, however, the steps might look like:

• Add wasm32-wasi-preview2 to the compiler
• Start a wasm-component-ld project which proxies to wasm-ld (e.g. rust-lld) and then runs wit-component
  • TBD: CLI flag syntax for configuring wasm-component-ld vs wasm-ld
  • TBD: CLI flags passed from rustc to wasm-component-ld, e.g. does it pretend to "just" be wasm-ld?
• Determine how to disambiguate wasm32-wasi and wasm32-wasi-preview2 in Rust code (e.g. cfg(target_os) or cfg(target_vendor))
• Get the standard library compiling for wasm32-wasi-preview2, e.g. configuring cfg(target_os = "wasi") to additionally work with this new target. For now this means wasm32-wasi-preview2's intermediate core wasm binary would still reference wasi_snapshot_preview1
• Implement enough of wasm-component-ld with a bundled adapter to generate a component.
• Test that everything works on nightly with cargo build -Zbuild-std plus environment configuration (e.g. wasm-component-ld is in $PATH). At this point there's feature parity with wasm32-wasi plus a manual wit-component run.
• Implement full support for std::net build on WASI preview2
  • Option A: implement this directly in Rust using WASI APIs directly as-generated by wit-bindgen
  • Option B: implement a POSIX API in wasi-libc and use that in libstd
• Start integrating the wasm32-wasi-preview2 target into community crates as appropriate (e.g. libc)
• Flesh out all of the above into a more "final" state
• Apply for Tier 2 status which would entail (in addition to Rust's preexisting requirements)
  • Working components with -Zbuild-std for a variety of applications/embeddings.
  • Full support for std::net implemented in the standard library.
  • Full support for std::fs and std::env as they exist today.
  • Integration with ecosystem crates such as getrandom and libc, perhaps even mio as a stretch goal.
  • Stability story for wasm-component-ld and its arguments and inputs finalized.
  • Agreed-upon distribution plan for wasm-component-ld alongside rust-lld with the Rust compiler distribution.

At this point, which is expected to be ideally no more than a year from Tier 3 acceptance, the intent is to apply for Tier 2 status for distribution through rustup to get more feedback for the component model an WASI.

Mentors or Reviewers

I plan on personally being on-the-hook for any reviews to the capacity that I'm
qualified to perform such a review. I won't be able to review integration with
the rest of the standard library since that falls under the libs implementation
team, but I can review any WASI-specific parts, however.

Process

The main points of the Major Change Process are as follows:

• File an issue describing the proposal.
• A compiler team member or contributor who is knowledgeable in the area can second by writing @rustbot second.
  • Finding a "second" suffices for internal changes. If however, you are proposing a new public-facing feature, such as a -C flag, then full team check-off is required.
  • Compiler team members can initiate a check-off via @rfcbot fcp merge on either the MCP or the PR.
• Once an MCP is seconded, the Final Comment Period begins. If no objections are raised after 10 days, the MCP is considered approved.

You can read more about Major Change Proposals on forge.

Comments

This issue is not meant to be used for technical discussion. There is a Zulip
stream for that. Use this issue to leave procedural comments, such as
volunteering to review, indicating that you second the proposal (or third, etc),
or raising a concern that you would like to be addressed.

[rustbot]

This issue is not meant to be used for technical discussion. There is a Zulip stream for that. Use this issue to leave procedural comments, such as volunteering to review, indicating that you second the proposal (or third, etc), or raising a concern that you would like to be addressed.

cc @rust-lang/compiler @rust-lang/compiler-contributors

[yoshuawuyts]

If I'm not mistaken this might be a duplicate of the already-accepted MCP in #594?

[alexcrichton]

Oh dear I'm so sorry for missing that when searching and additionally forgetting about that!

After some dicussion with Yosh we agree that this proposal is different enough it's likely best to get agreement again given the change in direction. I've added a "Note" to the top of this proposal in addition to a new "Relation to the prior MCP" section.

[ranile]

There's one concern I have with this approach: do we just keep adding new targets as new WASI previews come out? I assume we would want to stabilize the targets so people can use them without nightly. What happens when preview3 come out? Will we add a -preview3 target?

I feel like a better solution is to update wasm32-wasi such that it works with preview 2. It can emit a core wasm module which can then be converted to a component. If need be, there can be a target/compiler flag that also emits the binaries compatible with component modal. The big argument against this is breaking all preview1 (current) users. How badly will we break them, if such a change were to be made? I think it's safe to assume that moving forward, the runtimes will all be abandoning preview1 so the need for it isn't much. We can perhaps keep a way to opt into preview1 output for a certain amount of time, maybe?

[Noratrieb]

Technical discussion like this should go to the Zulip thread mentioned above.

[ranile]

I raised the concern here because the issue said:

Use this issue to leave procedural comments, such as
[...] raising a concern that you would like to be addressed.

I'll repost it on Zulip if that's better

[davidtwco]

@rustbot second

This strikes me as reasonable and I'm satisfied that all concerns raised in this thread and in Zulip have been considered.

[apiraino]

@rustbot label -final-comment-period +major-change-accepted