WASI support?

[Restioson]

Mio has limited support for the wasm32-wasi target and I think it should therefore be possible to implement support here too, using WASI's poll_oneoff interface. This could be useful to allow for async TcpStreams and similar on wasm32-wasi rust binaries using async-io

[notgull]

There's this branch, but I'm not sure if @yoshuawuyts is actively working on it.

We could probably support WASI with the same caveats the mio has (i.e. it can't be woken since WASI is single threaded... although now that WASI threads are a thing, shouldn't there be a way around that?). The actual issue would show up down the line is in async-io, since we can't spawn the "async-io" thread yet. We would need to conditionally use a single threaded polling strategy there.

[yoshuawuyts]

Oh, so I'm currently working together with Sunfish to land the wasm32-wasi-preview2 target in the compiler. This will bring preliminary async support to WASI, including basic polling APIs. Once that lands I intend to file a PR to this crate to update it, so that all crates which rely on polling will Just Work on the new WASI target. This unfortunately doesn't include threading support, so we'll indeed need to figure out some other strategy for that (threading will likely happen closer to WASI preview 3)

We probably shouldn't do anything for wasm32-wasi-preview1 (nee wasm32-wasi), since that target is no longer actively developed. It's probably better for everyone if we just go with preview2 and the component model straight away. This should all happen in a matter of months, not much longer.

Hope that helps!

[notgull]

Sounds good, thanks for clarifying! It's probably better to wait for preview2 to drop as opposed to trying to hammer preview1 into a usable state.

[Restioson]

Thanks for the update! That makes a lot of sense :)

As far as the lack of WASI threads goes, it at least feels like it should be theoretically possible to have some kind of async_io::block_on even with just one thread once preview2 lands?

[notgull]

wasm32-wasi-preview2 has now landed on Rust Nightly. So now it might be worth looking into this further.

[taiki-e]

wasm32-wasi-preview2 has now landed on Rust Nightly

(It is likely to be reverted once, though: rust-lang/rust#120434)

[yoshuawuyts]

A new version of the wasi crate has been published containing poll operations. An example usage of these bindings can be found here. It is possible to author and test this code already via cargo-component, even if we don't yet have a Rustc target for WASI 0.2. We could for example put it behind a feature flag. Though the WASI 0.2 target is expected to be merged soon, and maybe by the time we've implemented this we're okay.

edit: I think this roughly looks right. I'm currently working on an HTTP client crate for a project, and so I need to have some way to convert from the low-level WASI Pollable interfaces to Rust's Future impls. I'm trying to structure this in a way that should map to smol's polling/smol::Async abstractions, at least somewhat.

[notgull]

@yoshuawuyts Great! The only thing missing is a way to wake up the Poller from another thread. Although as WASI has no threads I don't think it's possible to do that yet.

[yoshuawuyts]

Although as WASI has no threads I don't think it's possible to do that yet.

Yeah that's right - though more specifically: WASI 0.2 doesn't have them. WASI 0.1 has something experimentally, but that's not the right thing. For 0.2 we're working on that now as a canonical built-in to WASI (rather than as an extension to Wasm core directly). The progress on that is looking good, and I think we can expect it to land probably around Q3 of this year? Experimentally probably even sooner?

Anyway, yeah, for now that's not a thing we need to be able to do.

[yoshuawuyts]

Finished writing a wasi native async runtime, and wrote down step by step instructions for how this all works here: https://blog.yoshuawuyts.com/building-an-async-runtime-for-wasi/.

If someone wants to integrate this into polling, they should be able to follow most of that and it should all work!

[vkrivopalov]

I started looking into adding WASI support to polling, and I am facing a conundrum here.

WASIp1 supports polling through wasi::poll_oneoff using wasi::Subscriptions that can be created from raw file descriptors.
This is exactly how WASI support is implemented in mio.

WASIp1 is supported by wasi-rs v0.11, versions 0.12 and up have completely different API and support WASIp2. It no longer uses wasi::poll_oneoff and instead relies on the Pollable type as explained in @yoshuawuyts's awesome article linked above.

From the comments in this thread I conclude that it is preferred to implement support WASIp2 rather than WASIp1. This is possible to do even though there is some mismatch between the public Poller API and how wasi::io::poll::Pollable combines the source and the interest in certain types of events.
But as far as I can tell only WASI-specific types such as wasi::sockets::tcp::TcpSocket expose methods to obtain Pollable, it cannot be constructed for standard library types such as std::net::TcpStream or std::net::TcpListener.

This sort of defeats the purpose of using polling for WASI because polling provides an abstraction over specific platform and a portable interface.

What is best to be done then, should we try to implement support for WASIp1 which can be used for standard library types or integrate the Pollable-based polling from WASIp2 but limit its application to WASI-specific types? Please advise.

[notgull]

Yeah, that's a good point. There's not really a way to traditionally "register" a file descriptor into a Poller at this point. So, the polling model doesn't fit WASI.

@smol-rs/admins I'm planning on closing this issue as "not planned", since it's not likely that we'll be able to implement this.

[yoshuawuyts]

@dicej has been doing work to upstream WASI 0.2 support to mio. From the sound of it this might result in changes to wasi-libc to provide backwards compatibility with the POSIX model. I believe this could be used by polling as well.

If I understand the plan correctly, this same shim would be guaranteed to keep working for WASI 0.3 as well, which should give this some longevity.

[dicej]

Correct. We can lean on wasi-libc to make WASI look like just another POSIX OS as a good first default, then add WASI-specific improvements later as desired.