Wasi-http: support inbound requests (proxy world) #7091

pchickey · 2023-09-26T17:49:46Z

This follows up #7056 with the remaining pieces used for the wasi-http proxy world.

We made a tweak to the wasi-http wit definition which was missed in the prev PR, but wasn't caught because the incoming request side was still stubbed out.

A new test suite wasi-http-proxy-tests has one trivial test for this new functionality.

A new cli subcommand wasmtime serve [--addr SOCKADDR] proxy_component.wasm runs components in the wasi-http proxy world.

We make some additional filesystem and cli imports to the proxy world in both the test runner and cli subdommand because the component adapter pulls them in unconditionally. We will work on fixing that in the adapter in the future.

… to run trivial test

…heck

prtest:full

Co-authored-by: Adam Foltzer <acfoltzer@fastly.com> Co-authored-by: Pat Hickey <phickey@fastly.com>

… own

alexcrichton

Looks great to me 👍

Small comments here and there, please feel free to defer whatever you like to either issues or future PRs.

alexcrichton · 2023-09-28T18:26:53Z

Cargo.toml

+http-body = "1.0.0-rc.2"
+http-body-util = "0.1.0-rc.2"


While you're at it mind putting = constraints on these as well? (works around issues with Cargo's handling of pre-releases)

alexcrichton · 2023-09-28T18:32:14Z

src/bin/wasmtime.rs

@@ -5,6 +5,8 @@

 use anyhow::Result;
 use clap::Parser;
+#[cfg(all(feature = "component-model", feature = "wasi-http"))]


Another possibility here, if you're interested, is to model this as subcommands-as-features. The wasm-tools CLI does this where there's a Cargo feature for each subcommand and they're all enabled by default. That way this could be enabled with cargo build --features serve and serve would implicitly enable the component model and wasi-http

I'll leave the decision up to y'all though, I don't feel strongly either way

I like that!

Actually, wasi-http already enables component-model, so we could make this only depend on wasi-http. However, I don't like that as much because it's not quite as declarative as the serve command feature.

alexcrichton · 2023-09-28T18:36:06Z

src/commands/serve.rs

+/// Runs a WebAssembly module
+#[derive(Parser)]
+#[structopt(name = "run")]
+pub struct ServeCommand {
+    #[clap(flatten)]
+    common: CommonOptions,
+
+    /// Socket address for the web server to bind to. Defaults to 0.0.0.0:8080.
+    #[clap(long = "addr", value_name = "SOCKADDR")]
+    addr: Option<std::net::SocketAddr>,
+
+    /// The WebAssembly component to run.
+    #[clap(value_name = "WASM", required = true)]
+    component: PathBuf,
+}


I think it'd be best if this had similar options to wasmtime run, for example --allow-precompiled, --dir, --env, and --profile. That being said I think it's also totally ok for this to be a follow-up PR too.

One thing that I think should also be done is to share the code that configures WASI, e.g. processing many of the -S options. For example in theory wasi-nn and such should all be possible to enable too. That would require some refactoring, however, so perhaps best to open an issue and/or file a follow-up PR for that.

Yep, agreed. I'll do that in a follow-up PR.

alexcrichton · 2023-09-28T18:36:46Z

src/commands/serve.rs

+                _ = tokio::signal::ctrl_c() => {
+                    Ok::<_, anyhow::Error>(())
+                }


Out of curiosity is it easy enough to have a graceful shutdown here where ctrl-c sets a flag "please exit ASAP"? Otherwise this looks like it cancel all pending work if it's happening during ctrl-c (in which case this probably isn't much better than avoiding installing a ctrl-c in the first place)

Yeah, I think that should be possible: we could share some state with the running thread that we check for each iteration of the accept loop.

I'm going to punt this to a follow-up PR, in the interest of landing this so that the resourcification work can proceed.

alexcrichton · 2023-09-28T18:38:46Z

src/commands/serve.rs

+                    eprintln!("error: {e:?}");
+                }
+
+                Ok::<_, anyhow::Error>(())


I think this can probably be dropped because the task doesn't return Err(_) anywhere

alexcrichton · 2023-09-28T18:39:48Z

src/commands/serve.rs

+            let (sender, receiver) = tokio::sync::oneshot::channel();
+
+            // TODO: need to track the join handle, but don't want to block the response on it
+            tokio::task::spawn(async move {
+                let host = Host::new()?;
+                let mut store = Store::new(&handler.engine, host);
+
+                let req = store.data_mut().new_incoming_request(
+                    req.map(|body| body.map_err(|e| anyhow::anyhow!(e)).boxed()),
+                )?;
+
+                let out = store.data_mut().new_response_outparam(sender)?;
+
+                let (proxy, _inst) = wasmtime_wasi_http::proxy::Proxy::instantiate_pre(
+                    &mut store,
+                    &handler.instance_pre,
+                )
+                .await?;
+
+                proxy
+                    .wasi_http_incoming_handler()
+                    .call_handle(store, req, out)
+                    .await?;
+
+                Ok::<_, anyhow::Error>(())
+            });
+
+            let resp = receiver.await.unwrap()?;
+            Ok(resp)


This can all be moved outside of the enclosing async move I think and the Future return could actually be just the receiver (or a boxed receiver, either way)

Great idea! I like that this makes it so much more clear that we spawn the task and return the future that waits on the oneshot channel, rather than returning a future that does both.

elliottt · 2023-09-28T21:17:35Z

Docker has an active incident that's causing status checks to spuriously fail. I'll hold off on putting this back in the queue until their incident has concluded.

) * Move the incoming_handler impl into http_impl * Remove the incoming handler -- we need to use it as a guest export * Start adding a test-programs test for the server side of wasi-http * Progress towards running a server test * Implement incoming-request-method * Validate outparam value * Initial incoming handler test * Implement more of the incoming api * Finish the incoming api implementations * Initial cut at `wasmtime serve` * fix warning * wasmtime-cli: invoke ServeCommand, and add enough stuff to the linker to run trivial test * fix warnings * fix warnings * argument parsing: allow --addr to specify sockaddr * rustfmt * sync wit definitions between wasmtime-wasi and wasmtime-wasi-http * cargo vet: add an import config and wildcard audit for wasmtime-wmemcheck * cargo vet: audit signal-hook-registry * Remove duplicate add_to_linker calls for preview2 interfaces prtest:full * Add a method to finish outgoing responses Co-authored-by: Adam Foltzer <acfoltzer@fastly.com> Co-authored-by: Pat Hickey <phickey@fastly.com> * Mark the result of the incoming_{request,response}_consume methods as own * Explicit versions for http-body and http-body-util * Explicit `serve` feature for the `wasmtime serve` command * Move the spawn outside of the future returned by `ProxyHandler::call` * Review feedback --------- Co-authored-by: Trevor Elliott <telliott@fastly.com> Co-authored-by: Adam Foltzer <acfoltzer@fastly.com>

* Do proper type checking for type handles. (#7065) Instead of relying purely on the assumption that type handles can be compared cheaply by pointer equality, fallback to a more expensive walk of the type tree that recursively compares types structurally. This allows different components to call into each other as long as their types are structurally equivalent. Signed-off-by: Ryan Levick <ryan.levick@fermyon.com> * Use wasi-streams in the wasi-http implementation (#7056) * Start refactoring wasi-http * Checkpoint * Initial implementation of response future handling * Lazily initialize response headers and body * make wasmtime-wasi-http compile * wasi-http wit: make a way to reject outgoing-request in outgoing-handler before waiting for the future to resolve * wasi: sync wit from wasi-http * outgoing handler impl: report errors to userland * test-programs: get wasi-http-components kicking over, delete modules and components-sync tests wasi-http-components-sync will come back once we get done with other stuff, but its superfulous for now. wasi-http-modules will not be returning. * Process headers * Add HostIncomingBody::new * Add trailers functions * Add TODO for body task outline * Rework incoming-response-consume to return a future-trailers value as well * Fix the wit * First cut at the worker loop * wasi-http: change how we represent bodies/trailers, and annotate own/borrow/child throughout * Update types_impl.rs for wit changes * Split body management into its own module * Checkpoint * more work on incoming body and future trailers * Fill out some more functions * Implement future-trailers-{subscribe,get} * Implement drop-future-trailers * Rework fields, but make the borrow checker mad * Fix borrow error * wasi-http-tests: fix build * test-runner: report errors with stdout/stderr properly * fix two trivial wasi-http tests the error type here changed from a types::Error to an outbound_handler::Error * Remove unnecessary drops * Convert a `bail!` to a `todo!` * Remove a TODO that documented the body worker structure * fill in a bunch more of OutputBody * Remove the custom FrameFut future in favor of using http_body_util * Move the outgoing body types to body.rs * Rework the handling of outgoing bodies * Fix the `outgoing request get` test * Avoid deadlocking the post tests * future_incoming_request_get shouldn't delete the resource * Fix the invalid_dnsname test * implement drop-future-incoming-response * Fix invalid_port and invalid_dnsname tests * Fix the post test * Passing a too large string to println! caused the large post test to fail * Format * Plumb through `between_bytes_timeout` * Downgrade hyper * Revert "Downgrade hyper" This reverts commit fa0750e0c42823cb785288fcf6c0507c5ae9fd64. * Restore old https connection setup * Sync the wasi and wasi-http http deps * Fix tests * Remove the module and component-sync tests, as they are currently not supported * Fix the reference to the large_post test in the components test * Fix wasi-http integration * sync implementation of wasi-http * Slightly more robust error checking * Ignore the wasi-http cli test prtest:full * Consistent ignore attributes between sync and async tests * Fix doc errors * code motion: introduce intermediate `HostIncomingBodyBuilder` rather than a tuple * explain design * Turn FieldMap into a type synonym * Tidy up some future state (#7073) Co-authored-by: Pat Hickey <phickey@fastly.com> * body HostInputStream: report runtime errors with StreamRuntimeError HostInputStream is designed wrong to need that in the first place. We will fix it in a follow-up as soon as resources land. --------- Co-authored-by: Pat Hickey <phickey@fastly.com> Co-authored-by: Alex Crichton <alex@alexcrichton.com> * Support resource maps in `component::bindgen!` (#7069) * Support resource maps in `component::bindgen!` This commit adds support to `component::bindgen!` to specify resource types using the `with` key of the macro. This can be used to configure the `T` of `Resource<T>` to use a preexisting type rather than unconditionally generating a new empty enum to have a fresh type. * Reenable tests * Include Wasm-defined globals and memories in core dumps (#6935) * Include Wasm-defined globals and memories in core dumps Co-authored-by: Nick Fitzgerald <fitzgen@gmail.com> * Narrow scope of unsafe block * Add missing skip-miri attribute for test that calls into Wasm --------- Co-authored-by: Nick Fitzgerald <fitzgen@gmail.com> * Remove manual impl of Error/Display (#7076) * Remove manual impl of Error/Display These conflict with auto-generated ones. * Fix tests * riscv64: Better AMode Matching (#7079) * riscv64: Delete `int_load_op` helper * riscv64: Delete `default_memflags` * riscv64: Rename `gen_amode` * riscv64: Better matching for amode * riscv64: Delete `emit_load` helper * riscv64: Rename some load variables * riscv64: Merge `iadd` into amode computations * riscv64: Add the remainder of Zca and Zcd instructions (#7080) * riscv64: Add `c.li` and `c.lui` * riscv64: Add CB type instructions `c.srli` / `c.srai` / `c.andi` * riscv64: Add `sp` relative load instructions * riscv64: Return Option from try_emit_compressed * riscv64: Implement stack based stores * riscv64: Add compressed stores * riscv64: Add compressed loads * Make `wasmtime::WasmCoreDump` serializable (#7078) This commit makes it so that the library type for core dumps is serializable into the standard binary format for core dumps. Additionally, this commit makes it so that we use the library type for generating core dumps in the CLI. We previously were using a one-off implementation of core dump generation that only had backtrace information and no instances, modules, globals, or memories included. The library type has all that information, so the core dumps produced by our CLI will both be more featureful and be generated by shared code paths going forward. Along the way, implementing all this required some new helper methods sprinkled throughout `wasmtime` and `wasmtime-runtime`: * `wasmtime::Instance::module`: get the module that a `wasmtime::Instance` is an instance of. This is public, since it seems generally useful. This involved adding a new return value from `ModuleRegistry::register_module` that is an identifier that can be used to recover a reference to the registered module. * `wasmtime::Instance::all_{globals,memories}`: get the full global/memory index space. I made these `pub(crate)` out of caution. I don't think we want to commit to exposing non-exported things in the public API, even if we internally need them for debugging-related features like core dumps. These also needed corresponding methods inside `wasmtime-runtime`. * `wasmtime::{Global,Memory}::hash_key`: this was needed to work around the fact that each time you call `{Global,Memory}::from_wasmtime`, it creates a new entry in the `StoreData` and so you can get duplicates. But we need to key some hash maps on globals and memories when constructing core dumps, so we can't treat the underlying `Stored<T>` as a hash key because it isn't stable across duplicate `StoreData` entries. So we have these new methods. They are only `pub(crate)`, are definitely implementation details, and aren't exposed in the public API. * `wasmtime::FrameInfo::module`: Each frame in a backtrace now keeps a handle to its associated module instead of just the name. This is publicly exposed because it seems generally useful. This means I also deprecated `wasmtime::FrameInfo::module_name` since you can now instead do `frame.module().name()` to get that exact same info. I updated callers inside the repo. * Optimize wasm calls ever-so-slightly (#7084) * Optimize wasm calls ever-so-slightly * Fix riscv64 * Remove stray comment * Shuffle around where `unsafe` lies * Add `Func::hash_key` and `Table::hash_key` methods (#7086) * Wasmtime: Move `Global` and `Table` to `externals` submodules Just mechanical code motion, not functional changes. * Wasmtime: Add `hash_key` methods for `Func` and `Table` * Rename `wasmtime::Func::caller_checked_func_ref` to `wasmtime::Func::vm_func_ref` We removed the "caller checked" part of the `VMFuncRef` type's name a while ago, so update this method to be in line with that. No functional changes, just mechanical renaming. * cargo fmt * Fix doc link * cranelift(x64): Make xmm{8..15} registers non-preferred (#7089) These registers require an additional byte to reference when encoded in certain AVX instruction formats (and maybe other situations as well?) so prefer xmm{0..7} when they are available and only fall back to xmm{8..15} when register pressure is higher. * Add docs and example for debugging with core dumps (#7087) * Add docs for debugging with core dumps * Fix reference to old style CLI flag * Add `no_run` to example that is only there to trap * cranelift(x64): Swap operands to save AVX instruction encoding size (#7093) For `XmmRmiRVex`-format instructions, when the opcode is commutative, the first operand is one of xmm{0..7}, and the second operand is one of xmm{8..15}, then we can swap the operands to save a byte on instruction encoding. * Update `ittapi` crate (#7092) This update gets rid of a panic that occurs when using `--profile=vtune` from the command line. * Bump wasm-tools crates (#7094) * Bump wasm-tools crates Two major changes/reasons for this update: * Primarily pulling in support for semicolons-in-WIT files. Semicolons are not currently required, though, so I'll follow-up later with actual semicolons. * The syntax for parsing `(if ...)` was fixed in `wast`. Previously it did not require `(then ...)` but this is required by the spec. New spec tests require this as well. This breaks existing text format tests which don't use `(then ...)` inside of an `(if ...)`. Most tests were updated by hand but `embenchen_*` tests were updated by running through the old parser to produce non-s-expression using code. * Fix an example `*.wat` * Minor preview2 impl cleanups (#7097) * delete preview2::pipe::pipe this is not used anywhere, not documented, and trivial for the user to build on their own if they need it. i suspect they probably only want one end of their pipe wrapped with AsyncReadStream or AsyncWriteStream. * fix cfg_attr to ignore forking tests on qemu * delete "async fd stdin" test, which is just the worker thread one anyway * Add MPK-protected stripes to the pooling allocator (#7072) * Add memory protection keys (MPK) In order to use MPK on an x86_64 Linux system, we need access to the underlying `pkey_*` system calls (`sys`), control of the x86 PKRU register (`pkru`), and a way of determining if MPK is even supported (`is_supported`). These various parts are wrapped in a `ProtectionKey` abstraction along with a `ProtectionMask` that can be used `allow` the CPU to access protected regions. * Integrate MPK into the pooling allocator This change adds "stripes" to the pooling allocator's `MemoryPool`. Now, when requesting a slot in which to instantiate, the user (i.e., `InstanceAllocationRequest`) will be transparently assigned to one of the stripes, each of which is associated with a protection key. The user can also request a specific protection key to use, which will override the original "find me a slot logic". This has implications for how instances get allocated: once a store is assigned a protection key, it will only allocate requests with that key, limiting how many slots it has access to. E.g., if 15 keys are active, the store can only ever access 1/15th of the slots. This change also includes a tri-bool configuration field, `memory_protection_keys`, which is disabled by default for the time being. * Address review comments This is a rollup of 43 commits addressing review comments of various kinds: bug fixes, refactorings, documentation improvements, etc. It also ensures that CI runs all checks. A big thanks to @fitzgen and @alexcrichton for the review! prtest:full Co-authored-by: Nick Fitzgerald <fitzgen@gmail.com> Co-authored-by: Alex Crichton <alex@alexcrichton.com> --------- Co-authored-by: Nick Fitzgerald <fitzgen@gmail.com> Co-authored-by: Alex Crichton <alex@alexcrichton.com> * x64: Fix false dependencies in int-to-float conversions (#7098) * x64: Fix false dependencies in int-to-float conversions This commit is a result of the investigation on #7085. The int-to-float conversion instructions used right now on the x64 backend will implicitly source the upper bits of the result from a different register. This implicitly creates a dependency on further consumers using the conversion result on whatever previously defined the upper bits, even though they aren't used. This false dependency is the primary reason for the slowdown witnessed in #7085. The fix chosen in this commit is to model the int-to-float instructions with a new shape of instruction instead of the previous `GprToXmm{,Vex}`. This previous shape was modeled as single-input and single-output, but this does not reflect the actual nature of the `cvtsi2s{s,d}` instructions. Instead these now use `CvtIntToFloat{,Vex}` which have two source operands and one destination operand, modeling how the upper bits of a different register are used. In lowerings using this instruction the upper bits to preserver are always sourced from a zero'd out register to force breaking dependencies between instructions. Closes #7085 * Remove now dead code * Remove outdated test Golden test output covers this test case anyway nowadays * Review comments * Fix emit tests * Update wit-{parser,component} deps (#7101) Pull in bytecodealliance/wasm-tools#1221 * wasmtime: Misc optimizations (#7102) * Mark some frequently used methods as `inline` * Seed the preview2 context with the thread_rng, instead of using a syscall * Remove stray `component_impl.rs` file (#7103) This has no longer been used since #7056 and I believe it's just an accident that it wasn't removed as part of that PR. * ci: Upgrade QEMU to `8.1.1` (#7096) * ci: Upgrade QEMU to `8.1.1` This adds support for RISC-V's Zcb extension that includes some extra compressed instructions. It also removes the current cpuinfo patch, that has been released in 8.1 * wasmtime: Don't assert the exact faulting address for wasm traps * Wasi-http: support inbound requests (proxy world) (#7091) * Move the incoming_handler impl into http_impl * Remove the incoming handler -- we need to use it as a guest export * Start adding a test-programs test for the server side of wasi-http * Progress towards running a server test * Implement incoming-request-method * Validate outparam value * Initial incoming handler test * Implement more of the incoming api * Finish the incoming api implementations * Initial cut at `wasmtime serve` * fix warning * wasmtime-cli: invoke ServeCommand, and add enough stuff to the linker to run trivial test * fix warnings * fix warnings * argument parsing: allow --addr to specify sockaddr * rustfmt * sync wit definitions between wasmtime-wasi and wasmtime-wasi-http * cargo vet: add an import config and wildcard audit for wasmtime-wmemcheck * cargo vet: audit signal-hook-registry * Remove duplicate add_to_linker calls for preview2 interfaces prtest:full * Add a method to finish outgoing responses Co-authored-by: Adam Foltzer <acfoltzer@fastly.com> Co-authored-by: Pat Hickey <phickey@fastly.com> * Mark the result of the incoming_{request,response}_consume methods as own * Explicit versions for http-body and http-body-util * Explicit `serve` feature for the `wasmtime serve` command * Move the spawn outside of the future returned by `ProxyHandler::call` * Review feedback --------- Co-authored-by: Trevor Elliott <telliott@fastly.com> Co-authored-by: Adam Foltzer <acfoltzer@fastly.com> * c-api: Correct wasmtime_module_image_range type (#7110) * Make C API usable from Rust (#6765) * Remove usage of `is-terminal` and `atty` crates (#7104) * Remove usage of `is-terminal` and `atty` crates This functionality is now folded into the standard library itself. * Fix syntax * Fix a unix/windows cfg * winch(x64): Call indirect (#7100) * winch(x64): Call indirect This change adds support for the `call_indirect` instruction to Winch. Libcalls are a pre-requisite for supporting `call_indirect` in order to lazily initialy funcrefs. This change adds support for libcalls to Winch by introducing a `BuiltinFunctions` struct similar to Cranelift's `BuiltinFunctionSignatures` struct. In general, libcalls are handled like any other function call, with the only difference that given that not all the information to fulfill the function call might be known up-front, control is given to the caller for finalizing the call. The introduction of function references also involves dealing with pointer-sized loads and stores, so this change also adds the required functionality to `FuncEnv` and `MacroAssembler` to be pointer aware, making it straight forward to derive an `OperandSize` or `WasmType` from the target's pointer size. Finally, given the complexity of the call_indirect instrunction, this change bundles an improvement to the register allocator, allowing it to track the allocatable vs non-allocatable registers, this is done to avoid any mistakes when allocating/de-allocating registers that are not alloctable. -- prtest:full * Address review comments * Fix typos * Better documentation for `new_unchecked` * Introduce `max` for `BitSet` * Make allocatable property `u64` * winch(calls): Overhaul `FnCall` This commit simplifies `FnCall`'s interface making its usage more uniform throughout the compiler. In summary, this change: * Avoids side effects in the `FnCall::new` constructor, and also makes it the only constructor. * Exposes `FnCall::save_live_registers` and `FnCall::calculate_call_stack_space` to calculate the stack space consumed by the call and so that the caller can decide which one to use at callsites depending on their use-case. * tests: Fix regset tests * Start to port Wasmtime to the new wasi-io API with resources. (#7029) * Rename `Host*` things to avoid name conflicts with bindings. * Update to the latest resource-enabled wit files. * Adapting the code to the new bindings. * Update wasi-http to the resource-enabled wit deps. * Start adapting the wasi-http code to the new bindings. * Make `get_directories` always return new owned handles. * Simplify the `poll_one` implementation. * Update the wasi-preview1-component-adapter. FIXME: temporarily disable wasi-http tests. Add logging to the cli world, since stderr is now a reseource that can only be claimed once. * Work around a bug hit by poll-list, fix a bug in poll-one. * Comment out `test_fd_readwrite_invalid_fd`, which panics now. * Fix a few FIXMEs. * Use `.as_ref().trapping_unwrap()` instead of `TrappingUnwrapRef`. * Use `drop_in_place`. * Remove `State::with_mut`. * Remove the `RefCell` around the `State`. * Update to wit-bindgen 0.12. * Update wasi-http to use resources for poll and I/O. This required making incoming-body and outgoing-body resourrces too, to work with `push_input_stream_child` and `push_output_stream_child`. * Re-enable disabled tests, remove logging from the worlds. * Remove the `poll_list` workarounds that are no longer needed. * Remove logging from the adapter. That said, there is no replacement yet, so add a FIXME comment. * Reenable a test that now passes. * Remove `.descriptors_mut` and use `with_descriptors_mut` instead. Replace `.descriptors()` and `.descriptors_mut()` with functions that take closures, which limits their scope, to prevent them from invalid aliasing. * Implement dynamic borrow checking for descriptors. * Add a cargo-vet audit for wasmtime-wmemcheck. * Update cargo vet for wit-bindgen 0.12. * Cut down on duplicate sync/async resource types (#1) * Allow calling `get-directories` more than once (#2) For now `Clone` the directories into new descriptor slots as needed. * Start to lift restriction of stdio only once (#3) * Start to lift restriction of stdio only once This commit adds new `{Stdin,Stdout}Stream` traits which take over the job of the stdio streams in `WasiCtxBuilder` and `WasiCtx`. These traits bake in the ability to create a stream at any time to satisfy the API of `wasi:cli`. The TTY functionality is folded into them as while I was at it. The implementation for stdin is relatively trivial since the stdin implementation already handles multiple streams reading it. Built-in impls of the `StdinStream` trait are also provided for helper types in `preview2::pipe` which resulted in the implementation of `MemoryInputPipe` being updated to support `Clone` where all clones read the same original data. * Get tests building * Un-ignore now-passing test * Remove unneeded argument from `WasiCtxBuilder::build` * Fix tests * Remove some workarounds Stdio functions can now be called multiple times. * If `poll_oneoff` fails part-way through, clean up properly. Fix the `Drop` implementation for pollables to only drop the pollables that have been successfully added to the list. This fixes the poll_oneoff_files failure and removes a FIXME. --------- Co-authored-by: Alex Crichton <alex@alexcrichton.com> * Cranelift: return programmatic error rather than panic when temporaries run out. (#7114) * Cranelift: return programmatic error rather than panic when temporaries run out. Cranelift currently has a limit of `2^21` vregs per function body in VCode. This is a consequence of (performance-motivated) bitpacking of `Operand`s into `u32`s in regalloc2. As a result of this, it is possible to produce a function body that will fail to compile by running out of vreg temporaries during lowering. Currently, this results in a panic. Ideally, we would propagate the error upward and return it programmatically. This PR does that, with a "deferred error" mechanism. A cleaner solution would be to properly thread the `Result` types through all layers of lowering. However, that would require supporting `Result`s in ISLE, and that is a deeper language-design and `islec`-hacking question that I think we can tackle later if we continue to see motivating cases. The deferral works by returning a valid but bogus `ValueReg`s to the lowering rule, but storing the error and checking for it in the toplevel lowering loop. (Note that we have to return a bogus `v0` rather than `VReg::invalid()`, because the latter causes the `ValueRegs` to think there is no register provided.) This PR also includes a test at the Wasmtime level. Note that it takes ~22s to run on my (relatively fast) laptop, because it has to run until it runs out of VRegs in a debug build of the compiler. We could remove the test if we feel we're otherwise confident. Thanks to Venkkatesh Sekar for reporting this issue! The integration test uses one of the example inputs from the report. * Review feedback. * Handle alloc after a deferred error occurs. * Add uncommitted fix for previous. * Remove some usage of UnsafeCell in the adapter (#7117) * Fix compile warnings in preview1 adapter This isn't part of the "main build" so it's not inheriting the "deny warnings" flag passed in CI, so fixup some warnings which have leaked through over time. * Remove some usage of `UnsafeCell` in the adapter Move to abstractions such as `OnceCell` from the standard library as well as `RefCell` to avoid some unsafety. This shouldn't have any practical ramifications on the adapter, mostly just trying to help over-time maintenance. * Deny test-programs warnings in CI * Remove `Table*Ext` traits for preview2 (#7118) * wasi: Add typed helpers to `Table` This will help cut down on the `Table*Ext` traits while retaining type safety. * Remove `TableNetworkExt` trait * Remove `TableTcpSocketExt` trait * Remove the `TableReaddirExt` trait * Remove `TableFsExt` trait This involed a fair bit of refactoring within the preview2-to-preview1 adapter to handle the new ownership of resources, but nothing major. * Remove `TableStreamExt` trait Additionally simplify some stream methods while I'm here. * Remove `TablePollableExt` trait * Fix tests * Fix some more tests * Use typed accessors for terminal-{input,output} * Remove dead code in `Table` * Temporarily disable wasi-nn on CI (#7126) This appears to be failing, so temporarily disable it while the issues are worked out. * c-api: Expose host memory creation (#7115) * c-api: Expose host memory creation Signed-off-by: Tyler Rockwood <rockwood@redpanda.com> * c-api: Review feedback Signed-off-by: Tyler Rockwood <rockwood@redpanda.com> * c-api: Fix invalid reference in docs Signed-off-by: Tyler Rockwood <rockwood@redpanda.com> * c-api: Fix doxygen documentation Signed-off-by: Tyler Rockwood <rockwood@redpanda.com> --------- Signed-off-by: Tyler Rockwood <rockwood@redpanda.com> * riscv64: Refactor and enable optimizing sign extensions (#7121) * riscv64: Remove some stray whitespace * riscv64: Refactor int-to-float emission Use ISLE matching to generate different instructions rather than having a method deeper in the backend select which instruction to use. Additionally remove the usage of the `normalize_fcvt_to_int` helper. * riscv64: Refactor and simplify `lower_bmask` Take a `Value` as input to simplify 128-bit handling and additionally simplify the rules a bit with other helpers. * riscv64: Refactor `normalize_cmp_value` and `truthy_to_reg` Remove the normalization completely by folding it into callers and refactor `truthy_to_reg` to perform sign extension internally in addition to handling 128-vs-64 * riscv64: Remove 128-bit handling from extension helper This is only needed in a small handful of spots which can pretty easily handle the sign extension on their own, and this further simplifies sign extension to purely dealing with a single register. * riscv64: Remove destination type in extension This is always `$I64`, so no need to pass it around. * riscv64: Remove `ext_if_need` helper Fold 128-bit handling into callers as necessary * riscv64: Refactor `sext` to take a `Value` This commit refactors the `sext` helper to take a single `Value` as an argument rather than an `XReg Type` combo. This will enable, in future commits, to pattern-match the structure of the input and skip the sign extension if possible. For example an `icmp` result is already sign-extended. This involved moving some `lower_*` declarations directly into `lower.isle`. Additionally a special-case of lowering `brif` was removed which caused regressions in generated code due to new `and` instructions being generated to sign-extend the result of an `icmp`. Once `zext` is also migrated to this pattern this regression should be fixed, so this should just be a temporary state of affairs. * riscv64: Fold `lower_umulhi` helper into lower.isle * riscv64: Fold `lower_uadd_overflow` into lower.isle * riscv64: Fold `lower_popcnt` into `lower.isle` This slightly pessmizes codegen when the `zbb` extension isn't available due to a sign extension for smaller-than-64-bit types, but this is possible to fix in the future if necessary and helps simplify this in the meantime. Additionally this lifts the special cases for `has_zbb` directly into the lowering rule. * riscv64: Refactor clz and cls This commit refactors the lowering rules for these two CLIF instructions to move the sext/zext operations to the top rather than having them buried in lowering rules. This additionally moves them to `lower.isle` instead of `inst.isle`. This did result in some minor duplication but overall feels a bit better due to them being less coupled now. * riscv64: Move `gen_bswap` to `lower.isle` Additionally shuffle rules around to remove the dependence on `zext` because when this zero-extension happens it's known that many of the extension-related optimizations are not applicable. This does pessimize code slightly but it's hopefully not too bad given the current size of the bswap lowering. * riscv64: Change `zext` to taking a `Value` This is similar to a prior commit for `sext` and will enable future optimizations to skip the zext entirely if the structure of the value allows. * riscv64: Fold `extend` helper into `zext` and `sext` This commit folds the existing rules of the `extend` helper into the corresponding functions of `zext` and `sext`. They are the only callers of `extend` and this keeps the sign extension-related logic truly in one final resting place now at this point. A new matcher `val_already_extended` is additionally added to add more cases in a follow-up commit to enable skipping sign-extension operations based on the structure of the `Value` input. * riscv64: Optimize some sign extensions As the culmination of all prior commits, this commit adds a few cases where a sign extension is not necessary in the backend and can be skipped. For example extending a sign-extended value is not required because the backend always extends to the full register width. Additionally extending the result of an `icmp` is not necessary since that always produces a 0 or 1 value at the full register width. * Fix compatibility with Rust 1.70.0 (#7128) This commit fixes a compatibility issue with Rust 1.70.0 on Windows targets. Rust 1.71.0 stabilized `AsSocket for Arc<T>` which is used here implicitly, so to compile successfully on 1.70.0, our current MSRV, a slight code change is required. Closes #7127 * riscv64: Add `Zcb` extension instructions (#7123) * riscv64: Add `c.mul` * riscv64: Add `c.not` * riscv64: Add zbb and zba dependent compressed instructions * riscv64: Add `Zcb` loads and stores * riscv64: Restrict immediate encoding for halfword compressed stores and loads * riscv64: Reverse imm bits for bytewise compressed loads and stores * riscv64: Optimize `uadd_overflow_trap` lowering (#7132) This commit removes the usage of `gen_icmp` in `uadd_overflow_trap`. The comparison previously done with an explicit comparison is now bundled directly into the conditional branch to go to the trap itself. * riscv64: Add another pattern for loading constants (#7131) Currently any 32-bit constant can be materialized without a load from a constant pool on RISCV-64 but once constants start getting larger than this they're always loaded from the constant pool. This commit adds another special case for loading constants which appears to match what LLVM does which is to consider materializing a smaller constant and than shifting it left. This is done by chopping off all trailing zeros from an immediate and then testing if the immediate can be materialized as a 32-bit constant. This means that the current constant loading sequence can optionally be followed by a trailing `slli` instruction to shift the zeros back into the constant. This namely means that loading `i64::MIN` (1 << 63) no longer falls back to the constant pool. * Refactor how `subscribe` works in WASI (#7130) This commit refactors how the `Pollable` resource is created in WASI. Previously this was created manually via two variants of a `Pollable` enum but this was somewhat error prone for a few reasons: * The more common representation, `TableEntry`, had boilerplate associated with it that had dynamic downcasts and such. This is now all hidden behind a more typed API. * Callers had to remember to use `push_child_resource` which is easy to forget. * The previous signature of the readiness check returned a `Result<()>` which made it accidentally easy for errors to be propagated into traps rather than being returned as a "last operation failed" error. This commit replaces the previous API with a single `subscribe` function which takes a `Resource<T>` and returns a `Resource<Pollable>`. The `T` type must implement a new trait, `Subscribe`, which indicates the asynchronous readiness check. This readiness check additionally returns `()` so it's no longer possible to accidentally return errors (or trap). This namely required refactoring the `HostOutputStream` trait and implementations. The trait itself now has a `check_write` method corresponding to to the same WASI function. The old `write_ready` function is now a convenience helper combo between `ready` and `check_write`. Concrete implementations are refactored to store errors discovered during `ready` to get communicated through `check-write` later on. * Fix a regression with `wasmtime serve`, and enable the serve feature in tests (#7137) * riscv64: Refactor implementation of `{u,s}{div,rem}` (#7136) * riscv64: Refactor implementation of `{u,s}{div,rem}` This commit's goal is to remove the usage of `gen_icmp` with division-related instructions in the riscv64 backend. I've opted for a slightly more verbose approach than what was present prior to make sign/zero extensions more precise and additionally enable some immediate-related optimizations. Divison/remainder by an immediate will no longer emit checks to see if a trap needs to be emitted. Instead only the raw `div`/`rem` instructions are emitted. Additionally a few minor sign extensions are now avoided such as the dividend in 32-bit remainders/division because only the low 32-bits are inspected. Finally, while I was here, I went ahead an added `has_m` guards to all these lowering rules. The M extension is always required with riscv64 right now and won't work if it's turned off, but I figure it's not too bad to add in terms of completeness for now. As to the meat of the commit, the check for trapping value as part of division now happens without `gen_icmp` but instead `gen_trapif` which enables avoiding materializing the comparison's value into a register. * Fix tests * Async support in the C API (#7106) * c-api: Add a feature for async Signed-off-by: Tyler Rockwood <rockwood@redpanda.com> * c-api: Add support for async config Signed-off-by: Tyler Rockwood <rockwood@redpanda.com> * c-api: Add support for calling async functions Signed-off-by: Tyler Rockwood <rockwood@redpanda.com> * c-api: Add ability to yield execution of Wasm in a store Signed-off-by: Tyler Rockwood <rockwood@redpanda.com> * c-api: Introduce wasmtime_linker_instantiate_async Signed-off-by: Tyler Rockwood <rockwood@redpanda.com> * c-api: Support defining async host functions Signed-off-by: Tyler Rockwood <rockwood@redpanda.com> * gitignore: ignore cmake cache for examples Signed-off-by: Tyler Rockwood <rockwood@redpanda.com> * examples: Add example of async API in C Signed-off-by: Tyler Rockwood <rockwood@redpanda.com> * c-api: Consolidate async functionality into a single place Put all the async stuff in it's own header and own rust source file Also remove the wasmtime_async_continuation_new function, users can just allocate it directly. Signed-off-by: Tyler Rockwood <rockwood@redpanda.com> * c-api: Make async function safe Signed-off-by: Tyler Rockwood <rockwood@redpanda.com> * c-api: Remove wasmtime_call_future_get_results Signed-off-by: Tyler Rockwood <rockwood@redpanda.com> * c-api: Simplify CHostCallFuture Move the result translation and hostcall_val_storage usage into an async function Signed-off-by: Tyler Rockwood <rockwood@redpanda.com> * c-api: Simplify C continuation implementation Remove the caller, which means that we don't need another struct for the future implementation. Signed-off-by: Tyler Rockwood <rockwood@redpanda.com> * c-api: Improve async.h documentation Signed-off-by: Tyler Rockwood <rockwood@redpanda.com> * c-api: Cleanup from previous changes Signed-off-by: Tyler Rockwood <rockwood@redpanda.com> * examples: Fix example Signed-off-by: Tyler Rockwood <rockwood@redpanda.com> * c-api: Simplify continuation callback This gives more duality with calling an async function and also means that the implementation can pretty much mirror the sync version. Signed-off-by: Tyler Rockwood <rockwood@redpanda.com> * c-api: Fix async.h documentation Signed-off-by: Tyler Rockwood <rockwood@redpanda.com> * c-api: Fix documentation for async.h Signed-off-by: Tyler Rockwood <rockwood@redpanda.com> * c-api: Review feedback Signed-off-by: Tyler Rockwood <rockwood@redpanda.com> * examples: Downgrade async.cpp example to C++11 Signed-off-by: Tyler Rockwood <rockwood@redpanda.com> * c-api: initialize continuation with a panic callback Signed-off-by: Tyler Rockwood <rockwood@redpanda.com> * prtest:full Signed-off-by: Tyler Rockwood <rockwood@redpanda.com> --------- Signed-off-by: Tyler Rockwood <rockwood@redpanda.com> * feat(preview1): implement `poll_oneoff` (#7099) Signed-off-by: Roman Volosatovs <rvolosatovs@riseup.net> * Handle `lower_branch` consistently amongst backends (#7133) * Handle `lower_branch` consistently amongst backends This commit is a refactoring to consistently implement `lower_branch` among Cranelift's backends. Previously each backend had its own means of extracting labels and shuffling along information, and now there's prelude methods for all backends to access and use. This changes a few display impls but the actual meat of what's emitted shouldn't change amongst the backends. * Fix compile * wasi-sockets: Implement initial listen backlog (#7034) * Allow backlog size to be set before initial listen * Move set_listen_backlog_size into example_body * Format code * Let cap_net_ext handle the default value. * retrigger checks * Got lost while pulling in changes from main. * Fix a flaky preview2 test (#7138) Found in CI for #7099 this commit updates the durations used in the `resolves_immediately` and `never_resolves` tests. For immediately resolving it should be ok to wait for a generous amount of time since the timeout is only a protection against tests hanging. For `never_resolves` it should be ok to wait a short amount of time and any failure there will show up as a spurious failure. * c-api: Support InstancePre (#7140) * c-api: Support InstancePre Signed-off-by: Tyler Rockwood <rockwood@redpanda.com> * c-api: Consolidate all the instance_pre functionality Signed-off-by: Tyler Rockwood <rockwood@redpanda.com> * c-api: Add async instantiate support to pre instances Signed-off-by: Tyler Rockwood <rockwood@redpanda.com> * c-api: Add star to comment for doxygen prtest:full Signed-off-by: Tyler Rockwood <rockwood@redpanda.com> --------- Signed-off-by: Tyler Rockwood <rockwood@redpanda.com> * riscv64: Refactor and simplify some branches/fcmp (#7142) * riscv64: Add codegen tests for fcmp and fcmp branches * riscv64: Refactor and simplify some branches/fcmp This commit is aimed at simplifying the layers necessary to generate a branch for a `brif` statement. Previously this involved a `lower_cond_br` helper which bottomed out in emitting a `CondBr` instruction. The intention here was to cut all that out and emit a `CondBr` directly. Along the way I've additionally taken the liberty of simplifying `fcmp` as well. This moves the "prefer ordered compares" logic into `emit_fcmp` so it can benefit the `fcmp` instruction as well. This additionally trimmed some abstractions around branches which shouldn't be necessary any longer. * Resourcify the wasi-http interface (#7135) * Fix race condition in `AsyncWriteStream` (#7144) * Fix race condition in `AsyncWriteStream` This commit fixes a synchronization issue with `AsyncWriteStream` where the writer task would get wedged and not make any further progress. The underlying reason is that `notify_waiters` was used which did not buffer its notification, meaning that if a waiter wasn't actually waiting then it would miss the notification and block forever on the next call to `notified`. By using `notify_one` instead this will buffer up a single notification for the other end for when it gets to waiting for work. Additionally this removes creation of the `Notified` future ahead-of-time as that's no longer necessary. * Fix http bodies too * riscv64: Update `replicated_{u,}imm5` to match `vconst` (#7141) This switches the `extractor` to a `pure partial` constructor to be used in `if-let` instead of the extraction position, which while not quite as ergonomic does enable matching more constant values in more locations. * Wasi input-stream: use same errors as output-stream (#7090) * streams.wit: delete stream-status, rename write-error to stream-error, transform all input-stream methods * preview2: use StreamError throughout input-stream * preview2: passes cargo test * preview1: fixes for input-stream stream-error. * wasmtime-wasi-http: fixes for HostInputStream trait changes * component adapter: fixes for input-stream changes * test programs: fixes for input-stream * component adapter: handle StreamError::Closed in fd_read * sync wit definitions to wasi-http * fix!! * preview1: handle eof and intr properly prtest:full * Fix preview1 stdin reading * Touch up stream documentation --------- Co-authored-by: Alex Crichton <alex@alexcrichton.com> * Implement the `wasi:sockets/ip-name-lookup` interface (#7109) * Implement the `wasi:sockets/ip-name-lookup` interface This commit is an initial implementation of the new `ip-name-lookup` interface from the `wasi-sockets` proposal. The intention is to get a sketch of what an implementation would look like for the preview2 release later this year. Notable features of this implementation are: * Name lookups are disabled by default and must be explicitly enabled in `WasiCtx`. This seems like a reasonable default for now while the full set of configuration around this is settled. * I've added new "typed" methods to `preview2::Table` to avoid the need for extra helpers when using resources. * A new `-Sallow-ip-name-lookup` option is added to control this on the CLI. * Implementation-wise this uses the blocking resolution in the Rust standard library, built on `getaddrinfo`. This doesn't invoke `getaddrinfo` "raw", however, so information such as error details can be lost in translation. This will probably need to change in the future. Closes #7070 * Validate the input domain name. * Use a crate-level helper for `spawn_blocking` --------- Co-authored-by: Dave Bakker <github@davebakker.io> * riscv64: Further optimize constants loading (#7139) * Further optimize RISCV constant loading * Simplify existing rule * Fix * Bless missing tests * Use right signedness for Imm20 * Refactor HTTP tests to be less flaky and more robust (#7143) * Refactor HTTP tests to be less flaky and more robust This commit refactors the wasi-http tests we have in this repository with a few changes: * Each test now uses a dedicated port for the test. The port 0 is bound and whatever the OS gives is used for the duration of the test. This prevents tests from possibly interfering with each other. * Server spawning is abstracted behind a single `Server` type now which internally has http1/2 constructors. * Timeouts for server shutdown are removed since they were failing in CI and are likely best handled for each test individually if necessary. As a minor cleanup the `tokio` usage in guest programs was removed as it was boilerplate in this case. This shouldn't affect the runtimes of tests, however. * Remove unused import * Don't panic from worker thread * Improve failure error message prtest:full * Fix some windows-specific issues * Fix async tests * Relax test assertion string * [wasmtime-wasi] make StdinStream and StdoutStream public (#7150) This allows embedders to implement those traits themselves rather than be restricted to using the built-in implementations. Fixes https://github.com/bytecodealliance/wasmtime/issues/7149 Signed-off-by: Joel Dice <joel.dice@fermyon.com> * wasmtime: Handle common cli args in `wasmtime serve` (#7134) * Refactor option handling in the `run` and `serve` commands * Use a real thread to manage epoch increments * Move module/component loading into src/common.rs * Require wasi-http and component-model to not be disabled in serve * riscv64: Consolidate conditional moves into one instruction (#7145) * riscv64: Add codegen tests for min/max * riscv64: Remove unused `Type` payload from `Select` * riscv64: Add codegen test for brif * riscv64: Consolidate conditional moves into one instruction This commit removes the `IntSelect` and `SelectReg` pseudo-instructions from the riscv64 backend and consolidates them into the `Select` instruction. Additionally the `Select` instruction is updated to subsume the functionality of these two previous instructions. Namely `Select` now operates with `ValueRegs` to handle i128 and additionally takes an `IntegerCompare` as the condition for the conditional branch to use. This commit touches a fair bit of the backend since conditional selection of registers was used in quite a few places. The previous `gen_select_*` functions are replaced with new typed equivalents of `gen_select_{xreg,vreg,freg,regs}`. Furthermore new `cmp_*` helpers were added to create `IntegerCompare` instructions which sort-of match conditional branch instructions, or at least the pnemonics they use. Finally since this affected the `select` CLIF instruction itself I went ahead and did some refactoring there too. The `select` instruction creates an `IntegerCompare` from its argument to use to generate the appropriate register selection instruction. This is basically the same thing that `brif` does and now both go through a new helper, `lower_int_compare`, which takes a `Value` and produces an `IntegerCompare` representing if that value is either true or false. This enables folding an `icmp` or an `fcmp`, for example, directly into a branching instruction. * Fix a test * Favor sign-extension in equality comparisons * riscv64: Add support for min/max instructions (#7146) * riscv64: Mark `fcmp` as already extended (#7151) * Fix printing error information in sync http tests (#7153) The usage of `?` accidentally caused the extra information to not get printed, so avoid the use of `?` until the extra error info has been attached. * Bump Wasmtime to 15.0.0 (#7154) Co-authored-by: Wasmtime Publish <wasmtime-publish@users.noreply.github.com> * riscv64: Move `sextend` optimizations into `sext` (#7156) This commit moves the optimizations from `sextend` for various shapes of instructions into the `sext` helper. This enables these optimizations to kick in when generating other instructions which require sign-extensions such as `icmp` and `brif`. This additionally fixes an issue introduced in #7121 where uextend-of-sextend wasn't translated correctly and the outer `uextend` was mistakenly omitted. * Update release notes for 14.0.0 (#7158) * Update release notes for 14.0.0 Wow quite a bit happened! * Review comments * Bump regalloc2 to 0.9.3 (#7160) * Bump regalloc2 to 0.9.3 * Update filetests for the regalloc bump * wasi-http: Use borrow syntax for borrowed resources (#7161) * Remove `own` and `borrow` annotations from wasi-http * Fix the tests, as borrows are now needed * Don't delete headers that we're borrowing * Add an error resource to WASI streams (#7152) * Change `bindgen!`'s trappable error to take an input type This commit removes the generated type for `trappable_error_type` from the `bindgen!` macro to allow users to provide a type instead. This works similarly as before with new conversion functions generated in traits which are used to convert the custom error into the ABI representation of what a WIT world expects. There are a few motivations for this commit: * This enables reducing the number of errors in play between async/sync bindings by using the same error type there. * This avoids an auto-generated type which is more difficult to inspect than one that's written down already and the source can more easily be inspected. * This enables richer conversions using `self` (e.g. `self.table_mut()`) between error types rather than purely relying on `Into`. This is important for #7017 where an error is going to be inserted into the table as it gets converted. * Fix tests * Update WASI to use new trappable errors This commit deals with the fallout of the previous commit for the WASI preview2 implementation. The main changes here are: * Bindgen-generated `Error` types no longer exist. These are replaced with `TrappableError<T>` where type aliases are used such as ```rust type FsError = TrappableError<wasi::filesystem::types::ErrorCode>; ``` * Type synonyms such as `FsResult<T>` are now added for more conveniently writing down fallible signatures. * Some various error conversions are updated from converting to the old `Error` type to now instead directly into corresponding `ErrorCode` types. * A number of cases where unknown errors were turned into traps now return bland error codes and log the error instead since these aren't fatal events. * The `StreamError` type does not use `TrappableError` since it has other variants that it's concerned with such as a `LastOperationFailed` variant which has an `anyhow::Error` payload. * Some minor preview1 issues were fixed such as trapping errors being turned into normal I/O errors by accident. * Add an `error` resource to WASI streams This commit adds a new `error` resource to the `wasi:io/streams` interface. This `error` resource is returned as part of `last-operation-failed` and serves as a means to discover through other interfaces more granular type information than a generic string. This error type has a new function in the `filesystem` interface, for example, which enables getting filesystem-related error codes from I/O performed on filesystem-originating streams. This is plumbed through to the adapter as well to return more than `ERRNO_IO` from failed read/write operations now too. This is not super fancy just yet but is intended to be a vector through which future additions can be done. The main thing this enables is to avoid dropping errors on the floor in the host and enabling the guest to discover further information about I/O errors on streams. Closes #7017 * Update crates/wasi-http/wit/deps/io/streams.wit Co-authored-by: Trevor Elliott <awesomelyawesome@gmail.com> * Update wasi-http wit too * Remove unnecessary clone --------- Co-authored-by: Trevor Elliott <awesomelyawesome@gmail.com> * add i128 support for iabs on x86_64 (#7166) * fuzzgen: Enable iabs.i128 on fuzzgen (#7168) * Add a whole lot of semicolons to WIT files (#7159) To prepare this commit I've run `WIT_REQUIRE_SEMICOLONS=1 ./ci/run-tests.sh` locally which configures `wit-parser` to generate an error for missing semicolons in WIT files. This led me to add quite a few semicolons in quite a few places in what is going to be the first of a few batches of semicolons. CI checks for this cannot be added just yet because the wasi-nn spec is a submodule which needs to be updated with semicolons before this repository can require semicolons. Nevertheless that doesn't stop us from using semicolons in the meantime (yay gradual rollout of changes!) so I figure this would be good to get in sooner rather than later. * wasi-nn: reenable CI task (#7164) The task was disabled due to a failing checksum. Since then, the `install-openvino-action` has improved and now _almost_ has cross-platform support. This change just restores the status quo in expectation that #6895 will significantly improve the testing story. prtest:full * fix(wasmtime-cli): wrong linker in wasi http (#7167) * chore: convert wasi http cli test to component * fix(wasmtime-cli): wrong linker in wasi http * fix(wasmtime-cli): fix multiple functions added to linker We will allow shadowing in linker when using WASI HTTP in order to prevent the following error: > import of `wasi:clocks/wall-clock` defined twice * chore: update to latest io streams wit definition * chore: rename main function of core module prtest:full * fix(wasmtime-cli): add only http functions to linker * chore: add test building http request and response * chore: disable temporarily wasi http test * docs(component): Fix typo in component translate (#7173) * Require semicolons in WIT files in CI (#7174) This commit updates CI to require semicolons in all WIT files and parsed WIT documents. Some minor updates were required in existing WIT files and the wasi-nn proposal was additionally updated to its latest version with semicolons. The wasi-nn update brought some minor changes to the WIT which required some minor changes here as well. * cranelift: Consider functions with TLS values as non leaf (#7177) * c-api: Fix type (#7178) This should be wasmtime_instance_pre_t Signed-off-by: Tyler Rockwood <rockwood@redpanda.com> * winch(x64): Add support for table instructions (#7155) * winch(x64): Add support for table instructions This change adds support for the following table insructions: `elem.drop`, `table.copy`, `table.set`, `table.get`, `table.fill`, `table.grow`, `table.size`, `table.init`. This change also introduces partial support for the `Ref` WebAssembly type, more conretely the `Func` heap type, which means that all the table instructions above, only work this WebAssembly type as of this change. Finally, this change is also a small follow up to the primitives introduced in https://github.com/bytecodealliance/wasmtime/pull/7100, more concretely: * `FnCall::with_lib`: tracks the presence of a libcall and ensures that any result registers are freed right when the call is emitted. * `MacroAssembler::table_elem_addr` returns an address rather than the value of the address, making it convenient for other use cases like `table.set`. -- prtest:full * chore: Make stack functions take impl IntoIterator<..> * Update winch/codegen/src/codegen/call.rs Co-authored-by: Trevor Elliott <awesomelyawesome@gmail.com> * Remove a dangling `dbg!` * Add comment on branching --------- Co-authored-by: Trevor Elliott <awesomelyawesome@gmail.com> * Skeleton and initial support for proof-carrying code. (#7165) * WIP veriwasm 2.0 Co-Authored-By: Chris Fallin <cfallin@fastly.com> * PCC: successfully parse some simple facts. Co-authored-by: Nick Fitzgerald <fitzgen@gmail.com> * PCC: plumb facts through VCode and add framework on LowerBackend to check them. Co-authored-by: Nick Fitzgerald <fitzgen@gmail.com> * PCC: code is carrying some proofs! Very simple test-case. Co-authored-by: Nick Fitzgerald <fitzgen@gmail.com> * PCC: add a `safe` flag for checked memory accesses. * PCC: add pretty-printing of facts to CLIF output. * PCC: misc. cleanups. * PCC: lots of cleanup. * Post-rebase fixups and some misc. fixes. * Add serde traits to facts. * PCC: add succeed and fail tests. * Review feedback: rename `safe` memflag to `checked`. * Review feedback. --------- Co-authored-by: Nick Fitzgerald <fitzgen@gmail.com> * Update syntax for `trappable_error_type` in `bindgen!` (#7170) * Update syntax for `trappable_error_type` in `bindgen!` Following through on a suggestion from #7152 this makes the syntax a bit more consistent with the rest of `bindgen!` today. This additionally updates the documentation on the `bindgen!` macro itself. * Update parsing to afford for versions * Support multiple versions in `bindgen!` (#7172) This commit fixes a bug in the `bindgen!` macro where when faced with multiple packages that differ only in version number invalid bindings were generated. The fix here is to add version number information to package module names if necessary in situations such as this. This required some refactoring internally to have a single source of truth for what the name of a module should be and avoid having it implicitly calculated in two locations. * riscv64: Improve `f{min,max}` (#7181) * PCC: add semantics for core add/shift/extend/amode ops on AArch64. (#7180) * PCC: add semantics for core add/shift/extend/amode ops on AArch64. This PR adds verification of facts on values produced by adds, shifts, and extends on AArch64, handling the various combination instructions (adds with builtin extends or shifts, for example), and also adds verification of all addressing modes, including those with builtin extends and shifts. It also splits the test suite into"succeed" and "fail" sets, and provides cases that PCC should catch. * Review feedback. * cranelift: Group labels by FuncId instead of Name (#7183) This prevents confusing label locations when there are multiple functions with the same Name. * wasi-sockets: Simplify & clarify TCP errors (#7120) * Use Rustix::Errno to unify error code mapping. * Clarify Connect failure state * Allow accept() to return transient errors. The original provision was added to align with preview3 streams that may only fail once. However, after discussing with Dan Gohman, we came to the conclusion that a stream of result<> could do the trick fine too. Fixes: https://github.com/WebAssembly/wasi-sockets/issues/22 * Fold `ephemeral-ports-exhausted` into `address-in-use` There is no cross-platform way to know the distinction between them * Clarify `local-address` behavior on unbound socket * Remove `concurrency-conflict` clutter, and just document it to be always possible. * Simplify state errors. They were unnecessarily detailed and mostly have no standardized equivalent in POSIX, so wasi-libc will probably just map them all back into a single EOPNOTSUPP or EINVAL or ... EISCONN/ENOTCONN can be derived in wasi-libc based on context and/or by checking `remote-address`. For example, `shutdown` can only be called on connected sockets, so if it returns `invalid-state` it can be unambiguously mapped to ENOTCONN. * Document that connect may return ECONNABORTED * Remove create-tcp/udp-socket not supported errors. These stem from back when the entire wasi-sockets proposal was one big single thing. In this day and age, when an implementation doesn't want to support TCP and/or UDP, it can simply _not_ implement that interface, rather than returning an error at runtime. * Simplify "not supported" and "invalid argument" error cases. There is a myriad of reasons why an argument might be invalid or an operation might be not supported. But there is few cross platform consistency in which of those error cases result in which error codes. The error codes that have been removed were fairly specific, but: - Were still missing error cases. So additional error codes would have needed to be created. - Implementations would have to bend over backwards to make it work cross platform, especially beyond just Win/Mac/Linux. - Didn't all have an equivalent in POSIX, so they would map back into a generic EINVAL anyways. * Move example_body out of lib.rs into its own test-case make room for other tests. * Refactor TCP integration tests: - Ad-hoc skeleton implementation of resources. - Add blocking wrappers around async operations. * Fix get/set_unicast_hop_limit on Linux * Test TCP socket states * Keep track of address family ourselves. Because I need the family for input validation. And the resulting code is more straightforward. * Add more tests and make it work on Linux * Fix Windows * Simplify integration tests. All platforms supported by wasmtime also support dualstack sockets. * Test ipv6_only inheritence * Test that socket options keep being respected, even if listen() has already been called * cargo fmt * Duplicate .wit changes to wasi-http * prtest:full * Fix BSD behavior of SO_SNDBUF/SO_RCVBUF * fmt * Fix type error * Got lost during merge * Implement listen backlog tests * Manually inherit buffer size from listener on MacOS. I suspect that these changes apply to any BSD platform, but I can't test that. * Keep track of IPV6_V6ONLY ourselves. - This provides cross-platform behaviour for `ipv6-only` - This eliminates the syscall in `validate_address_family` * Reject IPv4-compatible IPv6 addresses. * Remove intermediate SystemError trait * Fix ambiguous .into()'s * Fix IPV6_UNICAST_HOPS inheritance on MacOS * riscv64: Add GOT relocations for PIC code (#7184) * riscv64: Use `CALL_PLT` relocation for all calls * riscv64: Set the ELF RVC flag only when the C extension is enabled * riscv64: Use PC Relative Relocations for all Calls * riscv64: Add support for PIC symbols * feat(wasi-sockets): implement UDP (#7148) * feat(wasi-sockets): implement UDP This is based on TCP implementation Signed-off-by: Roman Volosatovs <rvolosatovs@riseup.net> * refactor(wasi-sockets): simplify UDP implementation This introduces quite a few changes compared to TCP, which should most probably be integrated there as well Signed-off-by: Roman Volosatovs <rvolosatovs@riseup.net> * feat(wasi-sockets): store UDP connect address in state Signed-off-by: Roman Volosatovs <rvolosatovs@riseup.net> * fix(wasi-sockets): avoid `shutdown` on `drop` Signed-off-by: Roman Volosatovs <rvolosatovs@riseup.net> * Remove explicit bind * Simplify start_connect&finish_connect. On UDP sockets, `connect` never blocks. * Move UDP test to single file, similar to `tcp_sample_application.rs` * Update UDP tests --------- Signed-off-by: Roman Volosatovs <rvolosatovs@riseup.net> Co-authored-by: Dave Bakker <github@davebakker.io> * Ensure exports are properly namespaced (#7196) Continuation of #7172 where imports and export _definitions_ are properly namespaced by version by the usage of exports was not. * add i128 support for iabs on aarch64 (#7185) * Reapply #7150 (#7197) Signed-off-by: Ryan Levick <ryan.levick@fermyon.com> * riscv64: Optimize `gen_bmask` slightly (#7200) If the input operand is an `icmp` or an `fcmp` there's no need to use `snez` since the output value is already guaranteed to be zero or one. * Refactor the test-programs test suite (#7182) * Refactor the test-programs test suite This commit is a large refactoring that reorganizes `test-programs` and how we tests wasms in Wasmtime. Often writing tests requires complicated interactions with the guest which can't be done via hand-written `*.wat` syntax and requires a compiler to get engaged. For this purpose Wasmtime currently has the `crates/test-programs/*` test suite which builds files from source and then runs the tests. This has been somewhat cumbersome in the past though and it's not been easy to extend this over time, so this commit attempts to address this. The scheme implemented in this PR looks like: * All wasm test programs live in `crates/test-programs/src/bin/*.rs`. All of them, no exceptions. * Wasm tests have shared support located at `crates/test-programs/src/lib.rs` and its submodules, such as bindings generation for WASI. * Wasm tests are built by a new `crates/test-programs/artifacts` crate. This crate compiles modules and additionally creates components for all test programs. The crate itself only records the path to these outputs and a small amount of testing support, but otherwise doesn't interact with `wasmtime`-the-crate itself. * All tests in `crates/test-programs/tests/*.rs` have moved. For example wasi-http tests now live at `crates/wasi-http/tests/*.rs`. Legacy tests of wasi-common now live at `crates/wasi-common/tests/*.rs`. Modern tests for preview2 live at `crates/wasi/tests/*.rs`. * Wasm tests are bucketed based on their filename prefix. For example `preview1_*` is tested in wasi-common and wasmtime-wasi. The `preview2_*` prefix is only tested with wasmtime-wasi, however. * A new `cli_*` prefix is used to execute tests as part of `tests/all/main.rs`. This is a new submodule in `tests/all/cli_tests.rs` which executes these components on the command line. Many old "command" tests were migrated here. * Helper macros are generated to assert that a test suite is run in its entirety. This way if a `preview1_*` test is added it's asserted to get added to both wasi-common and wasmtime-wasi in the various modes they run tests. Overall this moved a number of tests around and refactored some edges of the tests, but this should not lose any tests (except one that wasn't actually testing anything). Additionally the hope is that it's much easier to add tests in the future. The process is to add a new file in `crates/test-programs/src/bin/*.rs` named appropriately. For example a preview2 executable is `preview2_*` and a CLI tests is `cli_*`. When building the test suite an error is generated in the appropriate module then of "please write a test here", and then a test is written in the same manner as the other tests in the module. * Remove no-longer-needed fetches prtest:full * I'm worried wasi is running low on semicolons * Add the WASI target in all CI actions * Add unknown-unknown target on all CI builders too * Fix building test artifacts under miri Need to avoid wrappers for these cross-compiled targets * Break circular dependency for packaging Don't use the workspace dep for `wasmtime-wasi` since it injects a version, instead use a `path = '..'` dependency to fool Cargo into dropping the dependency during the package phase. * Fix some merge conflicts with tests * Fix rebase for new tests * Remove stray comment * Fix some flaky tests * Fix network tests in synchronous mode This commit is an attempt to fix some networking tests in synchronous mode in our test suite. Currently networking tests do…

elliottt added 10 commits September 21, 2023 17:16

Move the incoming_handler impl into http_impl

65fc5c7

Remove the incoming handler -- we need to use it as a guest export

1b817ea

Start adding a test-programs test for the server side of wasi-http

df83b75

Progress towards running a server test

4d36120

Implement incoming-request-method

54c2a5c

Validate outparam value

1697b77

Initial incoming handler test

25432ea

Implement more of the incoming api

d021210

Finish the incoming api implementations

b194929

Initial cut at wasmtime serve

894fdca

pchickey force-pushed the trevor/wasi-http-server branch from 4871e00 to 894fdca Compare September 26, 2023 17:53

Pat Hickey added 9 commits September 26, 2023 13:38

fix warning

0b4a07c

wasmtime-cli: invoke ServeCommand, and add enough stuff to the linker…

f5f20f7

… to run trivial test

fix warnings

06e120a

fix warnings

3f1c7d7

argument parsing: allow --addr to specify sockaddr

61d9093

rustfmt

622875b

sync wit definitions between wasmtime-wasi and wasmtime-wasi-http

0880e04

cargo vet: add an import config and wildcard audit for wasmtime-wmemc…

3585d2d

…heck

cargo vet: audit signal-hook-registry

e9f77e1

pchickey marked this pull request as ready for review September 27, 2023 18:19

pchickey requested review from a team as code owners September 27, 2023 18:19

pchickey requested review from alexcrichton and removed request for a team September 27, 2023 18:19

github-actions bot added the wasi Issues pertaining to WASI label Sep 27, 2023

elliottt force-pushed the trevor/wasi-http-server branch 4 times, most recently from 0ef0823 to 3a26912 Compare September 27, 2023 20:57

elliottt force-pushed the trevor/wasi-http-server branch from a58214f to 293930e Compare September 27, 2023 22:17

Remove duplicate add_to_linker calls for preview2 interfaces

a6115d8

prtest:full

elliottt force-pushed the trevor/wasi-http-server branch 2 times, most recently from f47f991 to f0b74dc Compare September 28, 2023 00:17

Add a method to finish outgoing responses

19a574b

Co-authored-by: Adam Foltzer <acfoltzer@fastly.com> Co-authored-by: Pat Hickey <phickey@fastly.com>

elliottt force-pushed the trevor/wasi-http-server branch from f0b74dc to 19a574b Compare September 28, 2023 00:23

Mark the result of the incoming_{request,response}_consume methods as…

2a2f33d

… own

alexcrichton approved these changes Sep 28, 2023

View reviewed changes

elliottt added 2 commits September 28, 2023 12:15

Explicit versions for http-body and http-body-util

698f0ff

Explicit serve feature for the wasmtime serve command

147713f

elliottt mentioned this pull request Sep 28, 2023

Consistent argument handling between the run and serve commands #7105

Closed

pchickey mentioned this pull request Sep 28, 2023

Start to port Wasmtime to the new wasi-io API with resources. #7029

Merged

elliottt added 2 commits September 28, 2023 12:50

Move the spawn outside of the future returned by ProxyHandler::call

cc24768

Review feedback

2c17fe7

elliottt mentioned this pull request Sep 28, 2023

Graceful shutdown with the serve command #7107

Open

elliottt enabled auto-merge September 28, 2023 20:55

elliottt added this pull request to the merge queue Sep 28, 2023

elliottt removed this pull request from the merge queue due to a manual request Sep 28, 2023

elliottt added this pull request to the merge queue Sep 28, 2023

elliottt removed this pull request from the merge queue due to a manual request Sep 28, 2023

elliottt added this pull request to the merge queue Sep 28, 2023

github-merge-queue bot removed this pull request from the merge queue due to failed status checks Sep 28, 2023

elliottt added this pull request to the merge queue Sep 28, 2023

Merged via the queue into main with commit 8a88ff6 Sep 28, 2023
34 checks passed

elliottt deleted the trevor/wasi-http-server branch September 28, 2023 23:42

alexcrichton mentioned this pull request Sep 29, 2023

Wasmtime wasi-http: support inbound requests #7025

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Wasi-http: support inbound requests (proxy world) #7091

Wasi-http: support inbound requests (proxy world) #7091

pchickey commented Sep 26, 2023 •

edited

Loading

alexcrichton left a comment

alexcrichton Sep 28, 2023

alexcrichton Sep 28, 2023

alexcrichton Sep 28, 2023

elliottt Sep 28, 2023

elliottt Sep 28, 2023

alexcrichton Sep 28, 2023

elliottt Sep 28, 2023

elliottt Sep 28, 2023

alexcrichton Sep 28, 2023

elliottt Sep 28, 2023

elliottt Sep 28, 2023

elliottt Sep 28, 2023

alexcrichton Sep 28, 2023

alexcrichton Sep 28, 2023

elliottt Sep 28, 2023

elliottt commented Sep 28, 2023

Wasi-http: support inbound requests (proxy world) #7091

Wasi-http: support inbound requests (proxy world) #7091

Conversation

pchickey commented Sep 26, 2023 • edited Loading

alexcrichton left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

elliottt commented Sep 28, 2023

pchickey commented Sep 26, 2023 •

edited

Loading