Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Skeleton and initial support for proof-carrying code. #7165

Merged
merged 13 commits into from
Oct 6, 2023

Conversation

cfallin
Copy link
Member

@cfallin cfallin commented Oct 6, 2023

This PR adds basic support for "facts" to annotate SSA values and VCode VRegs. The idea is that these facts describe properties of the values -- currently, "in a given range" or "points to valid memory of a given size" -- and by validating a chain of facts along the whole derivation of an accessed memory address, we can verify that the memory access is safe.

This is the start of work on the ideas described in #6090. It is a new approach that subsumes/replaces "VeriWasm": it replaces the post-hoc derive-all-facts analysis approach (which is slow and brittle) with a cooperative approach in which the codegen frontend generates a "proof witness" of sorts, we preserve that, and we check that in the machine code (actually the pre-regalloc VCode, and then we can use the regalloc's symbolic checker to obtain the same guarantee).

Right now, this includes (i) the "fact" data structures, (ii) support for parsing these facts in CLIF and dumping them back out, (iii) propagation of facts from CLIF to VCode, when no optimization is enabled, and (iv) validation of a few basic facts in VCode for the AArch64 backend. Everything else is still TODO and will come in further PRs.

Right now, this is sufficient to take the CLIF

test compile
set enable_pcc=true
target aarch64

function %simple1(i64 vmctx, i32) -> i8 {
block0(v0 ! points_to(0x1_0000_0000): i64, v1 ! max(32, 0xffff_ffff): i32):
    v2 ! max(64, 0xffff_ffff) = uextend.i64 v1
    v3 ! points_to(0x1) = iadd.i64 v0, v2
    v4 = load.i8 safe v3
    return v4
}

where the v0 ! fact syntax annotates v0 with a fact, and validate that the load is in fact safe. The safe memory-flag forces this check; if the points_to range is shrunk, or v1 is changed to an i64 and its max range is increased, this code will fail to compile with

 ERROR cranelift_filetests::concurrent > FAIL: compile
FAIL cranelift/filetests/filetests/pcc/simple.clif: compile

Caused by:
    Proof-carrying-code validation error: MissingFact

The idea is to continue further work along these lines:

  • Add support for validating facts produced by aarch64 add, shift, ... instructions -- whatever is used in basic address generation.
  • Add support for validating amode usage in x64 as well.
  • Propagate facts through the egraph layer so they are preserved during optimization.
  • Add a richer points_to language for facts to describe what they point to, so a load of a memory base address (for example) from vmctx yields another value with an attached fact (points to a region of this size...). These are the "memory capabilities" described in Verification of addressing / memory accesses with VeriWasm or similar #6090.
  • Emit facts on vmctx and on memory-access sequences generated by cranelift-wasm, and add the safe flag to loads/stores wherever possible. At this point, we should be able to validate single- and multi-memory Wasm modules with static memory configurations.
  • Add support for annotations on bounds-check logic. After this, we should be able to validate dynamic memories, and table accesses as well.
  • Finally, add a "strong enforcing" mode that disallows all non-safe loads/stores, and add whatever memory-capability info is needed so that, e.g., loads of func pointers, VM state, and everything else from vmctx is covered.

After my initial exploration and ideas in #6090, these ideas were further discussed with @fitzgen, @elliottt, @jameysharp, and others; and a large part of this PR was

Co-authored-by: Nick Fitzgerald fitzgen@gmail.com

@cfallin cfallin requested review from a team as code owners October 6, 2023 00:02
@cfallin cfallin requested review from elliottt and fitzgen and removed request for a team October 6, 2023 00:02
@github-actions github-actions bot added cranelift Issues related to the Cranelift code generator cranelift:area:machinst Issues related to instruction selection and the new MachInst backend. cranelift:area:aarch64 Issues related to AArch64 backend. cranelift:area:x64 Issues related to x64 codegen cranelift:meta Everything related to the meta-language. wasmtime:api Related to the API of the `wasmtime` crate itself labels Oct 6, 2023
@github-actions
Copy link

github-actions bot commented Oct 6, 2023

Subscribe to Label Action

cc @peterhuene

This issue or pull request has been labeled: "cranelift", "cranelift:area:aarch64", "cranelift:area:machinst", "cranelift:area:x64", "cranelift:meta", "wasmtime:api"

Thus the following users have been cc'd because of the following labels:

  • peterhuene: wasmtime:api

To subscribe or unsubscribe from this label, edit the .github/subscribe-to-label.json configuration file.

Learn more.

Copy link
Member

@fitzgen fitzgen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Very excited for this! r=me with comments, some of which can be deferred to follow up PRs.

cranelift/codegen/src/ir/memflags.rs Outdated Show resolved Hide resolved
cranelift/codegen/src/ir/pcc.rs Outdated Show resolved Hide resolved
cranelift/codegen/src/ir/pcc.rs Outdated Show resolved Hide resolved
cranelift/codegen/src/ir/pcc.rs Show resolved Hide resolved
cranelift/codegen/src/ir/pcc.rs Outdated Show resolved Hide resolved
cranelift/codegen/src/ir/pcc.rs Outdated Show resolved Hide resolved
cranelift/codegen/src/ir/pcc.rs Outdated Show resolved Hide resolved
cranelift/codegen/src/ir/pcc.rs Outdated Show resolved Hide resolved
cranelift/codegen/src/ir/pcc.rs Outdated Show resolved Hide resolved
cranelift/codegen/src/ir/pcc.rs Show resolved Hide resolved
Merged via the queue into bytecodealliance:main with commit f466aa2 Oct 6, 2023
22 checks passed
@cfallin cfallin deleted the pcc branch October 6, 2023 23:19
eduardomourar pushed a commit to eduardomourar/wasmtime that referenced this pull request Oct 9, 2023
…ce#7165)

* WIP veriwasm 2.0

Co-Authored-By: Chris Fallin <cfallin@fastly.com>

* PCC: successfully parse some simple facts.

Co-authored-by: Nick Fitzgerald <fitzgen@gmail.com>

* PCC: plumb facts through VCode and add framework on LowerBackend to check them.

Co-authored-by: Nick Fitzgerald <fitzgen@gmail.com>

* PCC: code is carrying some proofs! Very simple test-case.

Co-authored-by: Nick Fitzgerald <fitzgen@gmail.com>

* PCC: add a `safe` flag for checked memory accesses.

* PCC: add pretty-printing of facts to CLIF output.

* PCC: misc. cleanups.

* PCC: lots of cleanup.

* Post-rebase fixups and some misc. fixes.

* Add serde traits to facts.

* PCC: add succeed and fail tests.

* Review feedback: rename `safe` memflag to `checked`.

* Review feedback.

---------

Co-authored-by: Nick Fitzgerald <fitzgen@gmail.com>
github-merge-queue bot pushed a commit to near/wasmtime that referenced this pull request Oct 27, 2023
* Do proper type checking for type handles. (#7065)

Instead of relying purely on the assumption that type handles can be compared
cheaply by pointer equality, fallback to a more expensive walk of the
type tree that recursively compares types structurally.

This allows different components to call into each other as long as
their types are structurally equivalent.

Signed-off-by: Ryan Levick <ryan.levick@fermyon.com>

* Use wasi-streams in the wasi-http implementation (#7056)

* Start refactoring wasi-http

* Checkpoint

* Initial implementation of response future handling

* Lazily initialize response headers and body

* make wasmtime-wasi-http compile

* wasi-http wit: make a way to reject outgoing-request in outgoing-handler

before waiting for the future to resolve

* wasi: sync wit from wasi-http

* outgoing handler impl: report errors to userland

* test-programs: get wasi-http-components kicking over, delete modules and components-sync tests

wasi-http-components-sync will come back once we get done with other
stuff, but its superfulous for now. wasi-http-modules will not be
returning.

* Process headers

* Add HostIncomingBody::new

* Add trailers functions

* Add TODO for body task outline

* Rework incoming-response-consume to return a future-trailers value as well

* Fix the wit

* First cut at the worker loop

* wasi-http: change how we represent bodies/trailers, and annotate own/borrow/child throughout

* Update types_impl.rs for wit changes

* Split body management into its own module

* Checkpoint

* more work on incoming body and future trailers

* Fill out some more functions

* Implement future-trailers-{subscribe,get}

* Implement drop-future-trailers

* Rework fields, but make the borrow checker mad

* Fix borrow error

* wasi-http-tests: fix build

* test-runner: report errors with stdout/stderr properly

* fix two trivial wasi-http tests

the error type here changed from a types::Error to an
outbound_handler::Error

* Remove unnecessary drops

* Convert a `bail!` to a `todo!`

* Remove a TODO that documented the body worker structure

* fill in a bunch more of OutputBody

* Remove the custom FrameFut future in favor of using http_body_util

* Move the outgoing body types to body.rs

* Rework the handling of outgoing bodies

* Fix the `outgoing request get` test

* Avoid deadlocking the post tests

* future_incoming_request_get shouldn't delete the resource

* Fix the invalid_dnsname test

* implement drop-future-incoming-response

* Fix invalid_port and invalid_dnsname tests

* Fix the post test

* Passing a too large string to println! caused the large post test to fail

* Format

* Plumb through `between_bytes_timeout`

* Downgrade hyper

* Revert "Downgrade hyper"

This reverts commit fa0750e0c42823cb785288fcf6c0507c5ae9fd64.

* Restore old https connection setup

* Sync the wasi and wasi-http http deps

* Fix tests

* Remove the module and component-sync tests, as they are currently not
  supported
* Fix the reference to the large_post test in the components test

* Fix wasi-http integration

* sync implementation of wasi-http

* Slightly more robust error checking

* Ignore the wasi-http cli test

prtest:full

* Consistent ignore attributes between sync and async tests

* Fix doc errors

* code motion: introduce intermediate `HostIncomingBodyBuilder` rather than a tuple

* explain design

* Turn FieldMap into a type synonym

* Tidy up some future state (#7073)

Co-authored-by: Pat Hickey <phickey@fastly.com>

* body HostInputStream: report runtime errors with StreamRuntimeError

HostInputStream is designed wrong to need that in the first place. We
will fix it in a follow-up as soon as resources land.

---------

Co-authored-by: Pat Hickey <phickey@fastly.com>
Co-authored-by: Alex Crichton <alex@alexcrichton.com>

* Support resource maps in `component::bindgen!` (#7069)

* Support resource maps in `component::bindgen!`

This commit adds support to `component::bindgen!` to specify resource
types using the `with` key of the macro. This can be used to configure
the `T` of `Resource<T>` to use a preexisting type rather than
unconditionally generating a new empty enum to have a fresh type.

* Reenable tests

* Include Wasm-defined globals and memories in core dumps (#6935)

* Include Wasm-defined globals and memories in core dumps

Co-authored-by: Nick Fitzgerald <fitzgen@gmail.com>

* Narrow scope of unsafe block

* Add missing skip-miri attribute for test that calls into Wasm

---------

Co-authored-by: Nick Fitzgerald <fitzgen@gmail.com>

* Remove manual impl of Error/Display (#7076)

* Remove manual impl of Error/Display

These conflict with auto-generated ones.

* Fix tests

* riscv64: Better AMode Matching (#7079)

* riscv64: Delete `int_load_op` helper

* riscv64: Delete `default_memflags`

* riscv64: Rename `gen_amode`

* riscv64: Better matching for amode

* riscv64: Delete `emit_load` helper

* riscv64: Rename some load variables

* riscv64: Merge `iadd` into amode computations

* riscv64: Add the remainder of Zca and Zcd instructions (#7080)

* riscv64: Add `c.li` and `c.lui`

* riscv64: Add CB type instructions

`c.srli` / `c.srai` / `c.andi`

* riscv64: Add `sp` relative load instructions

* riscv64: Return Option from try_emit_compressed

* riscv64: Implement stack based stores

* riscv64: Add compressed stores

* riscv64: Add compressed loads

* Make `wasmtime::WasmCoreDump` serializable (#7078)

This commit makes it so that the library type for core dumps is serializable
into the standard binary format for core dumps.

Additionally, this commit makes it so that we use the library type for
generating core dumps in the CLI. We previously were using a one-off
implementation of core dump generation that only had backtrace information and
no instances, modules, globals, or memories included. The library type has all
that information, so the core dumps produced by our CLI will both be more
featureful and be generated by shared code paths going forward.

Along the way, implementing all this required some new helper methods sprinkled
throughout `wasmtime` and `wasmtime-runtime`:

* `wasmtime::Instance::module`: get the module that a `wasmtime::Instance` is an
  instance of. This is public, since it seems generally useful. This involved
  adding a new return value from `ModuleRegistry::register_module` that is an
  identifier that can be used to recover a reference to the registered module.

* `wasmtime::Instance::all_{globals,memories}`: get the full global/memory index
  space. I made these `pub(crate)` out of caution. I don't think we want to commit
  to exposing non-exported things in the public API, even if we internally need
  them for debugging-related features like core dumps. These also needed
  corresponding methods inside `wasmtime-runtime`.

* `wasmtime::{Global,Memory}::hash_key`: this was needed to work around the fact
  that each time you call `{Global,Memory}::from_wasmtime`, it creates a new
  entry in the `StoreData` and so you can get duplicates. But we need to key some
  hash maps on globals and memories when constructing core dumps, so we can't
  treat the underlying `Stored<T>` as a hash key because it isn't stable across
  duplicate `StoreData` entries. So we have these new methods. They are only
  `pub(crate)`, are definitely implementation details, and aren't exposed in the
  public API.

* `wasmtime::FrameInfo::module`: Each frame in a backtrace now keeps a handle to
  its associated module instead of just the name. This is publicly exposed
  because it seems generally useful. This means I also deprecated
  `wasmtime::FrameInfo::module_name` since you can now instead do
  `frame.module().name()` to get that exact same info. I updated callers inside
  the repo.

* Optimize wasm calls ever-so-slightly (#7084)

* Optimize wasm calls ever-so-slightly

* Fix riscv64

* Remove stray comment

* Shuffle around where `unsafe` lies

* Add `Func::hash_key` and `Table::hash_key` methods (#7086)

* Wasmtime: Move `Global` and `Table` to `externals` submodules

Just mechanical code motion, not functional changes.

* Wasmtime: Add `hash_key` methods for `Func` and `Table`

* Rename `wasmtime::Func::caller_checked_func_ref` to `wasmtime::Func::vm_func_ref`

We removed the "caller checked" part of the `VMFuncRef` type's name a while ago,
so update this method to be in line with that.

No functional changes, just mechanical renaming.

* cargo fmt

* Fix doc link

* cranelift(x64): Make xmm{8..15} registers non-preferred (#7089)

These registers require an additional byte to reference when encoded in certain
AVX instruction formats (and maybe other situations as well?) so prefer
xmm{0..7} when they are available and only fall back to xmm{8..15} when register
pressure is higher.

* Add docs and example for debugging with core dumps (#7087)

* Add docs for debugging with core dumps

* Fix reference to old style CLI flag

* Add `no_run` to example that is only there to trap

* cranelift(x64): Swap operands to save AVX instruction encoding size (#7093)

For `XmmRmiRVex`-format instructions, when the opcode is commutative, the first
operand is one of xmm{0..7}, and the second operand is one of xmm{8..15}, then
we can swap the operands to save a byte on instruction encoding.

* Update `ittapi` crate (#7092)

This update gets rid of a panic that occurs when using `--profile=vtune`
from the command line.

* Bump wasm-tools crates (#7094)

* Bump wasm-tools crates

Two major changes/reasons for this update:

* Primarily pulling in support for semicolons-in-WIT files. Semicolons are
  not currently required, though, so I'll follow-up later with actual
  semicolons.

* The syntax for parsing `(if ...)` was fixed in `wast`. Previously it
  did not require `(then ...)` but this is required by the spec. New
  spec tests require this as well. This breaks existing text format
  tests which don't use `(then ...)` inside of an `(if ...)`. Most tests
  were updated by hand but `embenchen_*` tests were updated by running
  through the old parser to produce non-s-expression using code.

* Fix an example `*.wat`

* Minor preview2 impl cleanups (#7097)

* delete preview2::pipe::pipe

this is not used anywhere, not documented, and trivial for the user to
build on their own if they need it. i suspect they probably only want
one end of their pipe wrapped with AsyncReadStream or AsyncWriteStream.

* fix cfg_attr to ignore forking tests on qemu

* delete "async fd stdin" test, which is just the worker thread one anyway

* Add MPK-protected stripes to the pooling allocator (#7072)

* Add memory protection keys (MPK)

In order to use MPK on an x86_64 Linux system, we need access to the
underlying `pkey_*` system calls (`sys`), control of the x86 PKRU
register (`pkru`), and a way of determining if MPK is even supported
(`is_supported`). These various parts are wrapped in a `ProtectionKey`
abstraction along with a `ProtectionMask` that can be used `allow` the
CPU to access protected regions.

* Integrate MPK into the pooling allocator

This change adds "stripes" to the pooling allocator's `MemoryPool`. Now,
when requesting a slot in which to instantiate, the user (i.e.,
`InstanceAllocationRequest`) will be transparently assigned to one of
the stripes, each of which is associated with a protection key. The user
can also request a specific protection key to use, which will override
the original "find me a slot logic".

This has implications for how instances get allocated: once a store is
assigned a protection key, it will only allocate requests with that key,
limiting how many slots it has access to. E.g., if 15 keys are active,
the store can only ever access 1/15th of the slots.

This change also includes a tri-bool configuration field,
`memory_protection_keys`, which is disabled by default for the time
being.

* Address review comments

This is a rollup of 43 commits addressing review comments of various
kinds: bug fixes, refactorings, documentation improvements, etc. It also
ensures that CI runs all checks. A big thanks to @fitzgen and
@alexcrichton for the review!

prtest:full

Co-authored-by: Nick Fitzgerald <fitzgen@gmail.com>
Co-authored-by: Alex Crichton <alex@alexcrichton.com>

---------

Co-authored-by: Nick Fitzgerald <fitzgen@gmail.com>
Co-authored-by: Alex Crichton <alex@alexcrichton.com>

* x64: Fix false dependencies in int-to-float conversions (#7098)

* x64: Fix false dependencies in int-to-float conversions

This commit is a result of the investigation on #7085. The int-to-float
conversion instructions used right now on the x64 backend will
implicitly source the upper bits of the result from a different
register. This implicitly creates a dependency on further consumers
using the conversion result on whatever previously defined the upper
bits, even though they aren't used. This false dependency is the primary
reason for the slowdown witnessed in #7085.

The fix chosen in this commit is to model the int-to-float instructions
with a new shape of instruction instead of the previous `GprToXmm{,Vex}`. This
previous shape was modeled as single-input and single-output, but this
does not reflect the actual nature of the `cvtsi2s{s,d}` instructions.
Instead these now use `CvtIntToFloat{,Vex}` which have two source
operands and one destination operand, modeling how the upper bits of a
different register are used. In lowerings using this instruction the
upper bits to preserver are always sourced from a zero'd out register to
force breaking dependencies between instructions.

Closes #7085

* Remove now dead code

* Remove outdated test

Golden test output covers this test case anyway nowadays

* Review comments

* Fix emit tests

* Update wit-{parser,component} deps (#7101)

Pull in bytecodealliance/wasm-tools#1221

* wasmtime: Misc optimizations (#7102)

* Mark some frequently used methods as `inline`

* Seed the preview2 context with the thread_rng, instead of using a syscall

* Remove stray `component_impl.rs` file (#7103)

This has no longer been used since #7056 and I believe it's just an
accident that it wasn't removed as part of that PR.

* ci: Upgrade QEMU to `8.1.1` (#7096)

* ci: Upgrade QEMU to `8.1.1`

This adds support for RISC-V's Zcb extension that includes some
extra compressed instructions.

It also removes the current cpuinfo patch, that has been released
in 8.1

* wasmtime: Don't assert the exact faulting address for wasm traps

* Wasi-http: support inbound requests (proxy world) (#7091)

* Move the incoming_handler impl into http_impl

* Remove the incoming handler -- we need to use it as a guest export

* Start adding a test-programs test for the server side of wasi-http

* Progress towards running a server test

* Implement incoming-request-method

* Validate outparam value

* Initial incoming handler test

* Implement more of the incoming api

* Finish the incoming api implementations

* Initial cut at `wasmtime serve`

* fix warning

* wasmtime-cli: invoke ServeCommand, and add enough stuff to the linker to run trivial test

* fix warnings

* fix warnings

* argument parsing: allow --addr to specify sockaddr

* rustfmt

* sync wit definitions between wasmtime-wasi and wasmtime-wasi-http

* cargo vet: add an import config and wildcard audit for wasmtime-wmemcheck

* cargo vet: audit signal-hook-registry

* Remove duplicate add_to_linker calls for preview2 interfaces

prtest:full

* Add a method to finish outgoing responses

Co-authored-by: Adam Foltzer <acfoltzer@fastly.com>
Co-authored-by: Pat Hickey <phickey@fastly.com>

* Mark the result of the incoming_{request,response}_consume methods as own

* Explicit versions for http-body and http-body-util

* Explicit `serve` feature for the `wasmtime serve` command

* Move the spawn outside of the future returned by `ProxyHandler::call`

* Review feedback

---------

Co-authored-by: Trevor Elliott <telliott@fastly.com>
Co-authored-by: Adam Foltzer <acfoltzer@fastly.com>

* c-api: Correct wasmtime_module_image_range type (#7110)

* Make C API usable from Rust (#6765)

* Remove usage of `is-terminal` and `atty` crates (#7104)

* Remove usage of `is-terminal` and `atty` crates

This functionality is now folded into the standard library itself.

* Fix syntax

* Fix a unix/windows cfg

* winch(x64): Call indirect (#7100)

* winch(x64): Call indirect

This change adds support for the `call_indirect` instruction to Winch.

Libcalls are a pre-requisite for supporting `call_indirect` in order to
lazily initialy funcrefs. This change adds support for libcalls to
Winch by introducing a `BuiltinFunctions` struct similar to Cranelift's
`BuiltinFunctionSignatures` struct.

In general, libcalls are handled like any other function call, with the
only difference that given that not all the information to fulfill the
function call might be known up-front, control is given to the caller
for finalizing the call.

The introduction of function references also involves dealing with
pointer-sized loads and stores, so this change also adds the required
functionality to `FuncEnv` and `MacroAssembler` to be pointer aware,
making it straight forward to derive an `OperandSize` or `WasmType` from
the target's pointer size.

Finally, given the complexity of the call_indirect instrunction, this
change bundles an improvement to the register allocator, allowing it to
track the allocatable vs non-allocatable registers, this is done to
avoid any mistakes when allocating/de-allocating registers that are not
alloctable.

--
prtest:full

* Address review comments

* Fix typos
* Better documentation for `new_unchecked`
* Introduce `max` for `BitSet`
* Make allocatable property `u64`

* winch(calls): Overhaul `FnCall`

This commit simplifies `FnCall`'s interface making its usage more
uniform throughout the compiler. In summary, this change:

* Avoids side effects in the `FnCall::new` constructor, and also makes
  it the only constructor.
* Exposes `FnCall::save_live_registers` and
  `FnCall::calculate_call_stack_space` to calculate the stack space
  consumed by the call and so that the caller can decide which one to
  use at callsites depending on their use-case.

* tests: Fix regset tests

* Start to port Wasmtime to the new wasi-io API with resources. (#7029)

* Rename `Host*` things to avoid name conflicts with bindings.

* Update to the latest resource-enabled wit files.

* Adapting the code to the new bindings.

* Update wasi-http to the resource-enabled wit deps.

* Start adapting the wasi-http code to the new bindings.

* Make `get_directories` always return new owned handles.

* Simplify the `poll_one` implementation.

* Update the wasi-preview1-component-adapter.

FIXME: temporarily disable wasi-http tests.

Add logging to the cli world, since stderr is now a reseource that
can only be claimed once.

* Work around a bug hit by poll-list, fix a bug in poll-one.

* Comment out `test_fd_readwrite_invalid_fd`, which panics now.

* Fix a few FIXMEs.

* Use `.as_ref().trapping_unwrap()` instead of `TrappingUnwrapRef`.

* Use `drop_in_place`.

* Remove `State::with_mut`.

* Remove the `RefCell` around the `State`.

* Update to wit-bindgen 0.12.

* Update wasi-http to use resources for poll and I/O.

This required making incoming-body and outgoing-body resourrces too, to
work with `push_input_stream_child` and `push_output_stream_child`.

* Re-enable disabled tests, remove logging from the worlds.

* Remove the `poll_list` workarounds that are no longer needed.

* Remove logging from the adapter.

That said, there is no replacement yet, so add a FIXME comment.

* Reenable a test that now passes.

* Remove `.descriptors_mut` and use `with_descriptors_mut` instead.

Replace `.descriptors()` and `.descriptors_mut()` with functions
that take closures, which limits their scope, to prevent them from
invalid aliasing.

* Implement dynamic borrow checking for descriptors.

* Add a cargo-vet audit for wasmtime-wmemcheck.

* Update cargo vet for wit-bindgen 0.12.

* Cut down on duplicate sync/async resource types (#1)

* Allow calling `get-directories` more than once (#2)

For now `Clone` the directories into new descriptor slots as needed.

* Start to lift restriction of stdio only once  (#3)

* Start to lift restriction of stdio only once

This commit adds new `{Stdin,Stdout}Stream` traits which take over the
job of the stdio streams in `WasiCtxBuilder` and `WasiCtx`. These traits
bake in the ability to create a stream at any time to satisfy the API
of `wasi:cli`. The TTY functionality is folded into them as while I was
at it.

The implementation for stdin is relatively trivial since the stdin
implementation already handles multiple streams reading it. Built-in
impls of the `StdinStream` trait are also provided for helper types in
`preview2::pipe` which resulted in the implementation of
`MemoryInputPipe` being updated to support `Clone` where all clones read
the same original data.

* Get tests building

* Un-ignore now-passing test

* Remove unneeded argument from `WasiCtxBuilder::build`

* Fix tests

* Remove some workarounds

Stdio functions can now be called multiple times.

* If `poll_oneoff` fails part-way through, clean up properly.

Fix the `Drop` implementation for pollables to only drop the pollables
that have been successfully added to the list.

This fixes the poll_oneoff_files failure and removes a FIXME.

---------

Co-authored-by: Alex Crichton <alex@alexcrichton.com>

* Cranelift: return programmatic error rather than panic when temporaries run out. (#7114)

* Cranelift: return programmatic error rather than panic when temporaries run out.

Cranelift currently has a limit of `2^21` vregs per function body in
VCode. This is a consequence of (performance-motivated) bitpacking of
`Operand`s into `u32`s in regalloc2.

As a result of this, it is possible to produce a function body that will
fail to compile by running out of vreg temporaries during lowering.
Currently, this results in a panic. Ideally, we would propagate the
error upward and return it programmatically.

This PR does that, with a "deferred error" mechanism. A cleaner solution
would be to properly thread the `Result` types through all layers of
lowering. However, that would require supporting `Result`s in ISLE, and
that is a deeper language-design and `islec`-hacking question that I
think we can tackle later if we continue to see motivating cases.

The deferral works by returning a valid but bogus `ValueReg`s to the
lowering rule, but storing the error and checking for it in the toplevel
lowering loop. (Note that we have to return a bogus `v0` rather than
`VReg::invalid()`, because the latter causes the `ValueRegs` to think
there is no register provided.)

This PR also includes a test at the Wasmtime level. Note that it takes
~22s to run on my (relatively fast) laptop, because it has to run until
it runs out of VRegs in a debug build of the compiler. We could remove
the test if we feel we're otherwise confident.

Thanks to Venkkatesh Sekar for reporting this issue! The integration
test uses one of the example inputs from the report.

* Review feedback.

* Handle alloc after a deferred error occurs.

* Add uncommitted fix for previous.

* Remove some usage of UnsafeCell in the adapter (#7117)

* Fix compile warnings in preview1 adapter

This isn't part of the "main build" so it's not inheriting the "deny
warnings" flag passed in CI, so fixup some warnings which have leaked
through over time.

* Remove some usage of `UnsafeCell` in the adapter

Move to abstractions such as `OnceCell` from the standard library as
well as `RefCell` to avoid some unsafety. This shouldn't have any
practical ramifications on the adapter, mostly just trying to help
over-time maintenance.

* Deny test-programs warnings in CI

* Remove `Table*Ext` traits for preview2 (#7118)

* wasi: Add typed helpers to `Table`

This will help cut down on the `Table*Ext` traits while retaining type
safety.

* Remove `TableNetworkExt` trait

* Remove `TableTcpSocketExt` trait

* Remove the `TableReaddirExt` trait

* Remove `TableFsExt` trait

This involed a fair bit of refactoring within the preview2-to-preview1
adapter to handle the new ownership of resources, but nothing major.

* Remove `TableStreamExt` trait

Additionally simplify some stream methods while I'm here.

* Remove `TablePollableExt` trait

* Fix tests

* Fix some more tests

* Use typed accessors for terminal-{input,output}

* Remove dead code in `Table`

* Temporarily disable wasi-nn on CI (#7126)

This appears to be failing, so temporarily disable it while the issues
are worked out.

* c-api: Expose host memory creation (#7115)

* c-api: Expose host memory creation

Signed-off-by: Tyler Rockwood <rockwood@redpanda.com>

* c-api: Review feedback

Signed-off-by: Tyler Rockwood <rockwood@redpanda.com>

* c-api: Fix invalid reference in docs

Signed-off-by: Tyler Rockwood <rockwood@redpanda.com>

* c-api: Fix doxygen documentation

Signed-off-by: Tyler Rockwood <rockwood@redpanda.com>

---------

Signed-off-by: Tyler Rockwood <rockwood@redpanda.com>

* riscv64: Refactor and enable optimizing sign extensions (#7121)

* riscv64: Remove some stray whitespace

* riscv64: Refactor int-to-float emission

Use ISLE matching to generate different instructions rather than having
a method deeper in the backend select which instruction to use.
Additionally remove the usage of the `normalize_fcvt_to_int` helper.

* riscv64: Refactor and simplify `lower_bmask`

Take a `Value` as input to simplify 128-bit handling and additionally
simplify the rules a bit with other helpers.

* riscv64: Refactor `normalize_cmp_value` and `truthy_to_reg`

Remove the normalization completely by folding it into callers and
refactor `truthy_to_reg` to perform sign extension internally in
addition to handling 128-vs-64

* riscv64: Remove 128-bit handling from extension helper

This is only needed in a small handful of spots which can pretty easily
handle the sign extension on their own, and this further simplifies sign
extension to purely dealing with a single register.

* riscv64: Remove destination type in extension

This is always `$I64`, so no need to pass it around.

* riscv64: Remove `ext_if_need` helper

Fold 128-bit handling into callers as necessary

* riscv64: Refactor `sext` to take a `Value`

This commit refactors the `sext` helper to take a single `Value` as an
argument rather than an `XReg Type` combo. This will enable, in future
commits, to pattern-match the structure of the input and skip the sign
extension if possible. For example an `icmp` result is already
sign-extended.

This involved moving some `lower_*` declarations directly into
`lower.isle`. Additionally a special-case of lowering `brif` was removed
which caused regressions in generated code due to new `and` instructions
being generated to sign-extend the result of an `icmp`. Once `zext` is
also migrated to this pattern this regression should be fixed, so this
should just be a temporary state of affairs.

* riscv64: Fold `lower_umulhi` helper into lower.isle

* riscv64: Fold `lower_uadd_overflow` into lower.isle

* riscv64: Fold `lower_popcnt` into `lower.isle`

This slightly pessmizes codegen when the `zbb` extension isn't available
due to a sign extension for smaller-than-64-bit types, but this is
possible to fix in the future if necessary and helps simplify this in
the meantime.

Additionally this lifts the special cases for `has_zbb` directly into
the lowering rule.

* riscv64: Refactor clz and cls

This commit refactors the lowering rules for these two CLIF instructions
to move the sext/zext operations to the top rather than having them
buried in lowering rules. This additionally moves them to `lower.isle`
instead of `inst.isle`. This did result in some minor duplication but
overall feels a bit better due to them being less coupled now.

* riscv64: Move `gen_bswap` to `lower.isle`

Additionally shuffle rules around to remove the dependence on `zext`
because when this zero-extension happens it's known that many of the
extension-related optimizations are not applicable. This does pessimize
code slightly but it's hopefully not too bad given the current size of
the bswap lowering.

* riscv64: Change `zext` to taking a `Value`

This is similar to a prior commit for `sext` and will enable future
optimizations to skip the zext entirely if the structure of the value
allows.

* riscv64: Fold `extend` helper into `zext` and `sext`

This commit folds the existing rules of the `extend` helper into the
corresponding functions of `zext` and `sext`. They are the only callers
of `extend` and this keeps the sign extension-related logic truly in one
final resting place now at this point. A new matcher
`val_already_extended` is additionally added to add more cases in a
follow-up commit to enable skipping sign-extension operations based on
the structure of the `Value` input.

* riscv64: Optimize some sign extensions

As the culmination of all prior commits, this commit adds a few cases
where a sign extension is not necessary in the backend and can be
skipped. For example extending a sign-extended value is not required
because the backend always extends to the full register width.
Additionally extending the result of an `icmp` is not necessary since
that always produces a 0 or 1 value at the full register width.

* Fix compatibility with Rust 1.70.0 (#7128)

This commit fixes a compatibility issue with Rust 1.70.0 on Windows
targets. Rust 1.71.0 stabilized `AsSocket for Arc<T>` which is used here
implicitly, so to compile successfully on 1.70.0, our current MSRV, a
slight code change is required.

Closes #7127

* riscv64: Add `Zcb` extension instructions (#7123)

* riscv64: Add `c.mul`

* riscv64: Add `c.not`

* riscv64: Add zbb and zba dependent compressed instructions

* riscv64: Add `Zcb` loads and stores

* riscv64: Restrict immediate encoding for halfword compressed stores and loads

* riscv64: Reverse imm bits for bytewise compressed loads and stores

* riscv64: Optimize `uadd_overflow_trap` lowering (#7132)

This commit removes the usage of `gen_icmp` in `uadd_overflow_trap`.
The comparison previously done with an explicit comparison is now
bundled directly into the conditional branch to go to the trap itself.

* riscv64: Add another pattern for loading constants (#7131)

Currently any 32-bit constant can be materialized without a load from a
constant pool on RISCV-64 but once constants start getting larger than
this they're always loaded from the constant pool. This commit adds
another special case for loading constants which appears to match what
LLVM does which is to consider materializing a smaller constant and than
shifting it left.

This is done by chopping off all trailing zeros from an immediate and
then testing if the immediate can be materialized as a 32-bit constant.
This means that the current constant loading sequence can optionally be
followed by a trailing `slli` instruction to shift the zeros back into
the constant. This namely means that loading `i64::MIN` (1 << 63) no
longer falls back to the constant pool.

* Refactor how `subscribe` works in WASI (#7130)

This commit refactors how the `Pollable` resource is created in WASI.
Previously this was created manually via two variants of a `Pollable`
enum but this was somewhat error prone for a few reasons:

* The more common representation, `TableEntry`, had boilerplate
  associated with it that had dynamic downcasts and such. This is now
  all hidden behind a more typed API.

* Callers had to remember to use `push_child_resource` which is easy to
  forget.

* The previous signature of the readiness check returned a `Result<()>`
  which made it accidentally easy for errors to be propagated into traps
  rather than being returned as a "last operation failed" error.

This commit replaces the previous API with a single `subscribe` function
which takes a `Resource<T>` and returns a `Resource<Pollable>`. The `T`
type must implement a new trait, `Subscribe`, which indicates the
asynchronous readiness check. This readiness check additionally returns
`()` so it's no longer possible to accidentally return errors (or trap).

This namely required refactoring the `HostOutputStream` trait and
implementations. The trait itself now has a `check_write` method
corresponding to to the same WASI function. The old `write_ready`
function is now a convenience helper combo between `ready` and
`check_write`. Concrete implementations are refactored to store errors
discovered during `ready` to get communicated through `check-write`
later on.

* Fix a regression with `wasmtime serve`, and enable the serve feature in tests (#7137)

* riscv64: Refactor implementation of `{u,s}{div,rem}` (#7136)

* riscv64: Refactor implementation of `{u,s}{div,rem}`

This commit's goal is to remove the usage of `gen_icmp` with
division-related instructions in the riscv64 backend. I've opted for a
slightly more verbose approach than what was present prior to make
sign/zero extensions more precise and additionally enable some
immediate-related optimizations.

Divison/remainder by an immediate will no longer emit checks to see if a
trap needs to be emitted. Instead only the raw `div`/`rem` instructions
are emitted. Additionally a few minor sign extensions are now avoided
such as the dividend in 32-bit remainders/division because only the
low 32-bits are inspected.

Finally, while I was here, I went ahead an added `has_m` guards to all
these lowering rules. The M extension is always required with riscv64
right now and won't work if it's turned off, but I figure it's not too
bad to add in terms of completeness for now.

As to the meat of the commit, the check for trapping value as part of
division now happens without `gen_icmp` but instead `gen_trapif` which
enables avoiding materializing the comparison's value into a register.

* Fix tests

* Async support in the C API (#7106)

* c-api: Add a feature for async

Signed-off-by: Tyler Rockwood <rockwood@redpanda.com>

* c-api: Add support for async config

Signed-off-by: Tyler Rockwood <rockwood@redpanda.com>

* c-api: Add support for calling async functions

Signed-off-by: Tyler Rockwood <rockwood@redpanda.com>

* c-api: Add ability to yield execution of Wasm in a store

Signed-off-by: Tyler Rockwood <rockwood@redpanda.com>

* c-api: Introduce wasmtime_linker_instantiate_async

Signed-off-by: Tyler Rockwood <rockwood@redpanda.com>

* c-api: Support defining async host functions

Signed-off-by: Tyler Rockwood <rockwood@redpanda.com>

* gitignore: ignore cmake cache for examples

Signed-off-by: Tyler Rockwood <rockwood@redpanda.com>

* examples: Add example of async API in C

Signed-off-by: Tyler Rockwood <rockwood@redpanda.com>

* c-api: Consolidate async functionality into a single place

Put all the async stuff in it's own header and own rust source file

Also remove the wasmtime_async_continuation_new function, users can just
allocate it directly.

Signed-off-by: Tyler Rockwood <rockwood@redpanda.com>

* c-api: Make async function safe

Signed-off-by: Tyler Rockwood <rockwood@redpanda.com>

* c-api: Remove wasmtime_call_future_get_results

Signed-off-by: Tyler Rockwood <rockwood@redpanda.com>

* c-api: Simplify CHostCallFuture

Move the result translation and hostcall_val_storage usage into an async
function

Signed-off-by: Tyler Rockwood <rockwood@redpanda.com>

* c-api: Simplify C continuation implementation

Remove the caller, which means that we don't need another struct for the
future implementation.

Signed-off-by: Tyler Rockwood <rockwood@redpanda.com>

* c-api: Improve async.h documentation

Signed-off-by: Tyler Rockwood <rockwood@redpanda.com>

* c-api: Cleanup from previous changes

Signed-off-by: Tyler Rockwood <rockwood@redpanda.com>

* examples: Fix example

Signed-off-by: Tyler Rockwood <rockwood@redpanda.com>

* c-api: Simplify continuation callback

This gives more duality with calling an async function and also means
that the implementation can pretty much mirror the sync version.

Signed-off-by: Tyler Rockwood <rockwood@redpanda.com>

* c-api: Fix async.h documentation

Signed-off-by: Tyler Rockwood <rockwood@redpanda.com>

* c-api: Fix documentation for async.h

Signed-off-by: Tyler Rockwood <rockwood@redpanda.com>

* c-api: Review feedback

Signed-off-by: Tyler Rockwood <rockwood@redpanda.com>

* examples: Downgrade async.cpp example to C++11

Signed-off-by: Tyler Rockwood <rockwood@redpanda.com>

* c-api: initialize continuation with a panic callback

Signed-off-by: Tyler Rockwood <rockwood@redpanda.com>

* prtest:full

Signed-off-by: Tyler Rockwood <rockwood@redpanda.com>

---------

Signed-off-by: Tyler Rockwood <rockwood@redpanda.com>

* feat(preview1): implement `poll_oneoff` (#7099)

Signed-off-by: Roman Volosatovs <rvolosatovs@riseup.net>

* Handle `lower_branch` consistently amongst backends (#7133)

* Handle `lower_branch` consistently amongst backends

This commit is a refactoring to consistently implement `lower_branch`
among Cranelift's backends. Previously each backend had its own means of
extracting labels and shuffling along information, and now there's
prelude methods for all backends to access and use. This changes a few
display impls but the actual meat of what's emitted shouldn't change
amongst the backends.

* Fix compile

* wasi-sockets: Implement initial listen backlog (#7034)

* Allow backlog size to be set before initial listen

* Move set_listen_backlog_size into example_body

* Format code

* Let cap_net_ext handle the default value.

* retrigger checks

* Got lost while pulling in changes from main.

* Fix a flaky preview2 test (#7138)

Found in CI for #7099 this commit updates the durations used in the
`resolves_immediately` and `never_resolves` tests. For immediately
resolving it should be ok to wait for a generous amount of time since
the timeout is only a protection against tests hanging. For
`never_resolves` it should be ok to wait a short amount of time and any
failure there will show up as a spurious failure.

* c-api: Support InstancePre (#7140)

* c-api: Support InstancePre

Signed-off-by: Tyler Rockwood <rockwood@redpanda.com>

* c-api: Consolidate all the instance_pre functionality

Signed-off-by: Tyler Rockwood <rockwood@redpanda.com>

* c-api: Add async instantiate support to pre instances

Signed-off-by: Tyler Rockwood <rockwood@redpanda.com>

* c-api: Add star to comment for doxygen

prtest:full

Signed-off-by: Tyler Rockwood <rockwood@redpanda.com>

---------

Signed-off-by: Tyler Rockwood <rockwood@redpanda.com>

* riscv64: Refactor and simplify some branches/fcmp  (#7142)

* riscv64: Add codegen tests for fcmp and fcmp branches

* riscv64: Refactor and simplify some branches/fcmp

This commit is aimed at simplifying the layers necessary to generate a
branch for a `brif` statement. Previously this involved a
`lower_cond_br` helper which bottomed out in emitting a `CondBr`
instruction. The intention here was to cut all that out and emit a
`CondBr` directly.

Along the way I've additionally taken the liberty of simplifying `fcmp`
as well. This moves the "prefer ordered compares" logic into `emit_fcmp`
so it can benefit the `fcmp` instruction as well. This additionally
trimmed some abstractions around branches which shouldn't be necessary
any longer.

* Resourcify the wasi-http interface (#7135)

* Fix race condition in `AsyncWriteStream` (#7144)

* Fix race condition in `AsyncWriteStream`

This commit fixes a synchronization issue with `AsyncWriteStream` where
the writer task would get wedged and not make any further progress. The
underlying reason is that `notify_waiters` was used which did not buffer
its notification, meaning that if a waiter wasn't actually waiting then
it would miss the notification and block forever on the next call to
`notified`. By using `notify_one` instead this will buffer up a single
notification for the other end for when it gets to waiting for work.

Additionally this removes creation of the `Notified` future
ahead-of-time as that's no longer necessary.

* Fix http bodies too

* riscv64: Update `replicated_{u,}imm5` to match `vconst` (#7141)

This switches the `extractor` to a `pure partial` constructor to be used
in `if-let` instead of the extraction position, which while not quite as
ergonomic does enable matching more constant values in more locations.

* Wasi input-stream: use same errors as output-stream (#7090)

* streams.wit: delete stream-status, rename write-error to stream-error, transform all input-stream methods

* preview2: use StreamError throughout input-stream

* preview2: passes cargo test

* preview1: fixes for input-stream stream-error.

* wasmtime-wasi-http: fixes for HostInputStream trait changes

* component adapter: fixes for input-stream changes

* test programs: fixes for input-stream

* component adapter: handle StreamError::Closed in fd_read

* sync wit definitions to wasi-http

* fix!!

* preview1: handle eof and intr properly

prtest:full

* Fix preview1 stdin reading

* Touch up stream documentation

---------

Co-authored-by: Alex Crichton <alex@alexcrichton.com>

* Implement the `wasi:sockets/ip-name-lookup` interface  (#7109)

* Implement the `wasi:sockets/ip-name-lookup` interface

This commit is an initial implementation of the new `ip-name-lookup`
interface from the `wasi-sockets` proposal. The intention is to get a
sketch of what an implementation would look like for the preview2
release later this year. Notable features of this implementation are:

* Name lookups are disabled by default and must be explicitly enabled in
  `WasiCtx`. This seems like a reasonable default for now while the full
  set of configuration around this is settled.

* I've added new "typed" methods to `preview2::Table` to avoid the need
  for extra helpers when using resources.

* A new `-Sallow-ip-name-lookup` option is added to control this on the
  CLI.

* Implementation-wise this uses the blocking resolution in the Rust
  standard library, built on `getaddrinfo`. This doesn't invoke
  `getaddrinfo` "raw", however, so information such as error details can
  be lost in translation. This will probably need to change in the
  future.

Closes #7070

* Validate the input domain name.

* Use a crate-level helper for `spawn_blocking`

---------

Co-authored-by: Dave Bakker <github@davebakker.io>

* riscv64: Further optimize constants loading (#7139)

* Further optimize RISCV constant loading

* Simplify existing rule

* Fix

* Bless missing tests

* Use right signedness for Imm20

* Refactor HTTP tests to be less flaky and more robust (#7143)

* Refactor HTTP tests to be less flaky and more robust

This commit refactors the wasi-http tests we have in this repository
with a few changes:

* Each test now uses a dedicated port for the test. The port 0 is bound
  and whatever the OS gives is used for the duration of the test. This
  prevents tests from possibly interfering with each other.

* Server spawning is abstracted behind a single `Server` type now which
  internally has http1/2 constructors.

* Timeouts for server shutdown are removed since they were failing in CI
  and are likely best handled for each test individually if necessary.

As a minor cleanup the `tokio` usage in guest programs was removed as it
was boilerplate in this case. This shouldn't affect the runtimes of
tests, however.

* Remove unused import

* Don't panic from worker thread

* Improve failure error message

prtest:full

* Fix some windows-specific issues

* Fix async tests

* Relax test assertion string

* [wasmtime-wasi] make StdinStream and StdoutStream public (#7150)

This allows embedders to implement those traits themselves rather than be
restricted to using the built-in implementations.

Fixes https://github.com/bytecodealliance/wasmtime/issues/7149

Signed-off-by: Joel Dice <joel.dice@fermyon.com>

* wasmtime: Handle common cli args in `wasmtime serve` (#7134)

* Refactor option handling in the `run` and `serve` commands

* Use a real thread to manage epoch increments

* Move module/component loading into src/common.rs

* Require wasi-http and component-model to not be disabled in serve

* riscv64: Consolidate conditional moves into one instruction  (#7145)

* riscv64: Add codegen tests for min/max

* riscv64: Remove unused `Type` payload from `Select`

* riscv64: Add codegen test for brif

* riscv64: Consolidate conditional moves into one instruction

This commit removes the `IntSelect` and `SelectReg` pseudo-instructions
from the riscv64 backend and consolidates them into the `Select`
instruction. Additionally the `Select` instruction is updated to subsume
the functionality of these two previous instructions. Namely `Select`
now operates with `ValueRegs` to handle i128 and additionally takes an
`IntegerCompare` as the condition for the conditional branch to use.

This commit touches a fair bit of the backend since conditional
selection of registers was used in quite a few places. The previous
`gen_select_*` functions are replaced with new typed equivalents of
`gen_select_{xreg,vreg,freg,regs}`. Furthermore new `cmp_*` helpers were
added to create `IntegerCompare` instructions which sort-of match
conditional branch instructions, or at least the pnemonics they use.

Finally since this affected the `select` CLIF instruction itself I went
ahead and did some refactoring there too. The `select` instruction
creates an `IntegerCompare` from its argument to use to generate the
appropriate register selection instruction. This is basically the same
thing that `brif` does and now both go through a new helper,
`lower_int_compare`, which takes a `Value` and produces an
`IntegerCompare` representing if that value is either true or false.
This enables folding an `icmp` or an `fcmp`, for example, directly into
a branching instruction.

* Fix a test

* Favor sign-extension in equality comparisons

* riscv64: Add support for min/max instructions (#7146)

* riscv64: Mark `fcmp` as already extended (#7151)

* Fix printing error information in sync http tests (#7153)

The usage of `?` accidentally caused the extra information to not get
printed, so avoid the use of `?` until the extra error info has been
attached.

* Bump Wasmtime to 15.0.0 (#7154)

Co-authored-by: Wasmtime Publish <wasmtime-publish@users.noreply.github.com>

* riscv64: Move `sextend` optimizations into `sext` (#7156)

This commit moves the optimizations from `sextend` for various shapes of
instructions into the `sext` helper. This enables these optimizations to
kick in when generating other instructions which require sign-extensions
such as `icmp` and `brif`.

This additionally fixes an issue introduced in #7121 where
uextend-of-sextend wasn't translated correctly and the outer `uextend`
was mistakenly omitted.

* Update release notes for 14.0.0 (#7158)

* Update release notes for 14.0.0

Wow quite a bit happened!

* Review comments

* Bump regalloc2 to 0.9.3 (#7160)

* Bump regalloc2 to 0.9.3

* Update filetests for the regalloc bump

* wasi-http: Use borrow syntax for borrowed resources (#7161)

* Remove `own` and `borrow` annotations from wasi-http

* Fix the tests, as borrows are now needed

* Don't delete headers that we're borrowing

* Add an error resource to WASI streams  (#7152)

* Change `bindgen!`'s trappable error to take an input type

This commit removes the generated type for `trappable_error_type` from
the `bindgen!` macro to allow users to provide a type instead. This
works similarly as before with new conversion functions generated in
traits which are used to convert the custom error into the ABI
representation of what a WIT world expects.

There are a few motivations for this commit:

* This enables reducing the number of errors in play between async/sync
  bindings by using the same error type there.

* This avoids an auto-generated type which is more difficult to inspect
  than one that's written down already and the source can more easily be
  inspected.

* This enables richer conversions using `self` (e.g. `self.table_mut()`)
  between error types rather than purely relying on `Into`. This is
  important for #7017 where an error is going to be inserted into the
  table as it gets converted.

* Fix tests

* Update WASI to use new trappable errors

This commit deals with the fallout of the previous commit for the WASI
preview2 implementation. The main changes here are:

* Bindgen-generated `Error` types no longer exist. These are replaced
  with `TrappableError<T>` where type aliases are used such as

  ```rust
  type FsError = TrappableError<wasi::filesystem::types::ErrorCode>;
  ```

* Type synonyms such as `FsResult<T>` are now added for more
  conveniently writing down fallible signatures.

* Some various error conversions are updated from converting to the old
  `Error` type to now instead directly into corresponding `ErrorCode` types.

* A number of cases where unknown errors were turned into traps now
  return bland error codes and log the error instead since these aren't
  fatal events.

* The `StreamError` type does not use `TrappableError` since it has
  other variants that it's concerned with such as a
  `LastOperationFailed` variant which has an `anyhow::Error` payload.

* Some minor preview1 issues were fixed such as trapping errors being
  turned into normal I/O errors by accident.

* Add an `error` resource to WASI streams

This commit adds a new `error` resource to the `wasi:io/streams`
interface. This `error` resource is returned as part of
`last-operation-failed` and serves as a means to discover through other
interfaces more granular type information than a generic string. This
error type has a new function in the `filesystem` interface, for
example, which enables getting filesystem-related error codes from I/O
performed on filesystem-originating streams. This is plumbed through to
the adapter as well to return more than `ERRNO_IO` from failed
read/write operations now too.

This is not super fancy just yet but is intended to be a vector through
which future additions can be done. The main thing this enables is to
avoid dropping errors on the floor in the host and enabling the guest to
discover further information about I/O errors on streams.

Closes #7017

* Update crates/wasi-http/wit/deps/io/streams.wit

Co-authored-by: Trevor Elliott <awesomelyawesome@gmail.com>

* Update wasi-http wit too

* Remove unnecessary clone

---------

Co-authored-by: Trevor Elliott <awesomelyawesome@gmail.com>

* add i128 support for iabs on x86_64 (#7166)

* fuzzgen: Enable iabs.i128 on fuzzgen (#7168)

* Add a whole lot of semicolons to WIT files (#7159)

To prepare this commit I've run `WIT_REQUIRE_SEMICOLONS=1 ./ci/run-tests.sh`
locally which configures `wit-parser` to generate an error for missing
semicolons in WIT files. This led me to add quite a few semicolons in
quite a few places in what is going to be the first of a few batches of
semicolons.

CI checks for this cannot be added just yet because the wasi-nn spec is
a submodule which needs to be updated with semicolons before this
repository can require semicolons. Nevertheless that doesn't stop us
from using semicolons in the meantime (yay gradual rollout of changes!)
so I figure this would be good to get in sooner rather than later.

* wasi-nn: reenable CI task (#7164)

The task was disabled due to a failing checksum. Since then, the
`install-openvino-action` has improved and now _almost_ has
cross-platform support. This change just restores the status quo in
expectation that #6895 will significantly improve the testing story.

prtest:full

* fix(wasmtime-cli): wrong linker in wasi http (#7167)

* chore: convert wasi http cli test to component

* fix(wasmtime-cli): wrong linker in wasi http

* fix(wasmtime-cli): fix multiple functions added to linker

We will allow shadowing in linker when using
WASI HTTP in order to prevent the following
error:
> import of `wasi:clocks/wall-clock` defined
twice

* chore: update to latest io streams wit definition

* chore: rename main function of core module

prtest:full

* fix(wasmtime-cli): add only http functions to linker

* chore: add test building http request and response

* chore: disable temporarily wasi http test

* docs(component): Fix typo in component translate (#7173)

* Require semicolons in WIT files in CI (#7174)

This commit updates CI to require semicolons in all WIT files and parsed
WIT documents. Some minor updates were required in existing WIT files
and the wasi-nn proposal was additionally updated to its latest version
with semicolons. The wasi-nn update brought some minor changes to the
WIT which required some minor changes here as well.

* cranelift: Consider functions with TLS values as non leaf (#7177)

* c-api: Fix type (#7178)

This should be wasmtime_instance_pre_t

Signed-off-by: Tyler Rockwood <rockwood@redpanda.com>

* winch(x64): Add support for table instructions (#7155)

* winch(x64): Add support for table instructions

This change adds support for the following table insructions:
`elem.drop`, `table.copy`, `table.set`, `table.get`, `table.fill`,
`table.grow`, `table.size`, `table.init`.

This change also introduces partial support for the `Ref` WebAssembly
type, more conretely the `Func` heap type, which means that all the
table instructions above, only work this WebAssembly type as of this
change.

Finally, this change is also a small follow up to the primitives
introduced in https://github.com/bytecodealliance/wasmtime/pull/7100,
more concretely:

* `FnCall::with_lib`: tracks the presence of a libcall and ensures that
  any result registers are freed right when the call is emitted.
* `MacroAssembler::table_elem_addr` returns an address rather than the
  value of the address, making it convenient for other use cases like
  `table.set`.

--

prtest:full

* chore: Make stack functions take impl IntoIterator<..>

* Update winch/codegen/src/codegen/call.rs

Co-authored-by: Trevor Elliott <awesomelyawesome@gmail.com>

* Remove a dangling `dbg!`

* Add comment on branching

---------

Co-authored-by: Trevor Elliott <awesomelyawesome@gmail.com>

* Skeleton and initial support for proof-carrying code. (#7165)

* WIP veriwasm 2.0

Co-Authored-By: Chris Fallin <cfallin@fastly.com>

* PCC: successfully parse some simple facts.

Co-authored-by: Nick Fitzgerald <fitzgen@gmail.com>

* PCC: plumb facts through VCode and add framework on LowerBackend to check them.

Co-authored-by: Nick Fitzgerald <fitzgen@gmail.com>

* PCC: code is carrying some proofs! Very simple test-case.

Co-authored-by: Nick Fitzgerald <fitzgen@gmail.com>

* PCC: add a `safe` flag for checked memory accesses.

* PCC: add pretty-printing of facts to CLIF output.

* PCC: misc. cleanups.

* PCC: lots of cleanup.

* Post-rebase fixups and some misc. fixes.

* Add serde traits to facts.

* PCC: add succeed and fail tests.

* Review feedback: rename `safe` memflag to `checked`.

* Review feedback.

---------

Co-authored-by: Nick Fitzgerald <fitzgen@gmail.com>

* Update syntax for `trappable_error_type` in `bindgen!` (#7170)

* Update syntax for `trappable_error_type` in `bindgen!`

Following through on a suggestion from #7152 this makes the syntax a bit
more consistent with the rest of `bindgen!` today. This additionally
updates the documentation on the `bindgen!` macro itself.

* Update parsing to afford for versions

* Support multiple versions in `bindgen!` (#7172)

This commit fixes a bug in the `bindgen!` macro where when faced with
multiple packages that differ only in version number invalid bindings
were generated. The fix here is to add version number information to
package module names if necessary in situations such as this. This
required some refactoring internally to have a single source of truth
for what the name of a module should be and avoid having it implicitly
calculated in two locations.

* riscv64: Improve `f{min,max}` (#7181)

* PCC: add semantics for core add/shift/extend/amode ops on AArch64. (#7180)

* PCC: add semantics for core add/shift/extend/amode ops on AArch64.

This PR adds verification of facts on values produced by adds, shifts,
and extends on AArch64, handling the various combination instructions
(adds with builtin extends or shifts, for example), and also adds
verification of all addressing modes, including those with builtin
extends and shifts.

It also splits the test suite into"succeed" and "fail" sets, and
provides cases that PCC should catch.

* Review feedback.

* cranelift: Group labels by FuncId instead of Name (#7183)

This prevents confusing label locations when there are multiple functions
with the same Name.

* wasi-sockets: Simplify & clarify TCP errors (#7120)

* Use Rustix::Errno to unify error code mapping.

* Clarify Connect failure state

* Allow accept() to return transient errors.

The original provision was added to align with preview3 streams that may only fail once. However, after discussing with Dan Gohman, we came to the conclusion that a stream of result<> could do the trick fine too.

Fixes: https://github.com/WebAssembly/wasi-sockets/issues/22

* Fold `ephemeral-ports-exhausted` into `address-in-use`

There is no cross-platform way to know the distinction between them

* Clarify `local-address` behavior on unbound socket

* Remove `concurrency-conflict` clutter,
and just document it to be always possible.

* Simplify state errors.

They were unnecessarily detailed and mostly have no standardized equivalent in POSIX, so wasi-libc will probably just map them all back into a single EOPNOTSUPP or EINVAL or ...

EISCONN/ENOTCONN can be derived in wasi-libc based on context and/or by checking `remote-address`. For example, `shutdown` can only be called on connected sockets, so if it returns `invalid-state` it can be unambiguously mapped to ENOTCONN.

* Document that connect may return ECONNABORTED

* Remove create-tcp/udp-socket not supported errors.

These stem from back when the entire wasi-sockets proposal was one big single thing. In this day and age, when an implementation doesn't want to support TCP and/or UDP, it can simply _not_ implement that interface, rather than returning an error at runtime.

* Simplify "not supported" and "invalid argument" error cases.

There is a myriad of reasons why an argument might be invalid or an operation might be not supported. But there is few cross platform consistency in which of those error cases result in which error codes.

The error codes that have been removed were fairly specific, but:
- Were still missing error cases. So additional error codes would have needed to be created.
- Implementations would have to bend over backwards to make it work cross platform, especially beyond just Win/Mac/Linux.
- Didn't all have an equivalent in POSIX, so they would map back into a generic EINVAL anyways.

* Move example_body out of lib.rs into its own test-case make room for other tests.

* Refactor TCP integration tests:
- Ad-hoc skeleton implementation of resources.
- Add blocking wrappers around async operations.

* Fix get/set_unicast_hop_limit on Linux

* Test TCP socket states

* Keep track of address family ourselves.

Because I need the family for input validation.
And the resulting code is more straightforward.

* Add more tests and make it work on Linux

* Fix Windows

* Simplify integration tests.

All platforms supported by wasmtime also support dualstack sockets.

* Test ipv6_only inheritence

* Test that socket options keep being respected, even if listen() has already been called

* cargo fmt

* Duplicate .wit changes to wasi-http

* prtest:full

* Fix BSD behavior of SO_SNDBUF/SO_RCVBUF

* fmt

* Fix type error

* Got lost during merge

* Implement listen backlog tests

* Manually inherit buffer size from listener on MacOS.

I suspect that these changes apply to any BSD platform, but I can't test that.

* Keep track of IPV6_V6ONLY ourselves.

- This provides cross-platform behaviour for `ipv6-only`
- This eliminates the syscall in `validate_address_family`

* Reject IPv4-compatible IPv6 addresses.

* Remove intermediate SystemError trait

* Fix ambiguous .into()'s

* Fix IPV6_UNICAST_HOPS inheritance on MacOS

* riscv64: Add GOT relocations for PIC code (#7184)

* riscv64: Use `CALL_PLT` relocation for all calls

* riscv64: Set the ELF RVC flag only when the C extension is enabled

* riscv64: Use PC Relative Relocations for all Calls

* riscv64: Add support for PIC symbols

* feat(wasi-sockets): implement UDP (#7148)

* feat(wasi-sockets): implement UDP

This is based on TCP implementation

Signed-off-by: Roman Volosatovs <rvolosatovs@riseup.net>

* refactor(wasi-sockets): simplify UDP implementation

This introduces quite a few changes compared to TCP, which should most probably be integrated there as well

Signed-off-by: Roman Volosatovs <rvolosatovs@riseup.net>

* feat(wasi-sockets): store UDP connect address in state

Signed-off-by: Roman Volosatovs <rvolosatovs@riseup.net>

* fix(wasi-sockets): avoid `shutdown` on `drop`

Signed-off-by: Roman Volosatovs <rvolosatovs@riseup.net>

* Remove explicit bind

* Simplify start_connect&finish_connect.
On UDP sockets, `connect` never blocks.

* Move UDP test to single file, similar to `tcp_sample_application.rs`

* Update UDP tests

---------

Signed-off-by: Roman Volosatovs <rvolosatovs@riseup.net>
Co-authored-by: Dave Bakker <github@davebakker.io>

* Ensure exports are properly namespaced (#7196)

Continuation of #7172 where imports and export _definitions_ are
properly namespaced by version by the usage of exports was not.

* add i128 support for iabs on aarch64 (#7185)

* Reapply #7150 (#7197)

Signed-off-by: Ryan Levick <ryan.levick@fermyon.com>

* riscv64: Optimize `gen_bmask` slightly (#7200)

If the input operand is an `icmp` or an `fcmp` there's no need to use
`snez` since the output value is already guaranteed to be zero or one.

* Refactor the test-programs test suite (#7182)

* Refactor the test-programs test suite

This commit is a large refactoring that reorganizes `test-programs` and
how we tests wasms in Wasmtime. Often writing tests requires complicated
interactions with the guest which can't be done via hand-written `*.wat`
syntax and requires a compiler to get engaged. For this purpose Wasmtime
currently has the `crates/test-programs/*` test suite which builds files
from source and then runs the tests. This has been somewhat cumbersome
in the past though and it's not been easy to extend this over time, so
this commit attempts to address this.

The scheme implemented in this PR looks like:

* All wasm test programs live in `crates/test-programs/src/bin/*.rs`.
  All of them, no exceptions.

* Wasm tests have shared support located at
  `crates/test-programs/src/lib.rs` and its submodules, such as bindings
  generation for WASI.

* Wasm tests are built by a new `crates/test-programs/artifacts` crate.
  This crate compiles modules and additionally creates components for
  all test programs. The crate itself only records the path to these
  outputs and a small amount of testing support, but otherwise doesn't
  interact with `wasmtime`-the-crate itself.

* All tests in `crates/test-programs/tests/*.rs` have moved. For example
  wasi-http tests now live at `crates/wasi-http/tests/*.rs`. Legacy
  tests of wasi-common now live at `crates/wasi-common/tests/*.rs`.
  Modern tests for preview2 live at `crates/wasi/tests/*.rs`.

* Wasm tests are bucketed based on their filename prefix. For example
  `preview1_*` is tested in wasi-common and wasmtime-wasi. The
  `preview2_*` prefix is only tested with wasmtime-wasi, however.

* A new `cli_*` prefix is used to execute tests as part of
  `tests/all/main.rs`. This is a new submodule in
  `tests/all/cli_tests.rs` which executes these components on the
  command line. Many old "command" tests were migrated here.

* Helper macros are generated to assert that a test suite is run in its
  entirety. This way if a `preview1_*` test is added it's asserted to
  get added to both wasi-common and wasmtime-wasi in the various modes
  they run tests.

Overall this moved a number of tests around and refactored some edges of
the tests, but this should not lose any tests (except one that wasn't
actually testing anything). Additionally the hope is that it's much
easier to add tests in the future. The process is to add a new file in
`crates/test-programs/src/bin/*.rs` named appropriately. For example a
preview2 executable is `preview2_*` and a CLI tests is `cli_*`. When
building the test suite an error is generated in the appropriate module
then of "please write a test here", and then a test is written in the
same manner as the other tests in the module.

* Remove no-longer-needed fetches

prtest:full

* I'm worried wasi is running low on semicolons

* Add the WASI target in all CI actions

* Add unknown-unknown target on all CI builders too

* Fix building test artifacts under miri

Need to avoid wrappers for these cross-compiled targets

* Break circular dependency for packaging

Don't use the workspace dep for `wasmtime-wasi` since it injects a
version, instead use a `path = '..'` dependency to fool Cargo into
dropping the dependency during the package phase.

* Fix some merge conflicts with tests

* Fix rebase for new tests

* Remove stray comment

* Fix some flaky tests

* Fix network tests in synchronous mode

This commit is an attempt to fix some networking tests in synchronous
mode in our test suite. Currently networking tests do…
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
cranelift:area:aarch64 Issues related to AArch64 backend. cranelift:area:machinst Issues related to instruction selection and the new MachInst backend. cranelift:area:x64 Issues related to x64 codegen cranelift:meta Everything related to the meta-language. cranelift Issues related to the Cranelift code generator wasmtime:api Related to the API of the `wasmtime` crate itself
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants