components: Implement the ability to call component exports (#4039)

* components: Implement the ability to call component exports

This commit is an implementation of the typed method of calling
component exports. This is intended to represent the most efficient way
of calling a component in Wasmtime, similar to what `TypedFunc`
represents today for core wasm.

Internally this contains all the traits and implementations necessary to
invoke component exports with any type signature (e.g. arbitrary
parameters and/or results). The expectation is that for results we'll
reuse all of this infrastructure except in reverse (arguments and
results will be swapped when defining imports).

Some features of this implementation are:

* Arbitrary type hierarchies are supported
* The Rust-standard `Option`, `Result`, `String`, `Vec<T>`, and tuple
  types all map down to the corresponding type in the component model.
* Basic utf-16 string support is implemented as proof-of-concept to show
  what handling might look like. This will need further testing and
  benchmarking.
* Arguments can be behind "smart pointers", so for example
  `&Rc<Arc<[u8]>>` corresponds to `list<u8>` in interface types.
* Bulk copies from linear memory never happen unless explicitly
  instructed to do so.

The goal of this commit is to create the ability to actually invoke wasm
components. This represents what is expected to be the performance
threshold for these calls where it ideally should be optimal how
WebAssembly is invoked. One major missing piece of this is a `#[derive]`
of some sort to generate Rust types for arbitrary `*.wit` types such as
custom records, variants, flags, unions, etc. The current trait impls
for tuples and `Result<T, E>` are expected to have fleshed out most of
what such a derive would look like.

There are some downsides and missing pieces to this commit and method of
calling components, however, such as:

* Passing `&[u8]` to WebAssembly is currently not optimal. Ideally this
  compiles down to a `memcpy`-equivalent somewhere but that currently
  doesn't happen due to all the bounds checks of copying data into
  memory. I have been unsuccessful so far at getting these bounds checks
  to be removed.
* There is no finalization at this time (the "post return" functionality
  in the canonical ABI). Implementing this should be relatively
  straightforward but at this time requires `wasmparser` changes to
  catch up with the current canonical ABI.
* There is no guarantee that results of a wasm function will be
  validated. As results are consumed they are validated but this means
  that if function returns an invalid string which the host doesn't look
  at then no trap will be generated. This is probably not the intended
  semantics of hosts in the component model.
* At this time there's no support for memory64 memories, just a bunch of
  `FIXME`s to get around to. It's expected that this won't be too
  onerous, however. Some extra care will need to ensure that the various
  methods related to size/alignment all optimize to the same thing they
  do today (e.g. constants).
* The return value of a typed component function is either `T` or
  `Value<T>`, and it depends on the ABI details of `T` and whether it
  takes up more than one return value slot or not. This is an
  ABI-implementation detail which is being forced through to the API
  layer which is pretty unfortunate. For example if you say the return
  value of a function is `(u8, u32)` then it's a runtime type-checking
  error. I don't know of a great way to solve this at this time.

Overall I'm feeling optimistic about this trajectory of implementing
value lifting/lowering in Wasmtime. While there are a number of
downsides none seem completely insurmountable. There's naturally still a
good deal of work with the component model but this should be a
significant step up towards implementing and testing the component model.

* Review comments

* Write tests for calling functions

This commit adds a new test file for actually executing functions and
testing their results. This is not written as a `*.wast` test yet since
it's not 100% clear if that's the best way to do that for now (given
that dynamic signatures aren't supported yet). The tests themselves
could all largely be translated to `*.wast` testing in the future,
though, if supported.

Along the way a number of minor issues were fixed with lowerings with
the bugs exposed here.

* Fix an endian mistake

* Fix a typo and the `memory.fill` instruction

crates/environ/src/component/info.rs

@@ -137,15 +137,24 @@ pub struct LiftedFunction {
 }
 
 /// Canonical ABI options associated with a lifted function.
-#[derive(Default, Debug, Clone, Serialize, Deserialize)]
+#[derive(Debug, Clone, Serialize, Deserialize)]
 pub struct CanonicalOptions {
-    /// The optionally-specified encoding used for strings.
-    pub string_encoding: Option<StringEncoding>,
+    /// The encoding used for strings.
+    pub string_encoding: StringEncoding,
     /// Representation of the `into` option where intrinsics are peeled out and
     /// identified from an instance.
     pub intrinsics: Option<Intrinsics>,
 }
 
+impl Default for CanonicalOptions {
+    fn default() -> CanonicalOptions {
+        CanonicalOptions {
+            string_encoding: StringEncoding::Utf8,
+            intrinsics: None,
+        }
+    }
+}
+
 /// Possible encodings of strings within the component model.
 #[derive(Debug, Clone, Copy, Serialize, Deserialize)]
 #[allow(missing_docs)]

crates/environ/src/component/translate.rs

@@ -650,13 +650,13 @@ impl<'a, 'data> Translator<'a, 'data> {
         for opt in opts {
             match opt {
                 wasmparser::CanonicalOption::UTF8 => {
-                    ret.string_encoding = Some(StringEncoding::Utf8);
+                    ret.string_encoding = StringEncoding::Utf8;
                 }
                 wasmparser::CanonicalOption::UTF16 => {
-                    ret.string_encoding = Some(StringEncoding::Utf16);
+                    ret.string_encoding = StringEncoding::Utf16;
                 }
                 wasmparser::CanonicalOption::CompactUTF16 => {
-                    ret.string_encoding = Some(StringEncoding::CompactUtf16);
+                    ret.string_encoding = StringEncoding::CompactUtf16;
                 }
                 wasmparser::CanonicalOption::Into(instance) => {
                     let instance = InstanceIndex::from_u32(*instance);

crates/wasmtime/src/component/func.rs

@@ -1,20 +1,25 @@
 use crate::component::instance::lookup;
 use crate::store::{StoreOpaque, Stored};
+use crate::{AsContext, StoreContextMut};
+use anyhow::{bail, Context, Result};
+use std::convert::TryFrom;
 use std::sync::Arc;
 use wasmtime_environ::component::{
     ComponentTypes, FuncTypeIndex, LiftedFunction, RuntimeInstanceIndex, StringEncoding,
 };
 use wasmtime_environ::PrimaryMap;
 use wasmtime_runtime::{Export, ExportFunction, ExportMemory, VMTrampoline};
 
+mod typed;
+pub use self::typed::*;
+
 /// A WebAssembly component function.
 //
 // FIXME: write more docs here
 #[derive(Copy, Clone, Debug)]
 pub struct Func(Stored<FuncData>);
 
 #[doc(hidden)]
-#[allow(dead_code)] // FIXME: remove this when fields are actually used
 pub struct FuncData {
     trampoline: VMTrampoline,
     export: ExportFunction,
@@ -23,18 +28,15 @@ pub struct FuncData {
     options: Options,
 }
 
-#[derive(Clone)]
-#[allow(dead_code)] // FIXME: remove this when fields are actually used
 pub(crate) struct Options {
-    string_encoding: Option<StringEncoding>,
+    string_encoding: StringEncoding,
     intrinsics: Option<Intrinsics>,
 }
 
-#[derive(Clone)]
-#[allow(dead_code)] // FIXME: remove this when fields are actually used
 struct Intrinsics {
     memory: ExportMemory,
     realloc: ExportFunction,
+    #[allow(dead_code)] // FIXME: remove this when actually used
     free: ExportFunction,
 }
 
@@ -80,4 +82,179 @@ impl Func {
             types: types.clone(),
         }))
     }
+
+    /// Attempt to cast this [`Func`] to a statically typed [`TypedFunc`] with
+    /// the provided `Params` and `Return`.
+    ///
+    /// This function will perform a type-check at runtime that the [`Func`]
+    /// takes `Params` as parameters and returns `Return`. If the type-check
+    /// passes then a [`TypedFunc`] will be returned which can be used to invoke
+    /// the function in an efficient, statically-typed, and ergonomic manner.
+    ///
+    /// The `Params` type parameter here is a tuple of the parameters to the
+    /// function. A function which takes no arguments should use `()`, a
+    /// function with one argument should use `(T,)`, etc.
+    ///
+    /// The `Return` type parameter is the return value of this function. A
+    /// return value of `()` means that there's no return (similar to a Rust
+    /// unit return) and otherwise a type `T` can be specified.
+    ///
+    /// Types specified here are mainly those that implement the
+    /// [`ComponentValue`] trait. This trait is implemented for built-in types
+    /// to Rust such as integer primitives, floats, `Option<T>`, `Result<T, E>`,
+    /// strings, and `Vec<T>`. As parameters you'll be passing native Rust
+    /// types.
+    ///
+    /// For the `Return` type parameter many types need to be wrapped in a
+    /// [`Value<T>`]. For example functions which return a string should use the
+    /// `Return` type parameter as `Value<String>` instead of a bare `String`.
+    /// The usage of [`Value`] indicates that a type is stored in linear memory.
+    //
+    // FIXME: Having to remember when to use `Value<T>` vs `T` is going to trip
+    // people up using this API. It's not clear, though, how to fix that.
+    ///
+    /// # Errors
+    ///
+    /// If the function does not actually take `Params` as its parameters or
+    /// return `Return` then an error will be returned.
+    ///
+    /// # Panics
+    ///
+    /// This function will panic if `self` is not owned by the `store`
+    /// specified.
+    ///
+    /// # Examples
+    ///
+    /// Calling a function which takes no parameters and has no return value:
+    ///
+    /// ```
+    /// # use wasmtime::component::Func;
+    /// # use wasmtime::Store;
+    /// # fn foo(func: &Func, store: &mut Store<()>) -> anyhow::Result<()> {
+    /// let typed = func.typed::<(), (), _>(&store)?;
+    /// typed.call(store, ())?;
+    /// # Ok(())
+    /// # }
+    /// ```
+    ///
+    /// Calling a function which takes one string parameter and returns a
+    /// string:
+    ///
+    /// ```
+    /// # use wasmtime::component::{Func, Value};
+    /// # use wasmtime::Store;
+    /// # fn foo(func: &Func, mut store: Store<()>) -> anyhow::Result<()> {
+    /// let typed = func.typed::<(&str,), Value<String>, _>(&store)?;
+    /// let ret = typed.call(&mut store, ("Hello, ",))?;
+    /// let ret = ret.cursor(&store);
+    /// println!("returned string was: {}", ret.to_str()?);
+    /// # Ok(())
+    /// # }
+    /// ```
+    ///
+    /// Calling a function which takes multiple parameters and returns a boolean:
+    ///
+    /// ```
+    /// # use wasmtime::component::Func;
+    /// # use wasmtime::Store;
+    /// # fn foo(func: &Func, mut store: Store<()>) -> anyhow::Result<()> {
+    /// let typed = func.typed::<(u32, Option<&str>, &[u8]), bool, _>(&store)?;
+    /// let ok: bool = typed.call(&mut store, (1, Some("hello"), b"bytes!"))?;
+    /// println!("return value was: {ok}");
+    /// # Ok(())
+    /// # }
+    /// ```
+    pub fn typed<Params, Return, S>(&self, store: S) -> Result<TypedFunc<Params, Return>>
+    where
+        Params: ComponentParams,
+        Return: ComponentReturn,
+        S: AsContext,
+    {
+        self.typecheck::<Params, Return>(store.as_context().0)?;
+        unsafe { Ok(TypedFunc::new_unchecked(*self)) }
+    }
+
+    fn typecheck<Params, Return>(&self, store: &StoreOpaque) -> Result<()>
+    where
+        Params: ComponentParams,
+        Return: ComponentReturn,
+    {
+        let data = &store[self.0];
+        let ty = &data.types[data.ty];
+
+        Params::typecheck(&ty.params, &data.types).context("type mismatch with parameters")?;
+        Return::typecheck(&ty.result, &data.types).context("type mismatch with result")?;
+
+        Ok(())
+    }
+
+    fn realloc<'a, T>(
+        &self,
+        store: &'a mut StoreContextMut<'_, T>,
+        old: usize,
+        old_size: usize,
+        old_align: u32,
+        new_size: usize,
+    ) -> Result<(&'a mut [u8], usize)> {
+        let (realloc, memory) = match &store.0[self.0].options.intrinsics {
+            Some(Intrinsics {
+                memory, realloc, ..
+            }) => (realloc.clone(), memory.clone()),
+            None => unreachable!(),
+        };
+
+        // Invoke the wasm malloc function using its raw and statically known
+        // signature.
+        let result = unsafe {
+            // FIXME: needs memory64 support
+            assert!(!memory.memory.memory.memory64);
+            usize::try_from(crate::TypedFunc::<(u32, u32, u32, u32), u32>::call_raw(
+                store,
+                realloc.anyfunc,
+                (
+                    u32::try_from(old)?,
+                    u32::try_from(old_size)?,
+                    old_align,
+                    u32::try_from(new_size)?,
+                ),
+            )?)?
+        };
+
+        let memory = self.memory_mut(store.0);
+
+        let result_slice = match memory.get_mut(result..).and_then(|s| s.get_mut(..new_size)) {
+            Some(end) => end,
+            None => bail!("realloc return: beyond end of memory"),
+        };
+
+        Ok((result_slice, result))
+    }
+
+    /// Asserts that this function has an associated memory attached to it and
+    /// then returns the slice of memory tied to the lifetime of the provided
+    /// store.
+    fn memory<'a>(&self, store: &'a StoreOpaque) -> &'a [u8] {
+        let memory = match &store[self.0].options.intrinsics {
+            Some(Intrinsics { memory, .. }) => memory,
+            None => unreachable!(),
+        };
+
+        unsafe {
+            let memory = &*memory.definition;
+            std::slice::from_raw_parts(memory.base, memory.current_length)
+        }
+    }
+
+    /// Same as above, just `_mut`
+    fn memory_mut<'a>(&self, store: &'a mut StoreOpaque) -> &'a mut [u8] {
+        let memory = match &store[self.0].options.intrinsics {
+            Some(Intrinsics { memory, .. }) => memory.clone(),
+            None => unreachable!(),
+        };
+
+        unsafe {
+            let memory = &*memory.definition;
+            std::slice::from_raw_parts_mut(memory.base, memory.current_length)
+        }
+    }
 }