Skip to content

Commit

Permalink
Refactor the test-programs test suite (#7182)
Browse files Browse the repository at this point in the history
* Refactor the test-programs test suite

This commit is a large refactoring that reorganizes `test-programs` and
how we tests wasms in Wasmtime. Often writing tests requires complicated
interactions with the guest which can't be done via hand-written `*.wat`
syntax and requires a compiler to get engaged. For this purpose Wasmtime
currently has the `crates/test-programs/*` test suite which builds files
from source and then runs the tests. This has been somewhat cumbersome
in the past though and it's not been easy to extend this over time, so
this commit attempts to address this.

The scheme implemented in this PR looks like:

* All wasm test programs live in `crates/test-programs/src/bin/*.rs`.
  All of them, no exceptions.

* Wasm tests have shared support located at
  `crates/test-programs/src/lib.rs` and its submodules, such as bindings
  generation for WASI.

* Wasm tests are built by a new `crates/test-programs/artifacts` crate.
  This crate compiles modules and additionally creates components for
  all test programs. The crate itself only records the path to these
  outputs and a small amount of testing support, but otherwise doesn't
  interact with `wasmtime`-the-crate itself.

* All tests in `crates/test-programs/tests/*.rs` have moved. For example
  wasi-http tests now live at `crates/wasi-http/tests/*.rs`. Legacy
  tests of wasi-common now live at `crates/wasi-common/tests/*.rs`.
  Modern tests for preview2 live at `crates/wasi/tests/*.rs`.

* Wasm tests are bucketed based on their filename prefix. For example
  `preview1_*` is tested in wasi-common and wasmtime-wasi. The
  `preview2_*` prefix is only tested with wasmtime-wasi, however.

* A new `cli_*` prefix is used to execute tests as part of
  `tests/all/main.rs`. This is a new submodule in
  `tests/all/cli_tests.rs` which executes these components on the
  command line. Many old "command" tests were migrated here.

* Helper macros are generated to assert that a test suite is run in its
  entirety. This way if a `preview1_*` test is added it's asserted to
  get added to both wasi-common and wasmtime-wasi in the various modes
  they run tests.

Overall this moved a number of tests around and refactored some edges of
the tests, but this should not lose any tests (except one that wasn't
actually testing anything). Additionally the hope is that it's much
easier to add tests in the future. The process is to add a new file in
`crates/test-programs/src/bin/*.rs` named appropriately. For example a
preview2 executable is `preview2_*` and a CLI tests is `cli_*`. When
building the test suite an error is generated in the appropriate module
then of "please write a test here", and then a test is written in the
same manner as the other tests in the module.

* Remove no-longer-needed fetches

prtest:full

* I'm worried wasi is running low on semicolons

* Add the WASI target in all CI actions

* Add unknown-unknown target on all CI builders too

* Fix building test artifacts under miri

Need to avoid wrappers for these cross-compiled targets

* Break circular dependency for packaging

Don't use the workspace dep for `wasmtime-wasi` since it injects a
version, instead use a `path = '..'` dependency to fool Cargo into
dropping the dependency during the package phase.

* Fix some merge conflicts with tests

* Fix rebase for new tests

* Remove stray comment

* Fix some flaky tests

* Fix network tests in synchronous mode

This commit is an attempt to fix some networking tests in synchronous
mode in our test suite. Currently networking tests don't actually run in
synchronous mode on CI which is why no failures have been surfaced yet,
but the refactoring in #7182 is going to start doing this.

Currently the `udp_sample_application.rs` test blocks infinitely in
synchronous mode for me locally, most of the time. This appears to be an
interaction between how Tokio handles readiness and how we're
entering the event loop. We're effectively entering the Tokio event loop
with a future that's always ready which ends up starving Tokio of
otherwise performing its background work such as updating flags for
readiness of reading/writing.

The fix here is to add a yield at the start of an `in_tokio` block which
is used in synchronous mode. This is a kludge fix but the intention is
to enable Tokio to have a chance to update readiness flags and process
events from epoll/kqueue/etc.

An additional fix to this issue is WebAssembly/wasi-sockets#64 where the
test is waiting on `READABLE` or `WRITABLE`, but in this specific case
it should only wait on `READABLE`. If it waited on just this then that
would also fix this issue. Nevertheless having a `yield_now` is expected
to have little-to-no overhead and otherwise fix this edge case of an
always-ready future.

* Fix passing empty arguments on the CLI

* Add another blocking accept

* Update crates/test-programs/src/bin/api_proxy.rs

Co-authored-by: Trevor Elliott <awesomelyawesome@gmail.com>

---------

Co-authored-by: Trevor Elliott <awesomelyawesome@gmail.com>
  • Loading branch information
alexcrichton and elliottt authored Oct 9, 2023
1 parent 23b75f6 commit f4be360
Show file tree
Hide file tree
Showing 143 changed files with 2,647 additions and 3,392 deletions.
4 changes: 4 additions & 0 deletions .github/actions/install-rust/action.yml
Original file line number Diff line number Diff line change
Expand Up @@ -65,6 +65,10 @@ runs:
shell: bash
run: echo WIT_REQUIRE_SEMICOLONS=1 >> "$GITHUB_ENV"

- name: Install the WASI target
shell: bash
run: rustup target add wasm32-wasi wasm32-unknown-unknown

- name: Choose registry cache key
shell: bash
# Update the registry index cache at most once per day. actions/cache
Expand Down
2 changes: 0 additions & 2 deletions .github/workflows/main.yml
Original file line number Diff line number Diff line change
Expand Up @@ -396,8 +396,6 @@ jobs:
if: matrix.target == 'x86_64-pc-windows-gnu'

- run: cargo fetch --locked
- run: cargo fetch --locked --manifest-path crates/test-programs/wasi-tests/Cargo.toml
- run: cargo fetch --locked --manifest-path crates/test-programs/wasi-http-tests/Cargo.toml

- uses: actions/cache@v3
with:
Expand Down
91 changes: 22 additions & 69 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

13 changes: 3 additions & 10 deletions Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -64,7 +64,6 @@ env_logger = { workspace = true }
log = { workspace = true }
filecheck = { workspace = true }
tempfile = { workspace = true }
test-programs = { path = "crates/test-programs" }
wasmtime-runtime = { workspace = true }
tokio = { workspace = true, features = ["rt", "time", "macros", "rt-multi-thread"] }
wast = { workspace = true }
Expand All @@ -83,6 +82,7 @@ libc = { workspace = true }
serde = { workspace = true }
serde_json = { workspace = true }
walkdir = { workspace = true }
test-programs-artifacts = { workspace = true }

[target.'cfg(windows)'.dev-dependencies]
windows-sys = { workspace = true, features = ["Win32_System_Memory"] }
Expand All @@ -102,16 +102,8 @@ members = [
"cranelift/serde",
"crates/bench-api",
"crates/c-api",
"crates/cli-flags",
"crates/environ/fuzz",
"crates/jit-icache-coherence",
"crates/test-programs/wasi-tests",
"crates/test-programs/wasi-http-tests",
"crates/test-programs/wasi-http-proxy-tests",
"crates/test-programs/wasi-sockets-tests",
"crates/test-programs/command-tests",
"crates/test-programs/reactor-tests",
"crates/wmemcheck",
"crates/test-programs",
"crates/wasi-preview1-component-adapter",
"crates/wasi-preview1-component-adapter/verify",
"crates/winch",
Expand Down Expand Up @@ -170,6 +162,7 @@ wasi-cap-std-sync = { path = "crates/wasi-common/cap-std-sync", version = "=15.0
wasmtime-fuzzing = { path = "crates/fuzzing" }
wasmtime-jit-icache-coherence = { path = "crates/jit-icache-coherence", version = "=15.0.0" }
wasmtime-wit-bindgen = { path = "crates/wit-bindgen", version = "=15.0.0" }
test-programs-artifacts = { path = 'crates/test-programs/artifacts' }

cranelift-wasm = { path = "cranelift/wasm", version = "0.102.0" }
cranelift-codegen = { path = "cranelift/codegen", version = "0.102.0" }
Expand Down
7 changes: 1 addition & 6 deletions ci/run-tests.sh
Original file line number Diff line number Diff line change
@@ -1,15 +1,10 @@
#!/bin/bash

cargo test \
--features "test-programs/test_programs" \
--features wasi-threads \
--features wasi-http \
--features component-model \
--features serve \
--workspace \
--exclude 'wasmtime-wasi-*' \
--exclude wasi-tests \
--exclude wasi-http-tests \
--exclude command-tests \
--exclude reactor-tests \
--exclude test-programs \
$@
41 changes: 4 additions & 37 deletions crates/test-programs/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -2,46 +2,13 @@
name = "test-programs"
version = "0.0.0"
authors = ["The Wasmtime Project Developers"]
readme = "README.md"
edition.workspace = true
publish = false
license = "Apache-2.0 WITH LLVM-exception"

[build-dependencies]
cfg-if = { workspace = true }
cargo_metadata = "0.15.3"
wit-component = { workspace = true }
heck = { workspace = true }

[dependencies]
anyhow = { workspace = true }
bytes = { workspace = true }
http = { version = "0.2.9" }
http-body = "1.0.0-rc.2"
http-body-util = "0.1.0-rc.2"
hyper = { version = "1.0.0-rc.3", features = ["full"] }
tokio = { workspace = true, features = ["net", "rt-multi-thread", "macros"] }
tracing = { workspace = true }

[dev-dependencies]
anyhow = { workspace = true }
tempfile = { workspace = true }
test-log = { workspace = true }
tracing = { workspace = true }
tracing-subscriber = { workspace = true }
lazy_static = "1"
wasmtime = { workspace = true, features = ['cranelift', 'component-model'] }

wasi-common = { workspace = true }
wasi-cap-std-sync = { workspace = true }
wasmtime-wasi = { workspace = true, default-features = true, features = [
"tokio",
] }
cap-std = { workspace = true }
cap-rand = { workspace = true }
tokio = { workspace = true, features = ["rt-multi-thread", "macros"] }

wasmtime-wasi-http = { workspace = true, features = ["sync"] }

[features]
test_programs = []
wasi = "0.11.0"
wit-bindgen = { workspace = true, features = ['default'] }
libc = { workspace = true }
getrandom = "0.2.9"
7 changes: 0 additions & 7 deletions crates/test-programs/README.md

This file was deleted.

12 changes: 12 additions & 0 deletions crates/test-programs/artifacts/Cargo.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
[package]
name = "test-programs-artifacts"
version = "0.0.0"
authors = ["The Wasmtime Project Developers"]
edition.workspace = true
publish = false
license = "Apache-2.0 WITH LLVM-exception"

[build-dependencies]
heck = { workspace = true }
wit-component = { workspace = true }
cargo_metadata = "0.15.3"
Loading

0 comments on commit f4be360

Please sign in to comment.