Skip to content

Commit

Permalink
winch: Adding support for integration tests (bytecodealliance#5588)
Browse files Browse the repository at this point in the history
* Adding in the foundations for Winch `filetests`

This commit adds two new crates into the Winch workspace:
`filetests` and `test-macros`. The intent is to mimic the
structure of Cranelift `filetests`, but in a simpler way.

* Updates to documentation

This commits adds a high level document to outline how to test Winch
through the `winch-tools` utility. It also updates some inline
documentation which gets propagated to the CLI.

* Updating test-macro to use a glob instead of only a flat directory
  • Loading branch information
KevinRizzoTO authored Jan 19, 2023
1 parent 7cea73a commit da03ff4
Show file tree
Hide file tree
Showing 16 changed files with 586 additions and 135 deletions.
36 changes: 36 additions & 0 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

6 changes: 6 additions & 0 deletions Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -156,6 +156,8 @@ cranelift-bforest = { path = "cranelift/bforest", version = "0.93.0" }
cranelift = { path = "cranelift/umbrella", version = "0.93.0" }

winch-codegen = { path = "winch/codegen", version = "=0.4.0" }
winch-filetests = { path = "winch/filetests" }
winch-test-macros = { path = "winch/test-macros" }

target-lexicon = { version = "0.12.3", default-features = false, features = ["std"] }
anyhow = "1.0.22"
Expand Down Expand Up @@ -186,6 +188,10 @@ bitflags = "1.2"
thiserror = "1.0.15"
async-trait = "0.1.42"
heck = "0.4"
similar = "2.1.0"
toml = "0.5.9"
serde = "1.0.94"
glob = "0.3.0"

[features]
default = [
Expand Down
3 changes: 3 additions & 0 deletions cranelift/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,9 @@ thiserror = { workspace = true }
walkdir = "2.2"
anyhow = { workspace = true }
clap = { workspace = true }
similar = { workspace = true }
toml = { workspace = true }
serde = { workspace = true }

[features]
default = ["disas", "wasm", "cranelift-codegen/all-arch", "cranelift-codegen/trace-log", "souper-harvest"]
Expand Down
6 changes: 3 additions & 3 deletions cranelift/filetests/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -26,10 +26,10 @@ num_cpus = "1.8.0"
target-lexicon = { workspace = true }
thiserror = { workspace = true }
anyhow = { workspace = true }
similar = "2.1.0"
similar ={ workspace = true }
wat.workspace = true
toml = "0.5.9"
serde = "1.0.94"
toml = { workspace = true }
serde = { workspace = true }
cranelift-wasm.workspace = true
wasmparser.workspace = true
cranelift.workspace = true
7 changes: 6 additions & 1 deletion winch/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -11,9 +11,10 @@ edition.workspace = true
name = "winch-tools"
path = "src/main.rs"


[dependencies]
winch-codegen = { workspace = true }
winch-filetests = { workspace = true }
winch-test-macros = { workspace = true }
wasmtime-environ = { workspace = true }
target-lexicon = { workspace = true }
anyhow = { workspace = true }
Expand All @@ -22,6 +23,10 @@ clap = { workspace = true }
wat = { workspace = true }
cranelift-codegen = { workspace = true }
capstone = { workspace = true }
similar = { workspace = true }
toml = { workspace = true }
serde = { workspace = true }
glob = { workspace = true }

[features]
default = ["all-arch"]
Expand Down
63 changes: 63 additions & 0 deletions winch/docs/testing.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,63 @@
# Testing Winch

Winch is tested through integration testing using the `winch-filetests` crate
and manual exploratory testing. A CLI is available to run these tests
conveniently. To add the `winch-tools` binary to your `PATH`, run `cargo install
--path winch` from the root of `wasmtime`. The CLI provides two commands: `test`
and `compile`. To see the help text for each command, run `winch-tools test
--help` or `winch-tools compile --help`.

## Integration Testing (`winch-tools test`)

The `test` command will run a suite of tests that validates Winch output for a
WebAssembly module is consistent with our expectations.

### Running `test`

Running `winch-tools test` will run all integration tests in the
`winch-filetests` crate. All arguments following two dashes (`--`) will be
passed directly to `cargo test -p winch-filetests`. This will allow you to
configure the tests to run based on your requirements. All tests in the
`winch-filetests` crate get named in the following convention:
`winch_filetests_${filepath}`. This makes it possible to filter if you don't
want to run the entire suite.

If the output of Winch changes for a test in a run due to code updates, the test
will fail and the difference between the two outputs will be shown. If the new
output is expected, the tests can be re-run with an `WINCH_TEST_BLESS`
environment variable set to `1`.

### Adding a test

To add new tests, create a `.wat` file in the `winch/filetests/filetests` folder
in the following format:

```wat
;;! target = "x86_64"
(module
(func (result i32)
(i32.const 42)
)
)
```

It is encouraged to use folders to organize tests. For example, tests targeting
the x86_64 architecture can be placed in the `winch/filetests/filetests/x64`.

The first block of comments are a TOML compatible configuration passed to Winch
during compilation with a `!` at the start of each line. The body of the file
will be the subject of the test. A final block of comments is reserved for the
output of the compilation, and it will be used to compare the output of the
current run with the output of previous runs.

## Manual Exploratory Tests (`winch-tools compile`)

The `compile` command will run Winch for particular architecture against
provided input file, and print the disassembled output to the console. Only
`.wat` files are supported.

### Running `compile`

```bash
winch-tools compile $wat_file --target $target_triple
```
22 changes: 22 additions & 0 deletions winch/filetests/Cargo.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
[package]
authors = ["The Winch Project Developers"]
name = "winch-filetests"
description = "Tests for the Winch compiler based on a set of known valid files"
license = "Apache-2.0 WITH LLVM-exception"
repository = "https://github.com/bytecodealliance/wasmtime"
version = "0.0.0"
publish = false
edition.workspace = true

[dependencies]
winch-test-macros = {workspace = true}
target-lexicon = { workspace = true }
winch-codegen = { workspace = true, features = ['all-arch'] }
wasmtime-environ = { workspace = true }
anyhow = { workspace = true }
wat = { workspace = true }
similar = { workspace = true }
toml = { workspace = true }
serde = { workspace = true }
cranelift-codegen = { workspace = true }
capstone = { workspace = true }
12 changes: 12 additions & 0 deletions winch/filetests/filetests/x64/simple.wat
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
;;! target = "x86_64"

(module
(func (result i32)
(i32.const 42)
)
)
;; 0: 55 push rbp
;; 1: 4889e5 mov rbp, rsp
;; 4: 48c7c02a000000 mov rax, 0x2a
;; b: 5d pop rbp
;; c: c3 ret
59 changes: 59 additions & 0 deletions winch/filetests/src/disasm.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,59 @@
//! Disassembly utilities.
use anyhow::{bail, Result};
use capstone::prelude::*;
use std::fmt::Write;
use target_lexicon::Architecture;
use winch_codegen::TargetIsa;

/// Disassemble and print a machine code buffer.
pub fn disasm(bytes: &[u8], isa: &dyn TargetIsa) -> Result<Vec<String>> {
let dis = disassembler_for(isa)?;
let insts = dis.disasm_all(bytes, 0x0).unwrap();

let disassembled_lines = insts
.iter()
.map(|i| {
let mut line = String::new();

write!(&mut line, "{:4x}:\t ", i.address()).unwrap();

let mut bytes_str = String::new();
let mut len = 0;
for b in i.bytes() {
write!(&mut bytes_str, "{:02x}", b).unwrap();
len += 1;
}
write!(&mut line, "{:21}\t", bytes_str).unwrap();
if len > 8 {
write!(&mut line, "\n\t\t\t\t").unwrap();
}

if let Some(s) = i.mnemonic() {
write!(&mut line, "{}\t", s).unwrap();
}

if let Some(s) = i.op_str() {
write!(&mut line, "{}", s).unwrap();
}

line
})
.collect();

Ok(disassembled_lines)
}

fn disassembler_for(isa: &dyn TargetIsa) -> Result<Capstone> {
let disasm = match isa.triple().architecture {
Architecture::X86_64 => Capstone::new()
.x86()
.mode(arch::x86::ArchMode::Mode64)
.build()
.map_err(|e| anyhow::format_err!("{}", e))?,

_ => bail!("Unsupported ISA"),
};

Ok(disasm)
}
Loading

0 comments on commit da03ff4

Please sign in to comment.