forked from bytecodealliance/wasmtime
-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
winch: Adding support for integration tests (bytecodealliance#5588)
* Adding in the foundations for Winch `filetests` This commit adds two new crates into the Winch workspace: `filetests` and `test-macros`. The intent is to mimic the structure of Cranelift `filetests`, but in a simpler way. * Updates to documentation This commits adds a high level document to outline how to test Winch through the `winch-tools` utility. It also updates some inline documentation which gets propagated to the CLI. * Updating test-macro to use a glob instead of only a flat directory
- Loading branch information
1 parent
7cea73a
commit da03ff4
Showing
16 changed files
with
586 additions
and
135 deletions.
There are no files selected for viewing
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
Oops, something went wrong.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,63 @@ | ||
# Testing Winch | ||
|
||
Winch is tested through integration testing using the `winch-filetests` crate | ||
and manual exploratory testing. A CLI is available to run these tests | ||
conveniently. To add the `winch-tools` binary to your `PATH`, run `cargo install | ||
--path winch` from the root of `wasmtime`. The CLI provides two commands: `test` | ||
and `compile`. To see the help text for each command, run `winch-tools test | ||
--help` or `winch-tools compile --help`. | ||
|
||
## Integration Testing (`winch-tools test`) | ||
|
||
The `test` command will run a suite of tests that validates Winch output for a | ||
WebAssembly module is consistent with our expectations. | ||
|
||
### Running `test` | ||
|
||
Running `winch-tools test` will run all integration tests in the | ||
`winch-filetests` crate. All arguments following two dashes (`--`) will be | ||
passed directly to `cargo test -p winch-filetests`. This will allow you to | ||
configure the tests to run based on your requirements. All tests in the | ||
`winch-filetests` crate get named in the following convention: | ||
`winch_filetests_${filepath}`. This makes it possible to filter if you don't | ||
want to run the entire suite. | ||
|
||
If the output of Winch changes for a test in a run due to code updates, the test | ||
will fail and the difference between the two outputs will be shown. If the new | ||
output is expected, the tests can be re-run with an `WINCH_TEST_BLESS` | ||
environment variable set to `1`. | ||
|
||
### Adding a test | ||
|
||
To add new tests, create a `.wat` file in the `winch/filetests/filetests` folder | ||
in the following format: | ||
|
||
```wat | ||
;;! target = "x86_64" | ||
(module | ||
(func (result i32) | ||
(i32.const 42) | ||
) | ||
) | ||
``` | ||
|
||
It is encouraged to use folders to organize tests. For example, tests targeting | ||
the x86_64 architecture can be placed in the `winch/filetests/filetests/x64`. | ||
|
||
The first block of comments are a TOML compatible configuration passed to Winch | ||
during compilation with a `!` at the start of each line. The body of the file | ||
will be the subject of the test. A final block of comments is reserved for the | ||
output of the compilation, and it will be used to compare the output of the | ||
current run with the output of previous runs. | ||
|
||
## Manual Exploratory Tests (`winch-tools compile`) | ||
|
||
The `compile` command will run Winch for particular architecture against | ||
provided input file, and print the disassembled output to the console. Only | ||
`.wat` files are supported. | ||
|
||
### Running `compile` | ||
|
||
```bash | ||
winch-tools compile $wat_file --target $target_triple | ||
``` |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,22 @@ | ||
[package] | ||
authors = ["The Winch Project Developers"] | ||
name = "winch-filetests" | ||
description = "Tests for the Winch compiler based on a set of known valid files" | ||
license = "Apache-2.0 WITH LLVM-exception" | ||
repository = "https://github.com/bytecodealliance/wasmtime" | ||
version = "0.0.0" | ||
publish = false | ||
edition.workspace = true | ||
|
||
[dependencies] | ||
winch-test-macros = {workspace = true} | ||
target-lexicon = { workspace = true } | ||
winch-codegen = { workspace = true, features = ['all-arch'] } | ||
wasmtime-environ = { workspace = true } | ||
anyhow = { workspace = true } | ||
wat = { workspace = true } | ||
similar = { workspace = true } | ||
toml = { workspace = true } | ||
serde = { workspace = true } | ||
cranelift-codegen = { workspace = true } | ||
capstone = { workspace = true } |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,12 @@ | ||
;;! target = "x86_64" | ||
|
||
(module | ||
(func (result i32) | ||
(i32.const 42) | ||
) | ||
) | ||
;; 0: 55 push rbp | ||
;; 1: 4889e5 mov rbp, rsp | ||
;; 4: 48c7c02a000000 mov rax, 0x2a | ||
;; b: 5d pop rbp | ||
;; c: c3 ret |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,59 @@ | ||
//! Disassembly utilities. | ||
use anyhow::{bail, Result}; | ||
use capstone::prelude::*; | ||
use std::fmt::Write; | ||
use target_lexicon::Architecture; | ||
use winch_codegen::TargetIsa; | ||
|
||
/// Disassemble and print a machine code buffer. | ||
pub fn disasm(bytes: &[u8], isa: &dyn TargetIsa) -> Result<Vec<String>> { | ||
let dis = disassembler_for(isa)?; | ||
let insts = dis.disasm_all(bytes, 0x0).unwrap(); | ||
|
||
let disassembled_lines = insts | ||
.iter() | ||
.map(|i| { | ||
let mut line = String::new(); | ||
|
||
write!(&mut line, "{:4x}:\t ", i.address()).unwrap(); | ||
|
||
let mut bytes_str = String::new(); | ||
let mut len = 0; | ||
for b in i.bytes() { | ||
write!(&mut bytes_str, "{:02x}", b).unwrap(); | ||
len += 1; | ||
} | ||
write!(&mut line, "{:21}\t", bytes_str).unwrap(); | ||
if len > 8 { | ||
write!(&mut line, "\n\t\t\t\t").unwrap(); | ||
} | ||
|
||
if let Some(s) = i.mnemonic() { | ||
write!(&mut line, "{}\t", s).unwrap(); | ||
} | ||
|
||
if let Some(s) = i.op_str() { | ||
write!(&mut line, "{}", s).unwrap(); | ||
} | ||
|
||
line | ||
}) | ||
.collect(); | ||
|
||
Ok(disassembled_lines) | ||
} | ||
|
||
fn disassembler_for(isa: &dyn TargetIsa) -> Result<Capstone> { | ||
let disasm = match isa.triple().architecture { | ||
Architecture::X86_64 => Capstone::new() | ||
.x86() | ||
.mode(arch::x86::ArchMode::Mode64) | ||
.build() | ||
.map_err(|e| anyhow::format_err!("{}", e))?, | ||
|
||
_ => bail!("Unsupported ISA"), | ||
}; | ||
|
||
Ok(disasm) | ||
} |
Oops, something went wrong.