From 18de837a4cb8779377f3b8e6703d35aa84d88c9a Mon Sep 17 00:00:00 2001 From: Viktar Makouski Date: Tue, 13 Feb 2024 16:48:40 +0300 Subject: [PATCH 1/3] init --- docs/zkasm/test-infra.md | 98 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 98 insertions(+) create mode 100644 docs/zkasm/test-infra.md diff --git a/docs/zkasm/test-infra.md b/docs/zkasm/test-infra.md new file mode 100644 index 000000000000..892c7a74ffbb --- /dev/null +++ b/docs/zkasm/test-infra.md @@ -0,0 +1,98 @@ +# About + +We want to build a robust testing infrastructure which will: + +- allow easily add new test. +- allow to reuse existing cranelift tests. +- be easy to introduce new type of tests. + +We want keep our testing infra flexible and friendly to user. Also, this version is not final, and will be improved. For example, testing the interpreter by running wasm in it is out of scope for now. + + +# Testing use cases + +We have two main use cases for our testing infra: + +- CI. We want run our tests in CI to keep quality of code in main branch on high level +- Developing. We want our tests to be easy to run by developer in local machine and see how current version of code works. This use case give us a new requirement -- our testing infra should be fast enough, ideally run in few seconds. + +# How to use testing infra + +## How test file looks like? + +Lets take a look how test file looks like. +Firstly, it have a header, which usually looks like this: + +``` +test interpret +test run +test run-zkasm +target aarch64 +target s390x +target x86_64 +target riscv64 has_m +target riscv64 has_c has_zcb +``` + +Here we see list of tests and list of targets. From it, we are interested only in `test run-zkasm`, this test tell parser that zkasm test should be executed. Our test is target independent, so, we don't care a lot about other tests and all targets. One line header `test run-zkasm` will work for us. + +Secondly, test file contains list of functions and invocations of this functions, they tipically look like this: + +``` +function %add_i64(i64, i64) -> i64 { +block0(v0: i64,v1: i64): + v2 = iadd v0, v1 + return v2 +} +; run: %add_i64(0, 0) == 0 +; run: %add_i64(0, 1) == 1 +; run: %add_i64(-1, 0) == -1 +; run: %add_i64(-1, 1) == 0 +``` + +Here, you first define function using `clif` ([Cranelift IR](https://github.com/bytecodealliance/wasmtime/blob/main/cranelift/docs/ir.md)). + +Than, you have a list of run commands, each of them compiles function to zkasm, runs function with given arguments, and checks the result. + +## How to run a test on my machine? + +To run a single test, you can use next command: +`cargo run -- test /testname.clif` + +To run multiple tests in a directory, you can use: +`cargo run -- test /` + +## How to add a new test? + +If you have add new `zkasm-run` test, you have two options: + +- firstly, you can simply add a new function and a list of it's invocations to the existing test file (or even add only some new invocations to an existing function). +- secondly, you can create new test file, make shure it contains `test run-zkasm` in the header, and put your functions and invocations there. `cranelift/filetests/filetests/zkasm` would be a good place to put your file in. + +# Test infra implementation + +Firstly, our testing infrastructure based on Cranelift testing infrastructure ([filetests](https://github.com/bytecodealliance/wasmtime/blob/main/cranelift/docs/testing.md)). It provides us powerful reusable parts, like test files parsing, and a number of ready to use testcases. + +## Files to look at + +Our `test run-zkasm` is a new test type in filetests, described in `cranelift/filetests/src/test_run_zkasm.rs`. +Also, it can be useful to take a look at `cranelift/filetests/src/zkasm_codegen.rs` with some helper functions. + +## What our test actually do? + +Firstly, test infra parses whole `.clif` file (different `.clif` files are parsed independently and don't affect each other). Than, takes all functions and invocations and build one big zkasm program from them, where assertions are compiled as a special assert function, which don't halt execution of the program if assert fails. Instead, it saves results of assert, and in the end this results are taken to build table with results of whole testfile. There can be four types of results: `pass`, `assert_fails`, `compilation_fails`, `run_time_error`. Unfortunately, with our approach (building one zkasm program for one test file), two last statuses will be applied to all tests in file if any of tests in this file get such result. But, we chosed this approach, because starting of zkasm program takes significant time, and we want our tests be fast and easy to use. Function with custom assert is described in `tests/zkasm/run-tests-zkasm.js` + +## Future improvements + +Probably, we will add some new test types, for example, test type which takes wasm code and executes it in interpreter. + +# CI policy + +In CI we have two types of tests. One of them are written\generated by us, or adopted from existing tests and we expect that each PR passes 100% of this tests. Also, we run all tests in `filetests/filtests/runtests` as well, but don't expect all of them pass. If some test have `pass` status in main branch, and other status in the incoming branch, our CI will post a message with this diff in PR discussion, and author with reviewers will decide merge this PR or not. During project development, tests will gradually migrate from second category to the first one. + + +# Decisions to think + +## One zkasm program per file + +This decision have it's own pros and cons: while it allows to run tests fastly, it leads to not good behaviour in case of compilation and runtime errors. It will be nice to find some better option here. From bdc87257f087ad0b47cea024e19250286951ccac Mon Sep 17 00:00:00 2001 From: Viktar Makouski Date: Wed, 14 Feb 2024 00:42:53 +0300 Subject: [PATCH 2/3] reference to cranelift docs instead of duplication --- docs/zkasm/test-infra.md | 40 +--------------------------------------- 1 file changed, 1 insertion(+), 39 deletions(-) diff --git a/docs/zkasm/test-infra.md b/docs/zkasm/test-infra.md index 892c7a74ffbb..ed8709c167bb 100644 --- a/docs/zkasm/test-infra.md +++ b/docs/zkasm/test-infra.md @@ -20,39 +20,7 @@ We have two main use cases for our testing infra: ## How test file looks like? -Lets take a look how test file looks like. -Firstly, it have a header, which usually looks like this: - -``` -test interpret -test run -test run-zkasm -target aarch64 -target s390x -target x86_64 -target riscv64 has_m -target riscv64 has_c has_zcb -``` - -Here we see list of tests and list of targets. From it, we are interested only in `test run-zkasm`, this test tell parser that zkasm test should be executed. Our test is target independent, so, we don't care a lot about other tests and all targets. One line header `test run-zkasm` will work for us. - -Secondly, test file contains list of functions and invocations of this functions, they tipically look like this: - -``` -function %add_i64(i64, i64) -> i64 { -block0(v0: i64,v1: i64): - v2 = iadd v0, v1 - return v2 -} -; run: %add_i64(0, 0) == 0 -; run: %add_i64(0, 1) == 1 -; run: %add_i64(-1, 0) == -1 -; run: %add_i64(-1, 1) == 0 -``` - -Here, you first define function using `clif` ([Cranelift IR](https://github.com/bytecodealliance/wasmtime/blob/main/cranelift/docs/ir.md)). - -Than, you have a list of run commands, each of them compiles function to zkasm, runs function with given arguments, and checks the result. +[Filetest docs describe it](https://github.com/bytecodealliance/wasmtime/blob/main/cranelift/filetests/README.md). To run our zkasm test simply add `test run-zkasm` to the header of `.clif` file. ## How to run a test on my machine? @@ -62,12 +30,6 @@ To run a single test, you can use next command: To run multiple tests in a directory, you can use: `cargo run -- test /` -## How to add a new test? - -If you have add new `zkasm-run` test, you have two options: - -- firstly, you can simply add a new function and a list of it's invocations to the existing test file (or even add only some new invocations to an existing function). -- secondly, you can create new test file, make shure it contains `test run-zkasm` in the header, and put your functions and invocations there. `cranelift/filetests/filetests/zkasm` would be a good place to put your file in. # Test infra implementation From d2b36de1ea6e00bb95ee7b74b1e4c7eb70b62558 Mon Sep 17 00:00:00 2001 From: Viktar Makouski Date: Wed, 14 Feb 2024 15:15:02 +0300 Subject: [PATCH 3/3] remove part with extra CI policy --- docs/zkasm/test-infra.md | 5 ----- 1 file changed, 5 deletions(-) diff --git a/docs/zkasm/test-infra.md b/docs/zkasm/test-infra.md index ed8709c167bb..2ae20793c3b0 100644 --- a/docs/zkasm/test-infra.md +++ b/docs/zkasm/test-infra.md @@ -48,11 +48,6 @@ Firstly, test infra parses whole `.clif` file (different `.clif` files are parse Probably, we will add some new test types, for example, test type which takes wasm code and executes it in interpreter. -# CI policy - -In CI we have two types of tests. One of them are written\generated by us, or adopted from existing tests and we expect that each PR passes 100% of this tests. Also, we run all tests in `filetests/filtests/runtests` as well, but don't expect all of them pass. If some test have `pass` status in main branch, and other status in the incoming branch, our CI will post a message with this diff in PR discussion, and author with reviewers will decide merge this PR or not. During project development, tests will gradually migrate from second category to the first one. - - # Decisions to think ## One zkasm program per file