docs: infra

docs/zkasm/infra.md

@@ -0,0 +1,64 @@
+# General Overview
+
+Firstly, we aim to implement the following types of tests and benchmarks:
+- Wasm spectest: Programs generated from Wasm spectest.
+- Benchmarks: Programs we manually write in Wasm or Rust and compile to Wasm.
+
+Both types of programs will undergo the same pipeline:
+- Compile to zkasm.
+- Run the generated zkasm & measure performance.
+- Track changes.
+- Alert in GitHub PRs in case of degradation.
+
+We use the snapshot-testing pattern in many places (link).
+
+Performance data is collected from both tests and benchmarks.
+Benchmarks also serve as global tests.
+
+## Test Creation and Generation
+We use a snapshot-based pattern for generating Wasm from spectest wast files. Benchmarks are added manually.
+
+Our goal is to simplify adding benchmarks (simply add a `.wat` file to the benchmarks directory, and it automatically goes through the pipeline) and to automatically pull all updates from Wasm spectests.
+
+## Wasm Compiling
+We aim to have a single test that compiles all `.wat` files to zkasm and writes results to a CSV file in a snapshot-based pattern. This script should be capable of:
+- Compiling a single file with verbose compiler messages.
+- Recompiling all files and saving them.
+- Compiling all files and checking if the results are the same as the committed files (used in CI).
+
+## Running Compiled zkasm
+We intend to implement a single command to run all zkasm and update the state table in a snapshot-based pattern. This script should be capable of:
+- Running a single file with verbose zkasm processor messages.
+- Rerunning all files and updating state files.
+- Running all files and checking that the state is the same as in the state files (used in CI).
+
+## Developer Tooling
+We aim for easy, intuitive, and fixed commands for the following tasks:
+- Recompile a single `.wat` file.
+- Recompile all `.wat` files and update zkasm files.
+- Compile all `.wat` files and check if the results are the same as in the zkasm files.
+- Run a single zkasm file.
+- Run all zkasm files and check the diff with the state.
+- Run all zkasm files and update the state.
+- Perform the entire CI pipeline locally.
+
+## Testing Wasm
+We have a script that tests all tests and benchmarks to ensure they are correct programs. It sets up the environment, runs Wasm, and checks if all programs finish with code 0.
+
+## Accepting PRs
+Our CI checks that tests and benchmarks are correct, that committed zkasm files correspond to files generated by the compiler, and that state files contain actual data.
+
+We plan to implement a tool that automatically checks for any degradation in state files, such as performance slowdowns or failing tests that previously passed.
+
+# Steps to Achieve "Finalized" Infrastructure
+- [ ] Rewrite wast parsing to a generic approach.
+- [ ] Reorder `.wat` and zkasm files: move all state.csv to one directory, consider removing hand-written tests, and possibly merge some tests into a single file.
+- [ ] Introduce asserts as host functions in zkasm.
+- [ ] Separate different parts of the infrastructure into different files.
+- [ ] Add npm caching in CI.
+- [ ] Disable unused tests in CI.
+- [ ] Create a developer CLI tool and aim to stabilize it.
+- [ ] Implement a GitHub bot that alerts if any of our state.csv files degrade.
+
+# Technical Details
+Here we will provide a description of the implemented infrastructure and instructions on how to add new hand-written tests or benchmarks. TODO: This section will be updated as the major part of the infrastructure will be rewritten.