swap-ir-impl starting point (#4)

* swap-ir-impl starting point

* use absolute paths in IsNode derive macro

* binary_op: move Add, implement Mul and Sub

* fix IsNode derive macro module name

* fix name conflict with rust keywords in IsNode derive macro

* blocks: Function, Evaluator, If, For, Fold

* structured_op: moved Fold, Call

* Implement Visitor on new structure

* Add some fields to Graph to match previous API

* IsNode derive macro: support enums

* unary_ops: Boundary and Enf
IsNode derive macro: support extra fields

* IsLeaf derive macro for enums

* leaf nodes: SpannedMirVariable + Parameter

* aggregated_op: Vector and Matrix

* represent Matrix as Node of Vectors

* First steps on updating pretty printer

* Start introducing new node structure in Mir

* Update passes

* Implement Unrolling (needs Hash & Option/Vecs)

* Implement Lowering MIR > AIR

* Avoid referencing a func/ev that has not been defined

+ cargo fmt

* Update Unrolling and Inlining with helper

* Comment out Old passes

* Cleaned up a bit

* Add Access node usage to MIR

* bugfix

* Finished duplicate_node helper

* Replace Call with last expr of the body

* rename Access => Accessor

* Fix Accessor unrolling

* cargo fmt

* Small bugfixes in Unrolling for Accessor

* Update passes try ir3 (#5)

* derive Hash for SpannedMirValue

* ir3: better node splitting, no macro, simpler api
missimg builder pattern

* replaced Link<Op> by Link<Owner> for Operations and Leaves

* added IndexAccess Operand

* renamed IndexAccess to Accessor
re use parser's AccessType
impl Hash for AccessType

* fix call being copied instead of pointed to by get_children()

* typesafe builder pattern for most ops. Misssing structured ops

* type-safe Builder pattern for structured ops

* MirGraph Builder editing

* Add Leaves to Op enum

* ir3: comment top level items

* isolate Leaf, Op, Owner, Root

* add converters to enums

* added converters to structs
renamed IndexAccess to Acccessor

* Initial merge ir3 > update_passes

* remove comments

* Update mod.rs

* Graph: implement public api

* Swapped in helpers of passes/mod.rs

* Node: all node types + converters

* Full inlining with Node + cargo fmt

* Remove Link for FoldOperator

* Swap unrolling

* Swap lowering Mir -> Air

* Update visitor (with suboptimal get_children() herlper, to refactor)

* Add MirType to Parameter

* Update inlining to handle evaluators

* most of translated adapted

* add type support for Parameter in translate

* in build_statement, handle Owner::None (integrity or boundary constraint)

---------

Co-authored-by: Thybault Alabarbe <thybault.alabarbe@gmail.com>

* Cargo format and make Lowering Mir > Air build

* Clean both pipelines, removed unused structures

* removed: unused derive_graph, allow unused

* accept non-vectors as For.iterators

* Fix ir translate and passes (#7)

* fix translate.rs not translating bodies

* expand fn and ev args, missing scoping

* expand all arguments in function and evaluators

* unpack vec in all cases

* lookup arguments in the access_map if not found in bindings

* stop unpacking call arguments, add Vector<Params> to bindings

* insert accessors to bound Vector<Param>

* fix Value Builder types

* insert links in enums

* added missing Child/Parent traits

* Remove fn edit(self) on builders

* Make translate_from_mir.rs compile

* fix double borrow

* Handle multiple parents

* remove some warns

* Adapt Visitor and Inlining

* Update inlining and visitor

* Fix translate of functions

* visitor with pre-scan

* Add unrolling2 + fmt

* Add comments to passes

* Add comment to translate.rs pass

* Update Inlining and Unrolling passes

Stilll some debug todo

* Cargo fmt

* Add TODOs

* Builder derive macro

* fix Builder derive macro compilation

* Builder derive macro: fix Default recursion

* Builder derive macro: fix mutability on non-link, fix transistion
derive for Function

* Builder derive macro: handle Vec<BackLink> + impl Sub

* Builder derive macro: docs & fix + test >2 required fields case

* Builder derive macro: derive on all structs

* remived unused buggy api to remoce_child

* mutability examples and potention solutions:
one bug left

* double borrow bug remaining

* fix test_mutability_wrap_enum: missing clone

* Fix doouble borrow in Link::update

* fix singleton updates in final design

* cleanup intermediary versions

* fix module name

* Translate - Split function / ev parameter translation

* Fix for_node children, don't put None selector

* Fix inlining - Needs mutability

* Cleanup unrolling

* cargo fmt

* fix warns

* Improve comments

* Improve context setting in Unrolling

* refactor checkpoint: wrap structs in Op/Root enums

* Fix let translation

* Add prints to unrolling

* Add diagnostics to translate

* swap with ir + fix Builder enumwrapper
checkpoint: does not compile

* Setup diagnostics for inlining and unrolling

* converer bug: return from local reference

* avoid double option in BackLink::to_link()

* wrap all structs in enum:
checkpoint: compiles, 81 tests pass

* revert examples

* fix missing ;

* fix compilation: 43 tests pass

* Remove warns, fix translate_from_mir.rs

* Fix Inlining

* Fix inlining after git merge inconsistency

* Have parameter reference ref_node, and better handle evaluator params / args

* Update inlining2.rs

* Only inline Params that target the ref_node we aim

* Improve diags on Evaluator args and params mismatch

* Improve diags again

* Reworked visit_if to work with selectors, kept visit_if_old

* cargo clippy

* Clippy fix

* Clippy fix and remove prints

* cargo fmt

* cargo clippy + fmt

* Remove prints

* Builder: ignore fields that start with underscore

* singleton converters

* patch link

* api for shared mutability

* transform BackLink to Link when comparing and hashing singlton enum
wrappers

* swap *obj.borrow_mut() = value to obj.set(value)

* use ptr as hashmap key to preserve 1 key per instance
restore PartialEq Invariant for HashMap key

* use pointer as HashMap keys

* Some fixes for translate + inlining

* fix BackLink fields' mutability in Builder derive macro

* cargo fmt

* fix Parameter.ref_node cyclic reference + restric to Owner + compare via get_ptr

fix Parameter.ref_node cyclic reference:
Parameter.ref_Node used to store a Link to its parent which caused
PartialEq and Hash to loop.
Replaced with a BackLink
 + restricted the field from Node -> Owner
 + expose inner get_ptr on Owner
 + comparison and hash via get_ptr

* Update translate, mod and inlining

* Fix nested evaluators

* fix Parameter PartialEq and Hash to work on disconnected identical graphs

* add debugging method to Op/Root + Link/BackLink

* filter stale node wrappers in visitor

* fix edgecase in `Link<Op>::set`

* manual pointer review checks to verify Op::set logic

* fix parameters

* Fix let translation

* Debug air-ir tests, begin codegen tests

* Set program name in Mir

* Fix mutability test

---------

Co-authored-by: Thybault Alabarbe <thybault.alabarbe@gmail.com>

* cargo fmt

* removed unused files

* rename passes files

* Cargo check fixes

* cargo clippy fixes

* Improve Accessor of trace columns handling

* Add Mir Exp Node, fix Accessor logic

* Update expected codegen results (equivalent, but less optimized)

* Fix functions_complex and evaluator codegen tests

* fix uneeded unwrap

* Make test pass

* cargo clippy fixes

* cargo fmt

* Defaults to Mir pipeline in CLI, expose CLI arg to bypass Mir

* removed redundant ignored tests

* derive Spanned on Evaluator, Function, Parameter, and For

* derive Span for Root

* chore: refactor and document inlining, translate, pass helpers

* derive Hash for Accessor

* Derive Spanned for all node types and wrappers

* Propagate span + cargo fmt

* unrolling documentation + cargo check fixes

* Add translate_from_mir documentation

* mirror PartalEq field usage in Hash to restore Hash invariant

* cargo clippy fixes

---------

Co-authored-by: Leo-Besancon <lb@massa.net>

air-script/Cargo.toml

@@ -17,6 +17,7 @@ name = "airc"
 path = "src/main.rs"
 
 [dependencies]
+mir = { package = "mir", path = "../ir", version = "0.4" }
 air-ir = { package = "air-ir", path = "../codegen/air", version = "0.4" }
 air-parser = { package = "air-parser", path = "../parser", version = "0.4" }
 air-pass = { package = "air-pass", path = "../pass", version = "0.1" }

air-script/src/cli/transpile.rs

@@ -21,6 +21,11 @@ impl Target {
         }
     }
 }
+#[derive(Copy, Clone, PartialEq, Eq, PartialOrd, Ord, ValueEnum)]
+pub enum Pipeline {
+    WithMIR,
+    WithoutMIR,
+}
 
 #[derive(Args)]
 pub struct Transpile {
@@ -40,28 +45,55 @@ pub struct Transpile {
         help = "Defines the target language, defaults to Winterfell"
     )]
     target: Option<Target>,
+
+    #[arg(
+        short,
+        long,
+        help = "Defines the compilation pipeline (WithMIR or WithoutMIR), defaults to WithMIR"
+    )]
+    pipeline: Option<Pipeline>,
 }
 
 impl Transpile {
     pub fn execute(&self) -> Result<(), String> {
         println!("============================================================");
-        println!("Transpiling...");
 
         let input_path = &self.input;
 
         let codemap = Arc::new(CodeMap::new());
         let emitter = Arc::new(DefaultEmitter::new(ColorChoice::Auto));
         let diagnostics = DiagnosticsHandler::new(Default::default(), codemap.clone(), emitter);
 
+        let pipeline = self.pipeline.unwrap_or(Pipeline::WithMIR);
         // Parse from file to internal representation
-        let air = air_parser::parse_file(&diagnostics, codemap, input_path)
-            .map_err(CompileError::Parse)
-            .and_then(|ast| {
-                let mut pipeline = air_parser::transforms::ConstantPropagation::new(&diagnostics)
-                    .chain(air_parser::transforms::Inlining::new(&diagnostics))
-                    .chain(air_ir::passes::AstToAir::new(&diagnostics));
-                pipeline.run(ast)
-            });
+        let air = match pipeline {
+            Pipeline::WithMIR => {
+                println!("Transpiling with Mir pipeline...");
+                air_parser::parse_file(&diagnostics, codemap, input_path)
+                    .map_err(CompileError::Parse)
+                    .and_then(|ast| {
+                        let mut pipeline =
+                            air_parser::transforms::ConstantPropagation::new(&diagnostics)
+                                .chain(mir::passes::AstToMir::new(&diagnostics))
+                                .chain(mir::passes::Inlining::new(&diagnostics))
+                                .chain(mir::passes::Unrolling::new(&diagnostics))
+                                .chain(air_ir::passes::MirToAir::new(&diagnostics));
+                        pipeline.run(ast)
+                    })
+            }
+            Pipeline::WithoutMIR => {
+                println!("Transpiling without Mir pipeline...");
+                air_parser::parse_file(&diagnostics, codemap, input_path)
+                    .map_err(CompileError::Parse)
+                    .and_then(|ast| {
+                        let mut pipeline =
+                            air_parser::transforms::ConstantPropagation::new(&diagnostics)
+                                .chain(air_parser::transforms::Inlining::new(&diagnostics))
+                                .chain(air_ir::passes::AstToAir::new(&diagnostics));
+                        pipeline.run(ast)
+                    })
+            }
+        };
 
         match air {
             Ok(air) => {

air-script/tests/main.rs

@@ -1 +1,2 @@
-mod codegen;
+mod tests_from_mir;
+mod tests_wo_mir;

air-script/tests/tests_from_mir/codegen/helpers.rs

@@ -0,0 +1,48 @@
+use std::sync::Arc;
+
+use air_ir::{CodeGenerator, CompileError};
+
+use air_pass::Pass;
+use miden_diagnostics::{
+    term::termcolor::ColorChoice, CodeMap, DefaultEmitter, DiagnosticsHandler,
+};
+
+pub enum Target {
+    Winterfell,
+    Masm,
+}
+
+pub struct Test {
+    input_path: String,
+}
+impl Test {
+    pub fn new(input_path: String) -> Self {
+        Test { input_path }
+    }
+
+    pub fn transpile(&self, target: Target) -> Result<String, CompileError> {
+        let codemap = Arc::new(CodeMap::new());
+        let emitter = Arc::new(DefaultEmitter::new(ColorChoice::Auto));
+        let diagnostics = DiagnosticsHandler::new(Default::default(), codemap.clone(), emitter);
+
+        // Parse from file to internal representation
+        let air = air_parser::parse_file(&diagnostics, codemap, &self.input_path)
+            .map_err(CompileError::Parse)
+            .and_then(|ast| {
+                let mut pipeline = air_parser::transforms::ConstantPropagation::new(&diagnostics)
+                    .chain(mir::passes::AstToMir::new(&diagnostics))
+                    .chain(mir::passes::Inlining::new(&diagnostics))
+                    .chain(mir::passes::Unrolling::new(&diagnostics))
+                    .chain(air_ir::passes::MirToAir::new(&diagnostics));
+                pipeline.run(ast)
+            })?;
+
+        let backend: Box<dyn CodeGenerator<Output = String>> = match target {
+            Target::Winterfell => Box::new(air_codegen_winter::CodeGenerator),
+            Target::Masm => Box::<air_codegen_masm::CodeGenerator>::default(),
+        };
+
+        // generate Rust code targeting Winterfell
+        Ok(backend.generate(&air).expect("code generation failed"))
+    }
+}

air-script/tests/tests_from_mir/codegen/masm.rs

@@ -0,0 +1,241 @@
+use super::helpers::{Target, Test};
+use expect_test::expect_file;
+
+// tests_from_mir
+// ================================================================================================
+
+#[test]
+fn aux_trace() {
+    let generated_masm = Test::new("tests/tests_from_mir/aux_trace/aux_trace.air".to_string())
+        .transpile(Target::Masm)
+        .unwrap();
+
+    let expected = expect_file!["../aux_trace/aux_trace.masm"];
+    expected.assert_eq(&generated_masm);
+}
+
+#[test]
+fn binary() {
+    let generated_masm = Test::new("tests/tests_from_mir/binary/binary.air".to_string())
+        .transpile(Target::Masm)
+        .unwrap();
+
+    let expected = expect_file!["../binary/binary.masm"];
+    expected.assert_eq(&generated_masm);
+}
+
+#[test]
+fn periodic_columns() {
+    let generated_masm =
+        Test::new("tests/tests_from_mir/periodic_columns/periodic_columns.air".to_string())
+            .transpile(Target::Masm)
+            .unwrap();
+
+    let expected = expect_file!["../periodic_columns/periodic_columns.masm"];
+    expected.assert_eq(&generated_masm);
+}
+
+#[test]
+fn pub_inputs() {
+    let generated_masm = Test::new("tests/tests_from_mir/pub_inputs/pub_inputs.air".to_string())
+        .transpile(Target::Masm)
+        .unwrap();
+
+    let expected = expect_file!["../pub_inputs/pub_inputs.masm"];
+    expected.assert_eq(&generated_masm);
+}
+
+#[test]
+fn system() {
+    let generated_masm = Test::new("tests/tests_from_mir/system/system.air".to_string())
+        .transpile(Target::Masm)
+        .unwrap();
+
+    let expected = expect_file!["../system/system.masm"];
+    expected.assert_eq(&generated_masm);
+}
+
+#[test]
+fn bitwise() {
+    let generated_masm = Test::new("tests/tests_from_mir/bitwise/bitwise.air".to_string())
+        .transpile(Target::Masm)
+        .unwrap();
+
+    let expected = expect_file!["../bitwise/bitwise.masm"];
+    expected.assert_eq(&generated_masm);
+}
+
+#[test]
+fn constants() {
+    let generated_masm = Test::new("tests/tests_from_mir/constants/constants.air".to_string())
+        .transpile(Target::Masm)
+        .unwrap();
+
+    let expected = expect_file!["../constants/constants.masm"];
+    expected.assert_eq(&generated_masm);
+}
+
+#[test]
+fn constant_in_range() {
+    let generated_masm =
+        Test::new("tests/tests_from_mir/constant_in_range/constant_in_range.air".to_string())
+            .transpile(Target::Masm)
+            .unwrap();
+
+    let expected = expect_file!["../constant_in_range/constant_in_range.masm"];
+    expected.assert_eq(&generated_masm);
+}
+
+#[test]
+fn evaluators() {
+    let generated_masm = Test::new("tests/tests_from_mir/evaluators/evaluators.air".to_string())
+        .transpile(Target::Masm)
+        .unwrap();
+
+    let expected = expect_file!["../evaluators/evaluators.masm"];
+    expected.assert_eq(&generated_masm);
+}
+
+#[test]
+fn functions_simple() {
+    let generated_masm =
+        Test::new("tests/tests_from_mir/functions/functions_simple.air".to_string())
+            .transpile(Target::Masm)
+            .unwrap();
+
+    let expected = expect_file!["../functions/functions_simple.masm"];
+    expected.assert_eq(&generated_masm);
+}
+
+#[test]
+fn functions_simple_inlined() {
+    // make sure that the constraints generated using inlined functions are the same as the ones
+    // generated using regular functions
+    let generated_masm =
+        Test::new("tests/tests_from_mir/functions/inlined_functions_simple.air".to_string())
+            .transpile(Target::Masm)
+            .unwrap();
+    let expected = expect_file!["../functions/functions_simple.masm"];
+    expected.assert_eq(&generated_masm);
+}
+
+#[test]
+fn functions_complex() {
+    let generated_masm =
+        Test::new("tests/tests_from_mir/functions/functions_complex.air".to_string())
+            .transpile(Target::Masm)
+            .unwrap();
+
+    let expected = expect_file!["../functions/functions_complex.masm"];
+    expected.assert_eq(&generated_masm);
+}
+
+#[test]
+fn variables() {
+    let generated_masm = Test::new("tests/tests_from_mir/variables/variables.air".to_string())
+        .transpile(Target::Masm)
+        .unwrap();
+
+    let expected = expect_file!["../variables/variables.masm"];
+    expected.assert_eq(&generated_masm);
+}
+
+#[test]
+fn trace_col_groups() {
+    let generated_masm =
+        Test::new("tests/tests_from_mir/trace_col_groups/trace_col_groups.air".to_string())
+            .transpile(Target::Masm)
+            .unwrap();
+
+    let expected = expect_file!["../trace_col_groups/trace_col_groups.masm"];
+    expected.assert_eq(&generated_masm);
+}
+
+#[test]
+fn indexed_trace_access() {
+    let generated_masm =
+        Test::new("tests/tests_from_mir/indexed_trace_access/indexed_trace_access.air".to_string())
+            .transpile(Target::Masm)
+            .unwrap();
+
+    let expected = expect_file!["../indexed_trace_access/indexed_trace_access.masm"];
+    expected.assert_eq(&generated_masm);
+}
+
+#[test]
+#[ignore] // TODO: There is some non-determinism in the IR creation, unskip this test once it is fixed
+fn random_values() {
+    let generated_masm =
+        Test::new("tests/tests_from_mir/random_values/random_values_simple.air".to_string())
+            .transpile(Target::Masm)
+            .unwrap();
+    let expected = expect_file!["../random_values/random_values.masm"];
+    expected.assert_eq(&generated_masm);
+
+    let generated_masm =
+        Test::new("tests/tests_from_mir/random_values/random_values_bindings.air".to_string())
+            .transpile(Target::Masm)
+            .unwrap();
+    let expected = expect_file!["../random_values/random_values.masm"];
+    expected.assert_eq(&generated_masm);
+}
+
+#[test]
+fn list_comprehension() {
+    let generated_masm =
+        Test::new("tests/tests_from_mir/list_comprehension/list_comprehension.air".to_string())
+            .transpile(Target::Masm)
+            .unwrap();
+
+    let expected = expect_file!["../list_comprehension/list_comprehension.masm"];
+    expected.assert_eq(&generated_masm);
+}
+
+#[test]
+fn list_folding() {
+    let generated_masm =
+        Test::new("tests/tests_from_mir/list_folding/list_folding.air".to_string())
+            .transpile(Target::Masm)
+            .unwrap();
+
+    let expected = expect_file!["../list_folding/list_folding.masm"];
+    expected.assert_eq(&generated_masm);
+}
+
+#[test]
+#[ignore] // TODO: There is some non-determinism in the IR creation, unskip this test once it is fixed
+fn selectors() {
+    let generated_masm = Test::new("tests/tests_from_mir/selectors/selectors.air".to_string())
+        .transpile(Target::Masm)
+        .unwrap();
+    let expected = expect_file!["../selectors/selectors.masm"];
+    expected.assert_eq(&generated_masm);
+
+    let generated_masm =
+        Test::new("tests/tests_from_mir/selectors/selectors_with_evaluators.air".to_string())
+            .transpile(Target::Masm)
+            .unwrap();
+    let expected = expect_file!["../selectors/selectors.masm"];
+    expected.assert_eq(&generated_masm);
+}
+
+#[test]
+fn constraint_comprehension() {
+    let generated_masm = Test::new(
+        "tests/tests_from_mir/constraint_comprehension/constraint_comprehension.air".to_string(),
+    )
+    .transpile(Target::Masm)
+    .unwrap();
+
+    let expected = expect_file!["../constraint_comprehension/constraint_comprehension.masm"];
+    expected.assert_eq(&generated_masm);
+
+    let generated_masm = Test::new(
+        "tests/tests_from_mir/constraint_comprehension/cc_with_evaluators.air".to_string(),
+    )
+    .transpile(Target::Masm)
+    .unwrap();
+
+    let expected = expect_file!["../constraint_comprehension/constraint_comprehension.masm"];
+    expected.assert_eq(&generated_masm);
+}