feat(raiko): generalized build pipeline for ZkVMs guests

[CeciliaZ030]

Problem:

• Each ZkVM has a different build pipeline and elf loading workflow. It's difficult to manage the pipeline with upstream updates, plus that we even have to make PR just to use desirable features / flags
• We want to compile unittests directly with #[test], which means we need to compile Rust's test harness into risv32 elf
• Also we want to customize the compilation of benchmarks.

Solution

I'm gonna make the build pipeline myself so that I don't want to bother about the upstream, and also to test and benchmark my code however i want. also. Here are some of the design considerations:

• In embedded system compilation, you need the correct target & toolchain for cargo and rustc, under the hood you might need to swap out gcc or llvm, which means you need to set a bunch of flags and args that go into the compilers.

• Currently most ZkVM build the guest in build.rs, and gen proofs & verify in main.rs, which means these are the processes actually being run:

  1. compiling build.rs, resulting in binary target/debug/build/hello-world-b4166fd/build-script-build
  2. run build-script-build, which invokes another cargo process that builds the guest ELF; here's how the command can look like:

  cd ../guest & CC="/home/ubuntu/.local/share/cargo-risczero/cpp/bin/riscv32-unknown-elf-gcc" CFLAGS_riscv32im_risc0_zkvm_elf="-march=rv32im -nostdlib" RUSTC="/home/ubuntu/.rustup/toolchains/risc0/bin/rustc" "cargo" "test" "--target" "riscv32im-risc0-zkvm-elf" "--no-run" "--manifest-path" "/home/ubuntu/xxx/hello-world/methods/guest/Cargo.toml" "--target-dir" "/home/ubuntu/xxx/hello-world/methods/guest" "--release"
  

  3. the ELF is produced, we enter main.rs which loads the ELF into our VM runtime and generates a proof.

I'm making the build stage explicit, which means having a builder crate to replace build.rs. Within the pipeline, there are two steps: 1) configure the right flags and env to generate a correct command 2) execute the command and place the ELFs in some user-specified directories. Resulting file structure:

example
├── Cargo.lock
├── Cargo.toml
├── builder
│   ├── Cargo.toml
│   └── src
├── driver
│   ├── Cargo.toml
│   └── src
├── guest
│   ├── Cargo.lock
│   ├── Cargo.toml
│   ├── methods
│   ├── src
│   └── target
└── target

Workflow

Build the ELFs

$ cd builder & cargo build --features <zkvm>

Run the ELFs

$ cd driver & cargo +nightly run --release --features <zkvm>

Builder

The resultant pipeline starts with parsing the Cargo manifest, then setup the Builder with the right flags.

let meta = parse_metadata(project);
let mut builder = GuestBuilder::new(&meta, "riscv32im-risc0-zkvm-elf", "risc0")
    .rust_flags(&[
        "passes=loweratomic",
        "link-arg=-Ttext=0x00200800",
        "link-arg=--fatal-warnings",
        "panic=abort",
    ])
    .cc_compiler(
        risc0_data()
            .unwrap()
            .join("cpp/bin/riscv32-unknown-elf-gcc"),
    )
    .c_flags(&["-march=rv32im", "-nostdlib"]);

The executor runs the desired action (e.g. cargo build or cargo test) and finally place the ELF to specified dest directory.

let executor = if !test {
    builder.build_command(profile, bins)
} else {
    builder.test_command(profile, bins)
};

executor
    .execute()
    .expect("Execution failed")
    .risc0_placement(dest)
    .expect("Failed to export Ris0 artifacts");

Test Harness

I implemented a light-weight harness to build test suits and run them in the ELF's main. The main with normal cargo test compilation is an alternative entrypoint is created that collide with the ZkVM entrypoint. The resulted test workflow looks like this:

// In program/src/main.rs
#![no_main]
sp1_zkvm::entrypoint!(main);
use harness::{zk_suits, zk_test, TestSuite};
pub fn main() {
    println!("Hello from example");

    #[cfg(test)]
    zk_suits!(test_ok, test_fail);
}

#[test]
pub fn test_ok() {
    assert_eq!(1, 1);
}

#[test]
pub fn test_fail() {
    assert_eq!(1, 2);
}

After you write tests in the guest program, compile them:

// In build.rs
pipeline::sp1::tests("../example-sp1", &["example", "foo"]);

This will gives you binaries equivalent to cargo test --no-run --bin example -- bin foo. Then in the host:

#[test]
fn test_example() {
    // Generate the proof for the given program.
    let mut client = ProverClient::new();
    let stdin = SP1Stdin::new();
    let (pk, vk) = client.setup(TEST_EXAMPLE);
    let proof = client.prove(&pk, stdin).expect("proving failed");
    client.verify(&proof, &vk).expect("verification failed");
}

[Brechtpd]

Will be merging this one already, but a few things to follow up in future PRs:

• Update CI/scripts to use the same scripts/makefile (with perhaps extra flags for special CI logic).
• maybe gitignoring the generated files because they are big and potentially hard to keep up to date (though difficulties with CI right now with not checking them in so will have to see for this)
• some other minor things

The makefile approach already available locally for building so people can already get used to that, but still needs to be extended for installing everything as well.

[CeciliaZ030]

• some other minor things

? dude what's the point of making this a point lol

[Brechtpd]

• some other minor things

? dude what's the point of making this a point lol

Minor things that I can't be bothered to type out. 🦆