To build and run one of the Oak example applications under Docker, run (after installing Docker if necessary):
./scripts/docker_pull # retrieve cached Docker image for faster builds
./scripts/docker_run xtask --logs --scope=all run-oak-functions-examples --example-name=weather_lookup --client-variant=rustThis should build the Runtime, an Oak Functions Application and a client for the Application, then run them all, with log output ending something like the following:
[21:03:26; E:0,O:0,R:1]: oak-functions examples ❯ weather_lookup ❯ run ❯ background server ❯ run clients ❯ rust ❯ build ⊢ Compiling oak_functions_client v0.1.0 (/workspace/oak_functions/client/rust)
Finished release [optimized] target(s) in 15.22sk_functions_client(bin)
OK [15s]
[21:03:42; E:0,O:0,R:1]: oak-functions examples ❯ weather_lookup ❯ run ❯ background server ❯ run clients ❯ rust ❯ run ⊢ Finished release [optimized] target(s) in 0.12s
Running `target/x86_64-unknown-linux-gnu/release/oak_functions_client '--uri=http://localhost:8080' '--request={"lat":0,"lng":0}' '--expected-response-pattern=\{"temperature_degrees_celsius":.*\}'`
{"temperature_degrees_celsius":29}
OK [1s]
[21:03:26; E:0,O:0,R:1]: oak-functions examples ❯ weather_lookup ❯ run ❯ background server ❯ run clients ❯ rust } ⊢ [OK] [17s]
[21:03:26; E:0,O:0,R:1]: oak-functions examples ❯ weather_lookup ❯ run ❯ background server ❯ run clients } ⊢ [OK] [17s]
[21:03:20; E:0,O:0,R:1]: oak-functions examples ❯ weather_lookup ❯ run ❯ background server ⊢ (waiting)
[21:03:20; E:0,O:0,R:1]: oak-functions examples ❯ weather_lookup ❯ run ❯ background server ⊢ (finished) OK
[21:03:20; E:0,O:0,R:1]: oak-functions examples ❯ weather_lookup ❯ run } ⊢ [OK] [25s]
[21:03:20; E:0,O:0,R:1]: oak-functions examples ❯ weather_lookup } ⊢ [OK] [25s]
[21:03:20; E:0,O:1,R:0]: oak-functions examples } ⊢ [OK] [25s]
Note: ./scripts/docker_pull and ./scripts/docker_run will need sudo if not
configured otherwise.
The remainder of this document explores what's going on under the covers here, allowing individual stages to be built and run independently.
The simplest way to get up and running with a development environment that contains all the required tools is to use VS Code with the Remote-Containers extension. After this is installed, and VS Code is running and pointing to the root of the Oak repository, from a separate terminal build the Docker image, if you haven't already:
./scripts/docker_pullThen from VS Code click on the Remote-Containers button in the bottom left corner of the status bar:
and then select "Remote-Containers: Reopen in Container".
This should attach VS Code to an instance of the Docker container with all the dev tools installed, and configured with most linters and Rust tools.
The Rust-Analyzer extension may prompt you to download the rust-analyzer server, in which case allow that by clicking on "Download now" in the notification.
To test that things work correctly, open a Rust file, start typing std::
somewhere, and autocomplete results should start showing up. Note that it may
take a while for the rust-analyzer extension to go through all the local code
and build its index. Alternatively, try xtask run-tests.
On Linux you might have to
post-installation steps for Linux
and run systemctl start docker. If you get a Permission denied try to
rebuild the Docker images with ./scripts/docker_build.
For any open-source project, the best way of figuring out what prerequisites and dependendencies are required is to look at the projects continuous integration (CI) scripts and configuration.
For Oak, the CI system is currently
Google Cloud Build, so the top-level
cloudbuild.yaml file holds the project's definitive
configuration, dependencies and build scripts.
The Cloudbuild setup for the project relies on a project
Dockerfile to capture precise toolchain dependencies and
versions. This also allows local use of Docker to match this build environment,
using the following wrapper scripts:
scripts/docker_pullretrieves the project's most recent cached Docker image(s).scripts/docker_runruns its arguments within Docker, so can be used as a prefix for any commands described later in this document.scripts/docker_shruns an interactive shell within Docker. This can also be used to run any commands described later in this document.
If you want to work on the project without using Docker, you should synchronize
your installed versions of all the tools described in the next section with the
versions installed in the Dockerfile. This path is not
recommended, since it requires a lot of effort to manually install all the tool
natively on your machine and keeping them up to date to the exact same version
specified in Docker.
xtask is a utility binary to perform a number of common tasks within the Oak
repository. It can be run by invoking ./scripts/xtask from the root of the
repository, or also directly xtask, and it has a number of flags and
sub-commands available, which should be self-explanatory, and it also supports
flag autocompletion when invoked from inside a Docker shell.
For commands that use cargo, by default the xtask runs the command only for
the modified files and the crates affected by those changes. Use --scope=all
to run the command for the entire code base.
Running one of the example Oak applications will confirm that all core prerequisites have been installed. Run one inside Docker with:
xtask --logs --scope=all run-oak-functions-examples --example-name=weather_lookup --client-variant=rustThat script:
- builds the Oak Functions Runtime (in Rust, built to run on the host system)
- builds a particular example, including both:
- the Oak Application itself (Rust code that is compiled to a WebAssembly binary)
- an external client (Rust code built to run on the host system)
- starts the Runtime as a background process, passing it the compiled WebAssembly for the Oak Functins Application (which it then runs in a WebAssembly engine)
- runs the external client for the Application
- closes everything down.
Those steps are broken down in the following subsections; this helps figure out where the problem is if something goes wrong.
The following command compiles the code for an example Oak Application from Rust to a WebAssembly module and then serializes it into a binary application configuration file to be loaded to the Oak Functions Server:
xtask --logs --scope=all run-oak-functions-examples --example-name=weather_lookup --client-variant=none --run-server=falseThis binary application configuration file includes the compiled Wasm code for the Oak Functions Application.
The following command builds the Oak Functions Runtime server. An initial build will take some time, but subsequent builds should be cached and so run much faster.
xtask build-oak-functions-server-variantsThe following command builds and runs an Oak Functions Server instance, running a specific Oak Application (which must already have been compiled into WebAssembly, as described above.
xtask --scope=all --logs run-oak-functions-examples --example-name=weather_lookup --client-variant=noneIn the end, you should end up with an Oak server running, end with log output something like:
2022-02-23T21:14:39Z INFO - refreshing lookup data from HTTP: https://storage.googleapis.com/oak_lookup_data/lookup_data_weather_sparse_s2 with auth None
2022-02-23T21:14:39Z INFO - fetched 8507683 bytes of lookup data in 626ms
2022-02-23T21:14:40Z INFO - parsed 143548 entries of lookup data in 102ms
2022-02-23T21:14:40Z DEBUG - lookup data write lock acquisition time: 709ns
2022-02-23T21:14:40Z INFO - ThreadId(3): Starting gRPC server on [::]:8080
The following command (run in a separate terminal) compiles the code for the client of an example Oak Application (as described above), and runs the client code locally.
xtask --scope=all --logs run-oak-functions-examples --example-name=weather_lookup --run-server=false --client-variant=rustThe client should run to completion and give output something like:
[21:16:17; E:0,O:0,R:1]: oak-functions examples ❯ weather_lookup ❯ run ❯ run clients ❯ rust ❯ build ⊢ Finished release [optimized] target(s) in 0.11s
OK [237ms]
[21:16:17; E:0,O:0,R:1]: oak-functions examples ❯ weather_lookup ❯ run ❯ run clients ❯ rust ❯ run ⊢ Finished release [optimized] target(s) in 0.14s
Running `target/x86_64-unknown-linux-gnu/release/oak_functions_client '--uri=http://localhost:8080' '--request={"lat":0,"lng":0}' '--expected-response-pattern=\{"temperature_degrees_celsius":.*\}'`
{"temperature_degrees_celsius":29}
OK [1s]
[21:16:17; E:0,O:0,R:1]: oak-functions examples ❯ weather_lookup ❯ run ❯ run clients ❯ rust } ⊢ [OK] [2s]
[21:16:17; E:0,O:0,R:1]: oak-functions examples ❯ weather_lookup ❯ run ❯ run clients } ⊢ [OK] [2s]
[21:16:17; E:0,O:0,R:1]: oak-functions examples ❯ weather_lookup ❯ run } ⊢ [OK] [2s]
[21:16:17; E:0,O:0,R:1]: oak-functions examples ❯ weather_lookup } ⊢ [OK] [2s]
[21:16:17; E:0,O:1,R:0]: oak-functions examples } ⊢ [OK] [2s]
We currently have fuzz testing enabled for Oak Functions on
OSS-Fuzz. In
addition, the xtask has a command for running fuzz targets run-fuzz-targets.
This command runs cargo-fuzz with the -O option for optimization, and
supports all libFuzzer options. These options must be provided as the last
argument. For instance, the following command runs all fuzz targets with a 2
seconds timeout for each target.
xtask run-cargo-fuzz -- -max_total_time=2The following lists all the libFuzzer options:
xtask --logs run-cargo-fuzz -- -help=1Moreover, crate-name alone or together with target-name could be specified
to run all targets for a specific crate, or to run a specific target,
respectively.
xtask --logs run-cargo-fuzz --crate-name=loader --target-name=wasm_invoke -- -max_total_time=20