Run x86_64 containers on WASI/browser using Bochs

[ktock]

This commit adds an experimental support for running x86_64 containers on WASI/browser leveraging Bochs emulator. This improves the performance of x86_64 containers on WASM by eliminating guest-side emulation (user-space qemu with binfmt in guest VM).

• This commit ports Bochs to WASM. Bochs relies on setjmp/longjmg and on emscripten, they are enabled by default. On WASI, we rely on Asyncify for implementing "user-space" emulation of setjmp/longjmp like done by Alon Zakai (Binaryen creator) and done by Ruby's WASM support.
• This commit uses Wizer for experimental pre-booting of linux kernel at build time. During runtime, container can startup with eliminating the boot sequence of linux.
• This commit introduces an initial WASI-on-browser suppprt using browser_wasi_shim. This is because booting linux+container on Bochs takes long (>= 30s) so we want to apply Wizer's pre-initialization optimization for on-browser images as well. This allows us running the container on the pre-booted linux on browser, eliminating the boot sequence of linux.

Running on WASI runtimes

$ c2w ubuntu:22.04 out.wasm
$ wasmtime out.wasm uname -a
Linux localhost 6.1.0 #1 PREEMPT_DYNAMIC Mon Jun  5 11:57:09 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

WASI runtime's directory mapping is unsupported by x86_64 containers as of now.
Non-x86_64 containers (based on TinyEMU) still supports directory mappings.

Running on browser

There are two methods for running the container on browser.

WASI-converted image on browser

The WASI-converted image can run on browser leveraging browser_wasi_shim.

• pros: Faster image startup by eliminating boot sequence of linux, leveraging Wizer .
• cons: WASM image size can be bigger than emscripten.

$ c2w ubuntu:22.04 /tmp/out-js2/htdocs/out.wasm
$ cp -R ./examples/wasi-browser/* /tmp/out-js2/ && chmod 755 /tmp/out-js2/htdocs
$ docker run --rm -p 8080:80 \
         -v "/tmp/out-js2/htdocs:/usr/local/apache2/htdocs/:ro" \
         -v "/tmp/out-js2/xterm-pty.conf:/usr/local/apache2/conf/extra/xterm-pty.conf:ro" \
         --entrypoint=/bin/sh httpd -c 'echo "Include conf/extra/xterm-pty.conf" >> /usr/local/apache2/conf/httpd.conf && httpd-foreground'

emscripten image on browser

--to-js flag still supports creating emscripten-compiled image.

$ c2w --to-js ubuntu:22.04 /tmp/out-js/htdocs/
$ cp -R ./examples/emscripten/* /tmp/out-js/ && chmod 755 /tmp/out-js/htdocs
$ docker run --rm -p 8080:80 \
         -v "/tmp/out-js/htdocs:/usr/local/apache2/htdocs/:ro" \
         -v "/tmp/out-js/xterm-pty.conf:/usr/local/apache2/conf/extra/xterm-pty.conf:ro" \
         --entrypoint=/bin/sh httpd -c 'echo "Include conf/extra/xterm-pty.conf" >> /usr/local/apache2/conf/httpd.conf && httpd-foreground'

TODOs

• Add demo
• We currently implements guest-emulator communication (guest notifying boot completion to the emulator during build) using the virutal FTP server which is very hacky so we should rewrite the guest-emulator communication in a cleaner way (e.g. virtio?).

[ktock]

We can see the performance improvement of x86_64 containers: https://github.com/ktock/container2wasm/actions/runs/5200577866/jobs/9379484009#step:3:4361

• echo hello: 6.66s(ff4cbd on main branch) -> 5.01s(this PR)
• sha256sum: 13.8s(ff4cbd on main branch) -> 5.26s(this PR)

NOTE: The main branch implemented x86_64 emulation by TinyEMU + QEMU(in the guest VM). This PR implements the emulation using Bochs emulator without QEMU in the guest VM.

(Measured on GitHub Actions ubuntu-22.04 runner)

goos: linux
goarch: amd64
pkg: github.com/ktock/container2wasm/tests/benchmark
cpu: Intel(R) Xeon(R) Platinum 8171M CPU @ 2.60GHz
BenchmarkHello/wasmtime-hello,alpine:3.17-2         	       3	5016648862 ns/op
--- BENCH: BenchmarkHello/wasmtime-hello,alpine:3.17-2
    benchmark_test.go:180: test root: /tmp/testc2w3079079573
    benchmark_test.go:180: test root: /tmp/testc2w2453526445
BenchmarkHello/wasmtime-sha256sum,alpine:3.17-2     	       3	5268534640 ns/op
--- BENCH: BenchmarkHello/wasmtime-sha256sum,alpine:3.17-2
    benchmark_test.go:180: test root: /tmp/testc2w3830332443
    benchmark_test.go:180: test root: /tmp/testc2w1198803430
BenchmarkHello/wasmtime-hello-legacy,alpine:3.17,--target-arch=amd64-2         	       3	6663733730 ns/op
--- BENCH: BenchmarkHello/wasmtime-hello-legacy,alpine:3.17,--target-arch=amd64-2
    benchmark_test.go:180: test root: /tmp/testc2w438708640
    benchmark_test.go:180: test root: /tmp/testc2w246183600
BenchmarkHello/wasmtime-sha256sum-legacy,alpine:3.17,--target-arch=amd64-2     	       3	13831033272 ns/op
--- BENCH: BenchmarkHello/wasmtime-sha256sum-legacy,alpine:3.17,--target-arch=amd64-2
    benchmark_test.go:180: test root: /tmp/testc2w2826949912
    benchmark_test.go:180: test root: /tmp/testc2w3878067291

[ktock]

Thanks @AkihiroSuda for introducing me Bochs emulator!