Initial support for networking

[ktock]

Related: #90 #64

This commit adds an initial support for networking. Please also refer to ./examples/networking/ dir for the documents.

As of now, we rely on the network stack running outside of the wasm image. We use gvisor-tap-vsock as the network stack written in Go. This commit ports TinyEMU's virtio-net device to Bochs and they are patched to forward all packets from the container to gvisor-tap-vsock and vice-versa, when -net=qemu flag is provided. And gvisor-tap-vsock forwards these packets to/from the destination.

The protocol between the emulator and gvisor-tap-vsock is the following (according to containers/gvisor-tap-vsock#22 )

The protocol used is: 32bits big endian size of packet then the packet itself.

The network stack can run with the different types of setup:

Containers on browser

WebSocket-based networking

gvisor-tap-vsock runs on the host (outside of the browser) and forwards packets to/from the on-browser container. We provide a wrapper command c2w-net that enables gvisor-tap-vsock to receive packets from the container over an WebSocket port (and vice-versa). The emulators running on browser connects to that WebSocket and forwards all packets to/from c2w-net over the WebSocket port and let c2w-net forward them on the host.

On emscripten, emscripten's WebSocket networking support can be used for connecting to the WebSocket port. On WASI-on-browser configuration, our example JS wrapper (/examples/wasi-browser/) uses the browser's WebSocket API and sends/receives packets to/from a socket fd of WASI. The emulator uses sock_* API for accessing to the fd.

• pros: Container can access to anywhere accesible from the network stack daemon running on the host.
• cons: The network stack daemon needs to run on the machine and forward packets received over WebSocket.

Example

This runs the network stack on the host:

$ c2w-net --listen-ws localhost:8888 &

The following runs alpine:3.18 on browser (with WASI-on-browser configuration).

$ c2w alpine:3.18 /tmp/out-js2/htdocs/out.wasm
$ cp -R ./examples/wasi-browser/* /tmp/out-js2/ && chmod 755 /tmp/out-js2/htdocs
$ docker run --rm -p 8080:80 \
         -v "/tmp/out-js2/htdocs:/usr/local/apache2/htdocs/:ro" \
         -v "/tmp/out-js2/xterm-pty.conf:/usr/local/apache2/conf/extra/xterm-pty.conf:ro" \
         --entrypoint=/bin/sh httpd -c 'echo "Include conf/extra/xterm-pty.conf" >> /usr/local/apache2/conf/httpd.conf && httpd-foreground'

Container on browser is available on localhost:8080/?net=delegate=ws://localhost:8888.
The parameter net=delegate tells the container's JS wrapper to forward packets over the specified WebSocket port listened by c2w-net.

Fetch-API-based networking

gvisor-tap-vsock is compiled to WASM(WASI) and runs on browser. HTTP(S) proxy runs on top of the network stack and the container can perform HTTP(S) networking via that proxy. The proxy is implemented relying on the browser's Fetch API for HTTP(S) networking on browser. c2w-net-proxy is the wrapper command for gvisor-tap-vsock that adds the proxy implementation and is runnable on browser.

For HTTPS, the proxy terminates the TLS connection from the container with its self-signed certificate and re-encrypt the connection to the destination using the Fetch API. So the proxy's certificate needs to be trusted by the processes in the container (SSL_CERT_FILE envvar is pre-configured). The proxy generates its self-signed certificate and our example JS wrapper (/examples/wasi-browser/) mounts it to /.wasmenv/proxy.crt. in the container

• pros: No need to run network stack daemon on the host. Networking is done based on browser's Fetch API and follows its security configuration including CORS restriction.

• cons: Container can send only HTTP/HTTPS packets to outside of the browser. And the set of accessible HTTP/HTTPS sites is limited by the browser's security rule (e.g. limited CORS).

• Current limitations:

  • Containers can't access to sites not allowing CORS access. For example, we haven't find apt mirrors accessible from browser so the container can't run apt-get. We expect more sites will allow CORS access.
  • The proxy supports only HTTP/HTTPS and the implementation isn't mature. So it's possible that some HTTP networking fails on some cases. We'll work on support for more features.
  • The set of accesible sites might be different among browsers and the configurations.
  • WASI-on-browser container is only supported. Emscripten support is the future work.

Example

The following builds c2w-net-proxy and puts it to the document root (/tmp/out-js2/htdocs/).
Go >= 1.21 is required.

$ PREFIX=/tmp/out-js2/htdocs/ make c2w-net-proxy.wasm

The following runs debian (with curl) on browser.

$ cat <<EOF | docker build -t debian-curl -
FROM debian:sid-slim
RUN apt-get update && apt-get install -y curl
EOF
$ c2w debian-curl /tmp/out-js2/htdocs/out.wasm
$ cp -R ./examples/wasi-browser/* /tmp/out-js2/ && chmod 755 /tmp/out-js2/htdocs
$ docker run --rm -p 8080:80 \
         -v "/tmp/out-js2/htdocs:/usr/local/apache2/htdocs/:ro" \
         -v "/tmp/out-js2/xterm-pty.conf:/usr/local/apache2/conf/extra/xterm-pty.conf:ro" \
         --entrypoint=/bin/sh httpd -c 'echo "Include conf/extra/xterm-pty.conf" >> /usr/local/apache2/conf/httpd.conf && httpd-foreground'

Container on browser is available on localhost:8080/?net=browser.
The set of accesible HTTP/HTTPS sites is limited by the browser's security rule (e.g. limited CORS).

Containers on WASI runtime

wasmtime and wazero are supported as of now. Other runtimes (including wamr #64) will be future works.

gvisor-tap-vsock runs on the host (outside of runtime) and forwards packets to/from the container. c2w-net is a wrapper command for gvisor-tap-vsock that connects gvisor-tap-vsock and the container. The emulator sends/receives packets to/from a socket provided by WASI runtime, using sock_* API. WASI runtime binds the socket on a TCP port (e.g. using wasmtime's --tcplisten flag and wazero's WithTCPListener option). c2w-net running on the host connects to that port and forwards the container's packets to/from the network stack.

The following runs alpine:3.18 on wasmtime with enabling networking. (Requirement: wasmtime needs to be installed)

$ c2w alpine:3.18 /tmp/out/out.wasm
$ c2w-net --invoke /tmp/out/out.wasm --net=qemu sh
connecting to NW...
INFO[0001] new connection from 127.0.0.1:1234 to 127.0.0.1:50470 
/ # apk update && apk add --no-progress figlet
apk update && apk add --no-progress figlet
apk update && apk add --no-progress figlet
fetch https://dl-cdn.alpinelinux.org/alpine/v3.18/main/x86_64/APKINDEX.tar.gz
fetch https://dl-cdn.alpinelinux.org/alpine/v3.18/community/x86_64/APKINDEX.tar.gz
v3.18.3-149-g8225da85c11 [https://dl-cdn.alpinelinux.org/alpine/v3.18/main]
v3.18.3-151-g6953e6f988a [https://dl-cdn.alpinelinux.org/alpine/v3.18/community]
OK: 20071 distinct packages available
(1/1) Installing figlet (2.2.5-r3)
Executing busybox-1.36.0-r9.trigger
OK: 8 MiB in 16 packages
/ # figlet hello
figlet hello
figlet hello
 _          _ _       
| |__   ___| | | ___  
| '_ \ / _ \ | |/ _ \ 
| | | |  __/ | | (_) |
|_| |_|\___|_|_|\___/ 
                      

• NOTE
  • Example with wazero is at ./tests/wazero/. And the docs available at ./examples/networking/wasi/README.md.