Runwasi with WasmEdge Advancements

Announcement

This repository is not the official version of Runwasi. Its purpose is solely to showcase the functionalities of Runwasi with WasmEdge through demonstrations and provide users with quick access to installation package releases. If you are new to Runwasi, I would recommend reading the official README first. The following operations and explanations assume that you have a certain level of familiarity to proceed.

LLAMA2 Example

(This video is run with Intel(R) Core(TM) i7-10700 CPU @ 2.90GHz)

Download model

curl -LO https://huggingface.co/TheBloke/Llama-2-7B-GGUF/resolve/main/llama-2-7b.Q5_K_M.gguf

Install wasmedge shim with ggml plugin

sudo ctr content fetch ghcr.io/second-state/runwasi-wasmedge-plugin:allinone.wasi_nn-ggml
sudo ctr install ghcr.io/second-state/runwasi-wasmedge-plugin:allinone.wasi_nn-ggml -l -r

Login github with your PAT first

docker login ghcr.io -u <UserName>

Run llama2 with docker + wasm

docker run --rm --runtime=io.containerd.wasmedge.v1 --platform wasi/wasm \
  -v /opt/containerd/lib:/opt/containerd/lib \
  -v $PWD:/resource \
  --env WASMEDGE_PLUGIN_PATH=/opt/containerd/lib \
  --env WASMEDGE_WASINN_PRELOAD=default:GGML:CPU:/resource/llama-2-7b.Q5_K_M.gguf \
  ghcr.io/second-state/runwasi-demo:llama-simple \
  --model-alias default --ctx-size 4096 --n-predict 128 --stream-stdout \
  --prompt 'Robert Oppenheimer most important achievement is '

Output

[INFO] prompt context size: 4096
[INFO] Number of tokens to predict: 128
[INFO] Number of layers to run on the GPU: 100
[INFO] Batch size for prompt processing: 4096
[INFO] Log enable: false
[INFO] Stream stdout: true
1945, when he was the director of the Manhattan Project, which was the development of the atomic bomb. surely, this is the most important achievement of his life.
Robert Oppenheimer was born in New York City on April 22, 1904. He was the son of Julius Oppenheimer, a wealthy textile merchant, and Ella Friedman Oppenheimer. He was the youngest of three children. His father was Jewish and his mother was Protestant.
Oppenheimer was a brilliant student and

Introduce

The current installation process for the latest release is based on containerd managing opt. We maintain installable images in several ghcr.io repositories. Here, we'll briefly explain how to access them.

runwasi-wasmedge

Maintain an image with only the shim and essential libraries.

runwasi-wasmedge-plugin

This is the new plugin functionality we currently support. The installation process for plugins is similar to the one mentioned earlier (shim only), which will be explained in more detail below. The difference lies in the use of tags to differentiate functionality. In general, the naming convention is as follows:

<package>.<plugin>

• <package>
  • lib : Library only, including plugin and its dependencies.
  • allinone : In addition to the library, it also includes the Wasmedge Shim binary.
• <plugin> : suppoted plugin
  • wasi_nn-pytorch
  • wasi_nn-ggml
  • wasm_bpf (TBV)
  • wasmedge_image (TBV)
  • wasmedge_tensorflow (TBV)
  • wasmedge_tensorflowlite (TBV)
  • wasi_crypto (TBV)
  • wasi_nn-tensorflowlite (TBV)
  • wasi_nn-openvino (TBV)

Attention

1. TBV means to be verified. It has been pre-packaged, but complete integration tests have not been added yet, so there may be issues during execution.
2. This feature is still in progress, and the naming conventions may change at later.

Instructions

• To install the above packages, you can simply execute the following commands.

sudo ctr content fetch <docker_img>:<tag>
sudo ctr install <docker_img>:<tag> -l -r

• To remove it, you will need to manually delete the files under the specified path (if they exist).

rm /opt/containerd/lib/*
rm /opt/containerd/bin/*

Attention

Installation from the ctr command will overwrite the default runtime (shim) search path. So, if you want to run the shim build from source, you should first execute the previous command to remove them and then restart containerd using the command below.

sudo systemctl daemon-reload && sudo systemctl restart containerd

More Examples

We have prepared three daily run demo actions, including ctr, docker, and llama2 (preview). You can check their usage and daily action status.

Prebuilt demo image

Here, we have prebuilt demo images available. You can obtain these same images by simply running make load_demo and make load (official test case) in the root folder. They are generated based on the test cases in this directory.