Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

wasi: replaces existing filesystem apis with fs.FS #394

Merged
merged 5 commits into from
Mar 24, 2022
Merged

wasi: replaces existing filesystem apis with fs.FS #394

Show file tree
Hide file tree
Changes from 4 commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
b15d8ea
wasi: replaces existing filesystem apis with fs.FS
Mar 18, 2022
1d82a43
verify workdir works
Mar 24, 2022
faef3d8
remove a TODO
Mar 24, 2022
562cea2
done TODO
Mar 24, 2022
fb2db37
grammar
Mar 24, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 15 additions & 5 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,12 +41,22 @@ on your machine unless you explicitly allow it.

System access is defined by an emerging specification called WebAssembly
System Interface ([WASI](https://github.com/WebAssembly/WASI)). WASI defines
how WebAssembly programs interact with the host embedding them. For example,
WASI defines functions for reading the time, or a random number.
how WebAssembly programs interact with the host embedding them.

This repository includes several [examples](examples) that expose system
interfaces, via the module `wazero.WASISnapshotPreview1`. These examples are
tested and a good way to learn what's possible with wazero.
For example, here's how you can allow WebAssembly modules to read
"/work/home/a.txt" as "/a.txt" or "./a.txt":
```go
wasi, err := r.InstantiateModule(wazero.WASISnapshotPreview1())
defer wasi.Close()

sysConfig := wazero.NewSysConfig().WithFS(os.DirFS("/work/home"))
module, err := wazero.StartWASICommandWithConfig(r, compiled, sysConfig)
defer module.Close()
...
```

The best way to learn this and other features you get with wazero is by trying
[examples](examples).

## Runtime

Expand Down
45 changes: 33 additions & 12 deletions config.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,12 +5,12 @@ import (
"errors"
"fmt"
"io"
"io/fs"
"math"

internalwasm "github.com/tetratelabs/wazero/internal/wasm"
"github.com/tetratelabs/wazero/internal/wasm/interpreter"
"github.com/tetratelabs/wazero/internal/wasm/jit"
"github.com/tetratelabs/wazero/wasi"
)

// NewRuntimeConfigJIT compiles WebAssembly modules into runtime.GOARCH-specific assembly for optimal performance.
Expand Down Expand Up @@ -216,21 +216,42 @@ func (c *SysConfig) WithEnv(key, value string) *SysConfig {

// WithFS assigns the file system to use for any paths beginning at "/". Defaults to not found.
//
// Ex. This sets a read-only, embedded file-system to serve files under the root ("/") and working (".") directories:
//
// //go:embed testdata/index.html
// var testdataIndex embed.FS
//
// rooted, err := fs.Sub(testdataIndex, "testdata")
// require.NoError(t, err)
//
// // "index.html" is accessible as both "/index.html" and "./index.html" because we didn't use WithWorkDirFS.
// sysConfig := wazero.NewSysConfig().WithFS(rooted)
//
// Note: This sets WithWorkDirFS to the same file-system unless already set.
func (c *SysConfig) WithFS(fs wasi.FS) *SysConfig {
func (c *SysConfig) WithFS(fs fs.FS) *SysConfig {
c.setFS("/", fs)
return c
}

// WithWorkDirFS indicates the file system to use for any paths beginning at ".". Defaults to the same as WithFS.
func (c *SysConfig) WithWorkDirFS(fs wasi.FS) *SysConfig {
// WithWorkDirFS indicates the file system to use for any paths beginning at "./". Defaults to the same as WithFS.
//
// Ex. This sets a read-only, embedded file-system as the root ("/"), and a mutable one as the working directory ("."):
//
// //go:embed appA
// var rootFS embed.FS
//
// // Files relative to this source under appA is available under "/" and files relative to "/work/appA" under ".".
// sysConfig := wazero.NewSysConfig().WithFS(rootFS).WithWorkDirFS(os.DirFS("/work/appA"))
//
// Note: os.DirFS documentation includes important notes about isolation, which also applies to fs.Sub. As of Go 1.18,
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

took some reading to discover there are no plans for an actual chrooted FS. Until then, folks with high isolation needs will have to write their own fs.FS impl (or find some library that isn't abandoned that tries to do this!)

// the built-in file-systems are not jailed (chroot). See https://github.com/golang/go/issues/42322
func (c *SysConfig) WithWorkDirFS(fs fs.FS) *SysConfig {
c.setFS(".", fs)
return c
}

// withFS is hidden especially until #394 as existing use cases should be possible by composing file systems.
// TODO: in #394 add examples on WithFS to accomplish this.
func (c *SysConfig) setFS(path string, fs wasi.FS) {
// setFS maps a path to a file-system. This is only used for base paths: "/" and ".".
func (c *SysConfig) setFS(path string, fs fs.FS) {
// Check to see if this key already exists and update it.
entry := &internalwasm.FileEntry{Path: path, FS: fs}
if fd, ok := c.preopenPaths[path]; ok {
Expand Down Expand Up @@ -265,13 +286,13 @@ func (c *SysConfig) toSysContext() (sys *internalwasm.SysContext, err error) {
rootFD := uint32(0) // zero is invalid
setWorkDirFS := false
preopens := c.preopens
for fd, fs := range preopens {
if fs.FS == nil {
err = fmt.Errorf("FS for %s is nil", fs.Path)
for fd, entry := range preopens {
if entry.FS == nil {
err = fmt.Errorf("FS for %s is nil", entry.Path)
return
} else if fs.Path == "/" {
} else if entry.Path == "/" {
rootFD = fd
} else if fs.Path == "." {
} else if entry.Path == "." {
setWorkDirFS = true
}
}
Expand Down
33 changes: 17 additions & 16 deletions config_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ import (
"io"
"math"
"testing"
"testing/fstest"

"github.com/stretchr/testify/require"

Expand Down Expand Up @@ -58,8 +59,8 @@ func TestRuntimeConfig_Features(t *testing.T) {
}

func TestSysConfig_toSysContext(t *testing.T) {
memFS := WASIMemFS()
memFS2 := WASIMemFS()
testFS := fstest.MapFS{}
testFS2 := fstest.MapFS{}

tests := []struct {
name string
Expand Down Expand Up @@ -186,7 +187,7 @@ func TestSysConfig_toSysContext(t *testing.T) {
},
{
name: "WithFS",
input: NewSysConfig().WithFS(memFS),
input: NewSysConfig().WithFS(testFS),
expected: requireSysContext(t,
math.MaxUint32, // max
nil, // args
Expand All @@ -195,14 +196,14 @@ func TestSysConfig_toSysContext(t *testing.T) {
nil, // stdout
nil, // stderr
map[uint32]*internalwasm.FileEntry{ // openedFiles
3: {Path: "/", FS: memFS},
4: {Path: ".", FS: memFS},
3: {Path: "/", FS: testFS},
4: {Path: ".", FS: testFS},
},
),
},
{
name: "WithFS - overwrites",
input: NewSysConfig().WithFS(memFS).WithFS(memFS2),
input: NewSysConfig().WithFS(testFS).WithFS(testFS2),
expected: requireSysContext(t,
math.MaxUint32, // max
nil, // args
Expand All @@ -211,14 +212,14 @@ func TestSysConfig_toSysContext(t *testing.T) {
nil, // stdout
nil, // stderr
map[uint32]*internalwasm.FileEntry{ // openedFiles
3: {Path: "/", FS: memFS2},
4: {Path: ".", FS: memFS2},
3: {Path: "/", FS: testFS2},
4: {Path: ".", FS: testFS2},
},
),
},
{
name: "WithWorkDirFS",
input: NewSysConfig().WithWorkDirFS(memFS),
input: NewSysConfig().WithWorkDirFS(testFS),
expected: requireSysContext(t,
math.MaxUint32, // max
nil, // args
Expand All @@ -227,13 +228,13 @@ func TestSysConfig_toSysContext(t *testing.T) {
nil, // stdout
nil, // stderr
map[uint32]*internalwasm.FileEntry{ // openedFiles
3: {Path: ".", FS: memFS},
3: {Path: ".", FS: testFS},
},
),
},
{
name: "WithFS and WithWorkDirFS",
input: NewSysConfig().WithFS(memFS).WithWorkDirFS(memFS2),
input: NewSysConfig().WithFS(testFS).WithWorkDirFS(testFS2),
expected: requireSysContext(t,
math.MaxUint32, // max
nil, // args
Expand All @@ -242,14 +243,14 @@ func TestSysConfig_toSysContext(t *testing.T) {
nil, // stdout
nil, // stderr
map[uint32]*internalwasm.FileEntry{ // openedFiles
3: {Path: "/", FS: memFS},
4: {Path: ".", FS: memFS2},
3: {Path: "/", FS: testFS},
4: {Path: ".", FS: testFS2},
},
),
},
{
name: "WithWorkDirFS and WithFS",
input: NewSysConfig().WithWorkDirFS(memFS).WithFS(memFS2),
input: NewSysConfig().WithWorkDirFS(testFS).WithFS(testFS2),
expected: requireSysContext(t,
math.MaxUint32, // max
nil, // args
Expand All @@ -258,8 +259,8 @@ func TestSysConfig_toSysContext(t *testing.T) {
nil, // stdout
nil, // stderr
map[uint32]*internalwasm.FileEntry{ // openedFiles
3: {Path: ".", FS: memFS},
4: {Path: "/", FS: memFS2},
3: {Path: ".", FS: testFS},
4: {Path: "/", FS: testFS2},
},
),
},
Expand Down
43 changes: 15 additions & 28 deletions examples/file_system_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,16 +2,20 @@ package examples

import (
"bytes"
"embed"
_ "embed"
"io"
"io/fs"
"testing"

"github.com/stretchr/testify/require"

"github.com/tetratelabs/wazero"
"github.com/tetratelabs/wazero/wasi"
)

// catFS is an embedded filesystem limited to cat.go
//go:embed testdata/cat.go
var catFS embed.FS

// catGo is the TinyGo source
//go:embed testdata/cat.go
var catGo []byte
Expand All @@ -29,18 +33,13 @@ func Test_Cat(t *testing.T) {

// First, configure where the WebAssembly Module (Wasm) console outputs to (stdout).
stdoutBuf := bytes.NewBuffer(nil)
sysConfig := wazero.NewSysConfig().WithStdout(stdoutBuf)

// Next, configure a sandboxed filesystem to include one file.
file := "cat.go" // arbitrary file
memFS := wazero.WASIMemFS()
err := writeFile(memFS, file, catGo)
// Since wazero uses fs.FS, we can use standard libraries to do things like trim the leading path.
rooted, err := fs.Sub(catFS, "testdata")
require.NoError(t, err)
sysConfig.WithWorkDirFS(memFS)

// Since this runs a main function (_start in WASI), configure the arguments.
// Remember, arg[0] is the program name!
sysConfig.WithArgs("cat", file)
// Combine the above into our baseline config, overriding defaults (which discard stdout and have no file system).
sysConfig := wazero.NewSysConfig().WithStdout(stdoutBuf).WithFS(rooted)

// Compile the `cat` module.
compiled, err := r.CompileModule(catWasm)
Expand All @@ -52,24 +51,12 @@ func Test_Cat(t *testing.T) {
defer wasi.Close()

// StartWASICommand runs the "_start" function which is what TinyGo compiles "main" to.
cat, err := wazero.StartWASICommandWithConfig(r, compiled, sysConfig)

// * Set the program name (arg[0]) to "cat" and add args to write "cat.go" to stdout twice.
// * We use both "/cat.go" and "./cat.go" because WithFS by default maps the workdir "." to "/".
cat, err := wazero.StartWASICommandWithConfig(r, compiled, sysConfig.WithArgs("cat", "/cat.go", "./cat.go"))
require.NoError(t, err)
defer cat.Close()

// To ensure it worked, verify stdout from WebAssembly had what we expected.
require.Equal(t, string(catGo), stdoutBuf.String())
}

func writeFile(fs wasi.FS, path string, data []byte) error {
f, err := fs.OpenWASI(0, path, wasi.O_CREATE|wasi.O_TRUNC, wasi.R_FD_WRITE, 0, 0)
if err != nil {
return err
}

if _, err := io.Copy(f, bytes.NewBuffer(data)); err != nil {
return err
}

return f.Close()
// We expect the WebAssembly function wrote "cat.go" twice!
require.Equal(t, append(catGo, catGo...), stdoutBuf.Bytes())
}
Loading