wasi: allow users to configure a filesystem that can accept new files

[codefromthecrypt]

WASI defines portable system calls to manipulate files. We currently use fs.FS to map to many of these. read-only operations are implicit, but the status quo goes beyond this in the standard library.

For example, fs.FS can write existing files as implementations are free to go beyond the functionality defined on fs.File. Some of this is encouraged. Concretely, os.File goes beyond with mutations such as chmod and write. os.FS

Also, there are standard extensions to fs.FS including fs.ReadDirFS. The point is that in many cases, there is an existing mapping of a go interface to WASI functions, even if it isn't required by fs.FS. By safe casts, you can get most functionality needed.

However, there are gaps, notably creating new files. This issue is solved when the core WASI APIs requested by end users have a corresponding implementation. This is ideally done without introducing any wazero-specific interfaces. As such research is needed and validation is needed prior to introducing an interface not in Go's standard library.

Moreover, as the README suggests, completion of all WASI functions is a non-goal. In other words, if there is a tricky API, and hasn't been asked directly by end-users yet.. wait until that's the case, then solve it. WASI is completely changing its filesystem API so we can conserve energy by only focusing on what users need today.

Solving this issue is a high priority and 1.0 will be held back until it is solved. However, it is likely this will be solved way before 1.0, like in a week or two.

References:

• Go has an issue tracking approaches to weave in write support to fs.FS proposal: io/fs: add writable interfaces golang/go#45757
• Last year, @Gaboose added a toe-hold implementation of many WASI features including a MemFS WASI: Add stdio and file system support #22 This was maintained over time, but we completely redid our API. Reads are now via fs.FS despite limitations of that.
• Recently, @nullpo-head attempted to make an extension of fs.FS in WASI: fs.FS as Preopen parameter with implementation of fd_prestat_get #362 with some custom APIs to add write support to it. This was closed out because it coupled read-side with writes and the former is easier to implement. This issue was raised to carry that forward.
• proposal: io/fs: add writable interfaces golang/go#45757 is a gold mine of insight both in what people are thinking about and their use cases. One example of a linked issue is interp: Initial support for fs.FS. mvdan/sh#799 which summarized to thinking that sharing the same interface of fs.FS for writes may be faulty. This is the main reason why this is a separate issue, to do thorough research and consider a completely different path for exposing writes as needed, even if internally we share a base fs.FS impl.

[codefromthecrypt]

I completely rewrote the description as the gap here is much smaller than I originally thought. While fs.FS doesn't specify extended APIs it is routine in practice to already support writes (ex os.File).

[mmathys]

Does that mean that it is possible to write new files from Wasm?

I have tried using the open syscall in WASI for non-existing files with writing mode, but it fails.

[codefromthecrypt]

@mmathys exactly. in fact this it is about time we got back to this one. if you don't mind, as it is usually interesting.. what's the use case you are working with that needs to open for writing? is it a log or config file? or are you sharing data?

[mmathys]

We are using files to keep state between function calls when writing functions in C.

[codefromthecrypt]

@ncruces noted in #740 that there's a corner case in the io.Reader docs hard to reason with when mapping to io.Reader based implementation of fdread

Callers should always process the n > 0 bytes returned before considering the error err. Doing so correctly handles I/O errors that happen after reading some bytes and also both of the allowed EOF behaviors.

https://pkg.go.dev/io#Reader

The "process the n > 0 bytes returned before considering the error err" ends up a bit dodgy as you have a choice to ignore that part or create expense by holding per-fd state to return the error when >n bytes are read.

[ncruces]

Looking at the issue and current implementation, I don't think there's anything missing from wazero in this regard.

For instance, if you use os.DirFS you'll get a read-only view into a directory and, from what I could tell, you won't be able to create or modify any files.

But you don't have to use os.DirFS, you can create a copy of it that does OpenFile with O_RDWR here, then with the current implementation you'll have writeable files in your WASI.

Or you can create an implementation that, instead or returning fs.ErrNotExist for a missing file, os.Creates one. With this, you should already able to write new files, no?

Or you don't create an actual os.File, but your own “file” object, that's a thin wrapper around a bytes.Buffer. Then you get memory backed writeable files.

Now, all this has subtleties, but it seems doable, and not something wazero needs to provide out-of-the-box.

As an example, for my usage I wanted to lock my WASM even harder, so I created a file system that can only ever open a single file (not an entire directory).

[mmathys]

To fix this, I just replaced instances of

f, err := c.fs.Open(fsOpenPath)

by

f, err := os.OpenFile(fsOpenPath, os.O_APPEND|os.O_CREATE|os.O_RDWR, 0644)

and I get a writable FS.

[codefromthecrypt]

The first way we'll implement this is via a new ModuleConfig which allows you to pass a path instead of an fs.FS. When that's the case, we'll use os.OpenFile after coercing WASI flags to go ones. This will happen by our next pre-release, and likely in the next day or two.

[codefromthecrypt]

https://github.com/tetratelabs/wazero/releases/tag/v1.0.0-pre.6 adds writefs.NewDirFS, which lets you add new files. We'll track pluggability of this in another issue.

[paralin]

This is awesome!

[mmathys]

Great work!

[karelbilek]

If someone is looking for it now and hits this issue and wonders, where it is; it's in

github.com/tetratelabs/wazero/experimental/sys

https://pkg.go.dev/github.com/tetratelabs/wazero@v1.6.0/experimental/sys#FS

wazero/experimental/sys/fs.go

Line 29 in 08998ed

type FS interface {

[paralin]

See #2076