Expose the FS interface

[wjam]

Is your feature request related to a problem? Please describe.
The way filesystems were accessed by WASM modules was recently changed (15dc3d7#diff-e1cd45f905e175c6cc7e0298d2fa2efcf732d7182465e7e73614c10132e5a3a5 I believe, but certainly between v1.0.0-pre.5...v1.0.0-pre.6). This change altered what was needed to get a writeable filesystem - needing to implement the FS interface. For example, this interface would need to be implemented if you wanted to certain files to come from an embedded fs.FS or if you wanted certain files to be immutable.

The problem is that the wazero FS interface lives in an internal/ package and so encourages people to not implement this interface.

Describe the solution you'd like
Move the interface somewhere more public so that it can be discovered by developers and implemented.

Describe alternatives you've considered

• Keep using a plain fs.FS - this won't work because attempting to create files is blocked by

  wazero/internal/sys/fs.go

  Line 312 in 4197caa

  case flags&os.O_CREATE != 0:

Additional context
Add any other context or screenshots about the feature request here.

[achille-roussel]

Hello @wjam!

I am not part of the core maintainer team of Wazero but I have been in touch with @codefromthecrypt in the #wazero-dev Slack channel on this topic.

I believe that the goal is eventually to expose the FS interface, but at this time the code in internal/syscallfs isn't stable enough yet to allow external programs to take a dependency on it.

[codefromthecrypt]

@wjam thanks for the feedback!

Currently, the internal filesystem is very much not encouraged to be implemented. It is changing almost every day, so not ready to consider exposing. Moreover, no one should think too deeply about its current form given that it is changing.

What is currently supported until our internal interface is done.

At a high level, we currently only support fs.FS and an existing directory as entrypoints. As you know fs.FS to create new files is a hack around a lack of an alternative. We can add an unsafe FS adapter meanwhile to help you along. I'll raise one even though the goal is to get the real filesystem together this month.

Why we are focusing on syscalls and when do we think we are done our internal interface

More context is we are working on first completing all syscalls needed for GO and WASI filesystem ABI, both are near 1-1 mapping with syscall apis inside go or x/sys. Both ABI are very coupled at the syscall layer, and we have to be careful to make sure how we approach maps correctly as things like errors panic/trap if interpreted incorrectly. This means the eventual completion of the internal be at a syscall abstraction, with at least two high-level entrypoints: fs.FS and an existing directory. The latter will allow full write support and the former will only allow what's possible in fs.FS. We may possibly test that you can implement it with x/sys as well, but that's not a hard requirement.

Once the internal interface exists and passes all tests on tinygo and zig with our internal one, on Linux darwin and windows, as well tests in https://github.com/WebAssembly/wasi-testsuite we will likely expose it, still at a syscall abstraction. We won't settle on any abstraction decisions until all this happens/

On a non-syscall abstraction

After all the tests pass, we may try to develop a more ergonomic non-syscall focused adapter after that point. However, this will not happen without cause. We need to know concrete use cases for what isn't an fs.FS and what isn't an os.File, and why x/sys also isn't ok.

In other words, wazero is not a filesystem abstraction project, and there are multiple incompatible filesystem interfaces out there, many of which may be able to match what we eventually end up with, and some not.

The only way we will make an abstraction beyond syscalls is concrete examples of things possible to make in open source which neither fit fs.FS, os.File and x/sys. These would not just be examples, but literally what end users would use.

If anyone has a specific thing they are doing now, e.g. an external FS lib that they would use if they could, please mention it. Note: that we will not do an abstraction solely to look more like os.File or anything, as abstraction is not its own goal. When mentioning use cases, please limit feedback to what implementation requirements are, as that conserves time for the work we need to complete.

[codefromthecrypt]

@wjam #1015 should plug the fs.FS hole, but let's leave this issue open until our filesystem abstraction is complete. In the mean time, if you can let us know why you still need to do the hack given we now have writefs.DirFS, it would be insightful. For example, maybe it is because you are doing something besides operating system files? Anyway if you can, please let us know and thanks for reporting the break.

[wjam]

@codefromthecrypt The root filesystem for the WASM module is made of up of other fs.FS, just like the compositeFS. Some directories will contain the inputs for the module, which are expected to be read-only, and some directories will potentially contain files generated by the any relevant output, so does have to be writeable. The contents of the input directories may be large and so don't want to copy into a new temporary directory. I also can't think of a way to use writefs.DirFS while keeping certain files unmodifiable without having to run the WASM under a different user who doesn't have permission to modify those files.

[codefromthecrypt]

@wjam thanks for the feedback! FWIW an upcoming change will support the pre-open concept used in WASI where you have an ordered list of mounts which individually will be some readable and some not. For us to finish our zig and tinygo tests, we have to support this (e.g. to mount TMPDIR write, some directories read and others write). So, for things backed by real filesystem there will be a supported way to do that. Would that solve you, or do you have mounts that are not backed by real directories?

[codefromthecrypt]

@wjam fyi while only a configuration API, this should help folks needing basic composition. We'll be working on exporting the underlying sysfs after it is completely functional and cleaned up (post-1.0), #1061

[codefromthecrypt]

@samyfodil you mentioned you use https://github.com/spf13/afero right? Any notes about what features of that you feel are important to you in wasm?

[shynome]

A usage description:
I want to add /dev/tcp/127.0.0.1/5432 virtual file system to provide network access, for this I need io.Writer interface

[codefromthecrypt]

@shynome for this one you should be able to do fs.FS correct? e.g. in normal os.DirFS the fs.File returned from open implements writer

[shynome]

Yes, I have implemented fs.FS correctly, but now fs.File does not expose io.Writer interface, so now I have implemented tcp read, I can implement tcp write when wazero.FS API is stable.
Hope wazero.FS arrives soon

[codefromthecrypt]

@shynome sorry what I mean to say is that we cast fs.File to io.Writer already because normal fs.File implementations already implement that (e.g. *os.File). IOTW you can implement this ahead of us exposing a raw interface, and fs.FS will be easier to implement than our raw one.

[shynome]

@codefromthecrypt ohhhhh, it is working. my fault, my program has wrong

[codefromthecrypt]

sorry about taking so long. This will be experimental for at least one minor release before formalizing into the sys package #1605