Switch to ZenFS

[oeway]

Hi, I am very much interested in accessing the entire daedalOS file system from pyodide/python, for that we will need to make the browserfs backends synchrous.

Interstingly, it looks like now browserfs has been deprecated and ZenFS fork is recommended. After a quick glance, it seems ZenFS has some really nice feature such as all backends are synchronous, and it has experimental support for the Emscripten backend. Most importantly, ZenFS is now actively maintained!

[DustinBrett]

Hi @oeway,

Thanks for the suggestion but I have followed the evolution of BrowserFS to ZenFS and I don't believe it's something I'd like to adopt. If I did switch the file system I would likely make my own from scratch as I have interest in this.

I don't agree that having the file system linked to pyodide would require large changes to the way it works and I believe an adapter could be made if someone wanted to do such a thing.

The version of BrowserFS I use is highly customized so in that way it's not deprecated, it's just maintained as a fork by me which is internal to daedalOS. BrowserFS has also always had sync and async, but the way I have mine setup it utilizes mostly async functionality.

I also have an Emscripten backend which was also part of BrowserFS and is built into daedalOS. I have a hook which handles this (useEmscriptenMount) and is used by several apps (7 usages of hook) which themselves mount Emscripten file systems.

[DustinBrett]

I played around with it a bit tonight but it would take time to make something. Possibly as pyodide supports Emscripten File Systems the BrowserFS one would have to mirror the existing IDB one.

Information on the Pyodide FS is here:
https://pyodide.org/en/stable/usage/file-system.html

As for FS mirrors, there are examples like AsyncMirror in the old BrowserFS docs for taking a file system like IndexedDb and changing it:
https://jvilk.com/browserfs/1.4.1/classes/_backend_asyncmirror_.asyncmirror.html

[oeway]

Hi @DustinBrett Thanks for looking into this! Sounds like you have a good understanding to this. Honestly, I have no idea what it means to switch to ZenFS.

As I understand, we can use AsyncMirror to map async fs (e.g. IndexedDb) to memory and mount it as Emscripten File Systems, however, it will require copy all the files into memory during initialization. This is not ideal since I would like to deal with large file system which will exceed the size of the memory.

On the other hand, we can create a emscripten file system which map all the async operation to sync directly, this method has been used by jupyterlite, see pyodide-kernel and drivefs.

Would it make sense to create something like a drivefs, then map all the operations to browserfs?

[DustinBrett]

I think you are on the right track with the code you referenced. Another option could be to emulate File System Access API and use mountNativeFS from Pyodide.

I don't think you would necessarily need to choose between sync or async, but you might need to do some tricks or a hollow mirror type of magic to make pyodide not really request the file until it needs it while still showing a full directory structure.

I don't have any good ideas at this moment but it would probably be easiest to write a custom backend/mirror into BrowserFS for this use case.

You can make an async into a sync one by blocking the main thread, which is not ideal but could work for your use case.

[oeway]

Sounds good!

I have been using mountNativeFs in pyodide, I should say that is has also some issue since it works similar to asyncMirror, the operation we did, e.g. create files in pyodide won't be reflected immediately on disk, instead we need to manually call a sync function from and to disk.

So I think we should perhaps go for the drivefs approach, ideally make it work with all endpoints mounted to browserfs ( I have a prototype with s3 and custom file server backend).

Would you be interested in giving it a try at some point? I could do that too, but I am not efficient in TS and not familiar with the daedalOS code base.

[DustinBrett]

Perhaps at some point I could indeed look into it. If you decide to try yourself I also still think the nativeFS option could work, in which you make an adapter from my daedalOS filesystem to a File System API object which you custom make.

The code in Pyodide for checking it and using it doesn't seem so hard, but you'd need to cover whatever functions it uses at least.

https://github.com/pyodide/pyodide/blob/main/src/js/nativefs.ts