Extend WASI Filesystem to support sqlite3

[tracker1]

Not sure if this the right place for this... Wanted to just suggest that it would be really nice to support enough filesystem access to get sqlite3 working to where it can work on a database also being accessed by another application process. I know it's somewhat odd to mention a specific library, but it seems that it's one of the most widely used libraries and focused around filesystem access, so being able to test it through webassembly/wasi and pass the sqlite test suite would be a massive win and prove the environment for most use cases of FS needs.

[devsnek]

It would be helpful if you elaborated on what specific functionality is missing.

[sunfishcode]

One missing feature is fcntl+F_SETLK-style file locking. We didn't add file locking APIs to WASI initially because POSIX defines them to be associated with the process, not with the file descriptor, which is problematic for implementations that don't have a one-to-one relationship between instances and host processes. However, most OS's have alternative locking APIs that we could use here. If anyone's interested in investigating this, the first step here would be to do a survey of what file locking APIs exist on popular OS's, so we can design an API that would use them.

[sunfishcode]

Compiling sqlite with

• -DSQLITE_THREADSAFE=0 due to threads
• -DSQLITE_OMIT_LOAD_EXTENSION=1 due to dlopen
• -D_WASI_EMULATED_MMAN to enable emulated mmap support

turns up the following features as being needed:

• File locking
• fchmod, which can be supported once Add functions for working with file permissions. WASI#170 lands in a snapshot
• utimes - a PR to implement this is now here: Enable support for utimes and futimesat. wasi-libc#242
• getpid - a PR to implement this is now here: Implement emulated support for getpid. wasi-libc#243
• fchown, geteuid are only used to detect when the process is running as root, and to change the database file owner to a non-root user. There's no pre-existing knob to disable these, but it's pretty easy to disable them.
• /dev/urandom - WASI has random_get, so one option would be to add an emulation to libc open to special case the string "/dev/urandom" and implement it with random_get instead. Another is to disable random in sqlite using -DSQLITE_OMIT_RANDOMNESS.

[neimanpinchas]

Anyone can point to a polyfill how we could patch until its added?

[sunfishcode]

File locking (flock style) is now added to the API in #69.

[sunfishcode]

Threads, dlopen and mmap are outside the scope of wasi-filesystem, and file locking is now added, and all the other functions above have suitable answers, so we can now close this!