Add WebTorrent VFS

[ydylla]

Hi @rhashimoto and thanks for your awesome work,
I have implemented a WebTorrent based read only VFS. Do you want to add it to the examples?

It will probably never be really fast but being able to "serverless" host a database is very interesting. In combination with a webseed this is also very similar to sql.js-httpvfs (Thanks @phiresky by the way for introducing me into this topic).

To make sure the demo always works it also uses a webseed (the GOOG.db file). To truly test the torrenting you have to open a new tab at https://instant.io/ and add the GOOG.db, which then acts as another peer. Also the adding of the webseed url has to be removed in demo/index.js:L64. When using a magnetUri you may have to increase the torrent ready timeout option of WebTorrentVFS, because loading the meta information from peers is rather slow.

[rhashimoto]

That's cool!

This seems like something that belongs in its own repo. I'd like to limit the scope of this project to enabling extensions like yours, and not so much to be their home. I'm just not prepared to support contributed code that I may not completely understand, and I haven't figured out what I want to do about CLAs.

I would definitely like to link to your VFS, though. If that's okay with you then can you give me a sentence or two of accompanying description and where you want the link to go? They can be updated later if necessary, e.g. I'd be happy to link to your fork now and a standalone repo later.

[ydylla]

I totally understand that. It is not really desirable to add a webtorrent dependency to wa-sqlite.
I added it as PR because I don't consider myself a javascript developer and thus have no experience with publishing or maintaining a javascript package. But I guess this is a good opportunity to learn it 😄

Do you think it is possible/feasible to create a wa-sqlite-base package that basically only contains VFS.js (the abstract base vfs and consts). That way vfs implementations could easily depend on it to get a typed interface they need to implement. The full wa-sqlite is then only required for tests or the actual user of the vfs. Since the vfs api is very stable this package probably will rarely change (or require maintenance).

I will try to setup a wa-sqlite-webtorrent package in the next days or on the weekend. Or do you have a better name suggestion?

P.S. I also noticed some sqlite constants are duplicated between VFS.js and sqlite-api.js. This could also be resolved by having all consts in wa-sqlite-base.

[rhashimoto]

Javascript is not my primary language either, which is why I'm not great with the tooling. 😀 The project is not on npm, so you'll need to use a git dependency. I would recommend using a "buildless" branch or tag instead of the master branch. I haven't been good about versioning and tagging; I'll try to be better going forward.

Originally VFS.js was considered example code, and it really still is just a convenience to help with implementing a VFS. There's no requirement to subclass from it; any object that has the appropriate properties will work. If you would rather bypass subclassing or copy VFS.js into your own tree, those are also options. I'm not clear on the advantages of a standalone VFS base class package - it seems to me that you would need the wrapper API at least as a devDependency for testing, and users will need the API to incorporate it.

Your repo/package name is totally up to you.

The duplication of constant definitions is indeed annoying. I'll see about changing that.

[ydylla]

Hi @rhashimoto,
after 2 days of javascript pain I finally found a setup I kind of like 😄

I have created 3 repos:
In wa-sqlite-base is a typescript interface for the vfs api and an abstract base vfs implementation.
wa-sqlite-webtorrent contains my webtorrent vfs and only has wa-sqlite-base as dependency.
wa-sqlite-webtorrent-demo is a Svelte-Kit based demo site very similar to your demo.

I still think that wa-sqlite-base could be useful for other vfs implementations. Especially since I added an abstract name function and a close function to the SQLiteVFS interface. My intention for that was, that a name function is much more obvious than requiring that a name property exists. But for that to work properly, it would be nice if you could switch vfs.name in libvfs.js#L12 and libvfs.js#L25 to vfs.name(). Or get rid of vfs.name altogether and add the name as an argument.

The idea behind close() was that all vfs's should have a defined cleanup function which could make the usage more similar/consistent.

A bit offtopic, have you considered switching to typescript? I found it easier to write typescript and have the .d.ts files get auto generated by the compiler. No need to maintain them by hand and they always match the source code. I ask because I could not get type hints to work for wa-sqlite when using it as dependency (but maybe it's just my bad javascript skills or IntelliJ).

Regarding this PR I will close it now.
You also do not have to link to wa-sqlite-webtorrent. It first needs some documentation and wa-sqlite-webtorrent-demo some fixes, so it can be deployed as github page. Svelte-Kit is just too new & work in progress, the production build is not playing along with monaco-editor currently.

[rhashimoto]

Re Typescript, I like to develop without a build step as much as I can get away with. There was a time when Typescript's advantages could overcome that, but modern Javascript and its browser support (along with Microsoft's willingness to adapt VSCode Typescript tooling to Javascript projects, like supporting JSDoc typing) have advanced to the point that I prefer working with plain Javascript.

[steida]

@rhashimoto You can use Deno to develop in TypeScript and deno bundle to make a bundle for browsers.

The advantage of Deno is all batteries are included. Tests, Docs, etc. And you don't have to public to npm :-)

[rhashimoto]

@steida deno bundle is a build step 😀 . The web server I use for development serves unbundled ES6 modules (technically this does perform an on-the-fly file transformation but I think this is as lightweight as it gets until import maps arrive).

That said, I really love the ideas behind deno. I was holding off because of WebCrypto but I see it's there now so I'll have to check again whether there's any remaining reason not to migrate.