Bundling sqlite-vec into a go-sqlite3 build

[asg017]

Hey Nuno! Just wanted to share something I was able to make: a WASM build of SQLite, built for go-sqlite3, with sqlite-vec builtin: https://github.com/asg017/sqlite-vec/releases/download/v0.0.1-alpha.27/sqlite-vec-0.0.1-alpha.27-ncruces-go.wasm

I have a demo here of how it works, using the sqlite3.Binary trick. And the makefile, that just downloads the WASM file.

To build it it's a bit complicated, but it works. I just git clone this repo and add this patch file: https://github.com/asg017/sqlite-vec/blob/main/bindings/go/ncruces/go-sqlite3.patch

Then I build the project, all on github actions: https://github.com/asg017/sqlite-vec/blob/main/.github/workflows/release.yaml#L211-L238

And it works well! You can ignore the "database is locked" error, that's a sqlite-vec bug I'll fix:

$ make demo; ./demo
curl -L -o sqlite3.vec.wasm https://github.com/asg017/sqlite-vec/releases/download/v0.0.1-alpha.27/sqlite-vec-0.0.1-alpha.27-ncruces-go.wasm
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100 1414k  100 1414k    0     0  1149k      0  0:00:01  0:00:01 --:--:-- 4471k
go build -o demo demo.go
sqlite_version=3.46.0, vec_version=v0.0.1-alpha.27
rowid=3, distance=0.000000
rowid=4, distance=0.200000
rowid=2, distance=0.200000
2024/07/12 16:59:42 sqlite3: database is locked: unable to close due to unfinalized statements or unfinished backups

Just wanted to share to see what you think! I don't mind the build steps, but if you have an tips would love to hear it!

(Feel free to close whenever you like)

[ncruces]

Converted to a discussion, as although it's not “actionable” (not a task) this is exactly the kind of thing I want to keep open and visible.

[ncruces]

This is looking great, and is pretty much exactly how I'd approach it.

If it becomes more official, I'd probably create an embed package/module to make it easier to embed and set sqlite3.Binary but that's it (who “owns” that module is up to you).

I'm curious about SQLITE_VEC_OMIT_FS and what I can do to help. Is this an important feature we're missing out?

I managed to avoid the main SQLite Wasm needing WASI, but this is probably something that can be made configurable.

[ncruces]

For SQLITE_VEC_OMIT_FS one possibility would be to sqlite3_vfs_find(0), then use the VFS methods to read. But obviously that's more/uglier code.

Depending on how you feel about the patching this repo and stuff (I don't think it's too complicated, this is definitely what I'd do in repo), it's also possible to compile/link files separately.

Any function, anywhere, marked __attribute__((constructor)) void bla() is good enough to call sqlite3_vec_init.

Having everything in the same compilation unit helps performance maybe a bit (this is mostly irrelevant, wasm-opt fixes it) and gives me access to some private APIs, which is the real reason I'm doing it this way.

If you feel any of these changes would be useful, I don't mind doing them myself and contributing upstream!

[asg017]

Thanks for the response!

Ya looking at it, I'll probably create a separate Go module to distribute the WASM, like github.com/asg017/sqlite-vec-ncruces that go:embed's the sqlite-vec WASM blob. I wanted to avoid that in the main repo because I didn't want to commit a WASM blob, but a separate repo makes more sense.

The sqlite3_vfs_find(0) approach also makes a lot of sense, but blocking it with SQLITE_VEC_OMIT_FS is easier for now. It's only a single feature that called fread/fseek directly (streaming reads from a numpy file), which I think will be pretty rare. Though I think I'll eventually use the VFS methods later down the road

I'd appreciate any contributions that you feel would help! I'm definitely not a WASM expert so any help would be great. I do worry a bit about embed/build.sh or sqlite3/main.c in this repo diverging from the patch file I have, but I don't mind manually keeping an eye on updates here and changing when needed. Likewise, sqlite-vec is going to have a lot of updates/beta release between now and v1, so I don't mind "owning" the release process for now.

[asg017]

Just added that new repository for distributing. The demo is a lot easier now: https://github.com/asg017/sqlite-vec/blob/main/examples/simple-go-ncruces/demo.go#L10

[ncruces]

Great! I'll call out that in a README somewhere as a nice example of "you can build in your own native extensions like this” which I always wanted to do.

[ncruces]

https://github.com/ncruces/go-sqlite3/tree/main/embed#customizing-the-build