Can't load extension from crawshaw.io/sqlite

[anacrolix]

I usually get this:

fatal error: bad sweepgen in refill

goroutine 1 [running, locked to thread]:
runtime.throw({0x12c8b6862?, 0x8?})
	/Users/anacrolix/src/go1.19/src/runtime/panic.go:1047 +0x5d fp=0xc000574ef0 sp=0xc000574ec0 pc=0x12c3560dd
runtime.(*mcache).refill(0x1017ae5b8, 0x0?)
	/Users/anacrolix/src/go1.19/src/runtime/mcache.go:156 +0x216 fp=0xc000574f30 sp=0xc000574ef0 pc=0x12c336896
runtime.(*mcache).nextFree(0x1017ae5b8, 0xb)
	/Users/anacrolix/src/go1.19/src/runtime/malloc.go:819 +0x85 fp=0xc000574f78 sp=0xc000574f30 pc=0x12c32c785
runtime.mallocgc(0x2b, 0x0, 0x0)
	/Users/anacrolix/src/go1.19/src/runtime/malloc.go:1018 +0x4c5 fp=0xc000574ff0 sp=0xc000574f78 pc=0x12c32cde5
runtime.rawstring(0x2b)
	/Users/anacrolix/src/go1.19/src/runtime/string.go:273 +0x37 fp=0xc000575040 sp=0xc000574ff0 pc=0x12c371d97
runtime.rawstringtmp(0x1017ae5b8?, 0x10?)
	/Users/anacrolix/src/go1.19/src/runtime/string.go:135 +0x3c fp=0xc000575060 sp=0xc000575040 pc=0x12c37179c
runtime.concatstrings(0x7?, {0xc0005750f8, 0x3, 0x7?})
	/Users/anacrolix/src/go1.19/src/runtime/string.go:51 +0xc5 fp=0xc0005750d8 sp=0xc000575060 pc=0x12c3710e5
runtime.concatstring3(0xc0000021a0?, {0x12c8b87b9?, 0xc0005400d0?}, {0xc0005400d0?, 0xc000575190?}, {0x12c8b032b?, 0x10254c000?})
	/Users/anacrolix/src/go1.19/src/runtime/string.go:64 +0x68 fp=0xc000575138 sp=0xc0005750d8 pc=0x12c3713c8
go.riyazali.net/sqlite.go_sqlite3_extension_init(0x10254c000?, 0x102304640, 0x70000d1cee60)
	/Users/anacrolix/src/riyaz-ali-sqlite/extension.go:40 +0xe5 fp=0xc0005751a0 sp=0xc000575138 pc=0x12c3f7165
_cgoexp_31f8983b1e63_go_sqlite3_extension_init(0x70000d1cee10)
	_cgo_gotypes.go:1659 +0x2c fp=0xc0005751c8 sp=0xc0005751a0 pc=0x12c3fc36c
runtime.cgocallbackg1(0x12c3fc340, 0x0?, 0x0)
	/Users/anacrolix/src/go1.19/src/runtime/cgocall.go:316 +0x2b4 fp=0xc000575298 sp=0xc0005751c8 pc=0x12c323994
runtime.cgocallbackg(0xc0000021a0?, 0x300000002?, 0xc0000021a0?)
	/Users/anacrolix/src/go1.19/src/runtime/cgocall.go:235 +0x109 fp=0xc000575328 sp=0xc000575298 pc=0x12c323629
runtime.cgocallbackg(0x12c3fc340, 0x70000d1cee10, 0x0)
	<autogenerated>:1 +0x2f fp=0xc000575350 sp=0xc000575328 pc=0x12c38beef
runtime: g 1: unexpected return pc for runtime.cgocallback called from 0x10006d920

or

sqlite.Conn.LoadExtension: SQLITE_ERROR: error during initialization: no extension with name 'default' registered.

The first I suspect to be some kind of GC disagreement. The latter might be related to multiple Go runtimes? I'm not sure what happens if a Go process tries to load to load a c-shared binary that I think also contains a Go runtime.

[anacrolix]

I'm not sure if it's this repo, or the other, or some issue due to this being Go->C->C->Go. The issue persists even when trying to register stuff directly on the sqlite3 * inside the crawshaw DB (so not having the c-shared or using runtime extensions).

[asg017]

Re the 1st error: I have seen this when loading Go SQLite extensions in a Go environment. To put it shortly, I don't think it's possible, but finding an exact reason why has been hard.

This thread might have more info: https://groups.google.com/g/golang-nuts/c/omI57CgJj20

From what I can tell, Go's buildmode=c-archive doesn't work if you're trying to load it from a Go runtime. I think the multiple runtimes where the "host" conflicts with a "guest" (from a loadable SQLite extension) causing the panic message you see.

However, I've been able to run multiple Go SQLite extensions from a "host" from C or Rust (ex loading 2 extensions built with this library from the sqlite3 CLI). So having a "host" of Go and a "guest" extension in Go doesn't work, but having a "host" of C/sqlite3 and multiple "guest" Go SQLite extensions work fine.

Re the 2nd error: That error message seems to originate here, which was a recent addition to the library (#10) - do you have more info on how you get this?

[anacrolix]

I have done more experimentation on this. I think both messages above are due to having 2 runtimes loaded, it's just nondeterministic where you will see the first failure. I am able to load the extension from crawshaw importing my extension code directly in Go (without the main and the implicit registration) and creating an ExtensionApi with my sqlite handle (crawshaw's one) inside it and passing that to my extension register function that I would normally passed to Register for a standalone c-shared extension. I also had to fiddle the build tags to tell this repo to build without doing its own init or including sqlite3. I made all this occur with the static tag. See riyaz-ali:77a6464...anacrolix:df377da.

I think some of this has some value to be contributed, I mentioned in #2 (comment). It would be great to know if I've missed how to do this already or if my changes were actually necessary for this use case. Thanks for jumping in @asg017 !

[anacrolix]

I do believe my theory above is correct. The only available solution (Go limitation) for now is to document how to work around this case. See #2.