Support for PostgREST

[amitu]

Rust is a great language for writing WASM plugins, and if the plugins need database, PostgREST is an option because HTTP is available to WASM code, but not raw sockets (so DB drivers are not possible in WASM context, unless you use WASI, but if you plan to target web browser, you will probably never get raw socket access there).

For making DB calls diesel is pretty great option, but does not work in WASM context today because there is no diesel connection implementation. I think it would be great if it was possible.

A developer in my team is looking at implementing support for it (its going to be open source), but we are just getting started. If you are already working on this, or want to join us, maybe even in paid capacity, I would love to hear from you.

Disclosure: we are a (seed) funded startup, FifthTry.com, and this is part of an open source technology we are working on.

[weiznich]

Thanks for opening this post here. This sounds like a great idea. We do not have any planes to add additional connection/backend implementations to diesel itself. As you already noticed any additional implementation should be provided by a third party crate. We are happy to support the development of such third party implementations by exposing any necessary type via the i-implement-a-third-party-backend-and-opt-into-breaking-changes feature flag, so please reach out if you feel that something is missing.

For the actual implementation: I would likely go for a fully custom backend in this case, as this gives you the necessary control about which kind of query is supported, how to represent bind/result values and so on.

If you are already working on this, or want to join us, maybe even in paid capacity, I would love to hear from you.

At least I personally do not have any capacity to work on this on paid basis. I cannot comment for other team members.

[amitu]

An update. I have given up on PostgREST, and have implemented PostgreSQL support within WASM. Here is the relevant code that runs inside wasm: ft-sys/diesel-pg, and the corresponding code that runs in the host fastn-ds/wasm/pg. @weiznich your diesel_async was very helpful, thanks a lot.

Now I am trying to do the same for sqlite, but unfortunately, there is no sqlite-backend feature in diesel, and so I can not re-use some of the stuff as I was able to do for PG. I can try taking a stab at it (tho I have to admit, I am still very shaky about how things work). In the meanwhile I think I will have to borrow some code from diesel.

[weiznich]

Thanks for reaching out.
There is currently no sqlite backend feature in diesel as most of the implementation is quite tightly coupled to what diesel does internally. I might be open to accept a PR that changes this as long as it does not regress performance.

[amitu]

We are using an owning version for bind collector and sqlite value. Checkout our PR, any feedback most welcome.

[derive(Debug, Default, serde::Serialize)]
pub struct SqliteBindCollector<'a> {
    pub binds: Vec<super::Value>,
    pub _m: std::marker::PhantomData<&'a ()>,
}

pub struct SqliteValue<'a> {
    pub(crate) raw_value: &'a Value,
}

#[derive(Clone, Debug, serde::Deserialize, serde::Serialize)]
pub enum Value {
    Null,
    Integer(i64),
    Real(f64),
    Text(String),
    Blob(Vec<u8>),
}

Where as one in diesel uses pointer versions.

#[allow(missing_debug_implementations, missing_copy_implementations)]
pub struct SqliteValue<'row, 'stmt, 'query> {
    // This field exists to ensure that nobody
    // can modify the underlying row while we are
    // holding a reference to some row value here
    _row: Ref<'row, PrivateSqliteRow<'stmt, 'query>>,
    // we extract the raw value pointer as part of the constructor
    // to safe the match statements for each method
    // According to benchmarks this leads to a ~20-30% speedup
    //
    // This is sound as long as nobody calls `stmt.step()`
    // while holding this value. We ensure this by including
    // a reference to the row above.
    value: NonNull<ffi::sqlite3_value>,
}

pub(crate) enum InternalSqliteBindValue<'a> {
    BorrowedString(&'a str),
    String(Box<str>),
    BorrowedBinary(&'a [u8]),
    Binary(Box<[u8]>),
    I32(i32),
    I64(i64),
    F64(f64),
    Null,
}

We are building it for wasm, so we need the owned values, so we can serialise and send it to wasmtime.

My first attempt was to put a feature flag, and try to have two versions, but the code started becoming quite difficult to follow (it may very well be my fault, and next attempt may be better).

Just to clarify, you consider these owning versions a performance regressions, and would not accept diesel converting everything to the simpler owning types?

[weiznich]

There is already #3977 which implements something similar, that might be useful for you too.

Just to clarify, you consider these owning versions a performance regressions, and would not accept diesel converting everything to the simpler owning types?

Using owned versions of every value is quite a large performance regression as far as I can tell. We talk about something from 30% slower up to a magnitude of order slower, which is not acceptable in my opinion.

My first attempt was to put a feature flag, and try to have two versions, but the code started becoming quite difficult to follow (it may very well be my fault, and next attempt may be better).

Just be aware that even the current code is already quite complex due to the needed FFI interactions and all the invariants that need to be uphold. That's one of the main reason no sqlite-backend feature exists, as it wouldn't be useful without relaying on too much internals and I've not found a good way yet to introduce an abstraction boundary there without regressing performance by much. (That's likely because sqlite is much more integrated into the application, it doesn't do network IO which shadows inefficiencies for the other backends)