Automatic conversion of types between Rust and Javascript via serde

[matklad]

Hi!

I've been using neon successfully in my side-project and everything works great, except that there's quite a bit of boilerplate required to marshal types between Rust and JS. I wonder if this can be automated away with the help of serde. It's sort of possible today, if you literally serialize and deserialize JSON from strings, but it would be cool to be able to use serde with JsValues directly!

[GabrielCastro]

I've been working on this https://github.com/GabrielCastro/neon-serde , It's got most of the uses cases covered, but is lacking in documentation

[matklad]

Thanks a lot, @GabrielCastro! I've used it in my project and it worked great!

Here's some feedback:

1. Actually, I find the example in Readme the most helpful, so the docs are great to me 👍

2. Looks like it is in theory possible to eliminate some boilerplate from argument parsing, so that instead of doing call.arguments.require(scope, 0)?.check::<JsInteger>(), one can use something like

let (foo, bar, baz): (i32, String, MyType) = call.arguments.deserialize()?

3. to_value function is constrained only to the RootScope, and not T: Scope, so it's impossible to use it, for example, inside Task's complete method.

[GabrielCastro]

@matklad Thanks for the feed back, I've made a couple issues from your comments. Please feel free to comment there

[maciejhirsz]

Just stumbled upon this.

I think in principle it should be possible for neon to implement a function wrapper with generic implementations for:

fn<R>() -> R
fn<A, R>(A) -> R
fn<A, B, R>(A, B) -> R
fn<A, B, C, R>(A, B, C) -> R
...
fn<A, B, C, D, E, F, G, H, R>(A, B, C, D, E, F, G, H) -> R

With generics being:

where
    A: Deserialize,
    B: Deserialize,
    C: Deserialize,
    D: Deserialize,
    E: Deserialize,
    F: Deserialize,
    G: Deserialize,
    H: Deserialize,
    R: Serialize,

With that in place the end user can just write a fn foo(String, i32) -> String or anything really, without writing any JsValue glue code by hand. With some inlining and ingenuity I don't see why this couldn't also be completely zero cost.

[maciejhirsz]

Turns out doing it with traits as I intended wasn't possible (at least not without some nightly features to create a 0-type structs that implement Fn and then pass those as a generic parameter...). Macros do, however, and it looks pretty clean: GabrielCastro/neon-serde#11

[antonok-edm]

@dherman @kjvalencik would there be any interest in getting neon-serde under the @neon-bindings org? It's extremely useful, but @GabrielCastro's original repo is quite outdated now and it seems like there's a big tangle of forks in the community. For example, see the Network graph for the original neon-serde repo; there's also neon-serde2 and neon-serde3 each published to crates.io with only one version. (I'm guilty of making my own fork as well).

As it stands currently, the most up-to-date one I could find is matrix-org's fork which has about 1200 lines of Rust code. It's not tiny, but compared to the current neon codebase it's probably not too significant of an additional maintenance burden either. At least having an "official" version that everyone's using, even if there are problems with it today, would make it much easier for the community to track issues and contribute meaningful fixes/features.

[kjvalencik]

@antonok-edm I considered that here, mostly for performance, but was disappointed that it was slower than serde_json + JSON in nearly every benchmark.

However, given the popularity of neon-serde, perhaps it's something that should be reconsidered? What are your thoughts?

[antonok-edm]

@kjvalencik ah, that's good to know!

The main benefit of neon-serde from my perspective is to be able to work with JsValues directly, rather than having to convert to/from strings on either side of the API. I don't particularly mind the implementation details - obviously better performance is nice if possible, but I suspect most neon users do most of the heavy work in Rust code and do significantly less work crossing back and forth over the FFI. If there's a way to have neon-serde just be a thin wrapper around serde_json, that seems like the best of both worlds.

But regardless of implementation, I think neon-serde has had enough interest that it'd be better to have an officially maintained version (even if it contains a big disclaimer in the README saying "you might want to use serde_json instead for performance"). Right now, there's no single place to go for this kind of functionality, whether for code, information, or filing issues or PRs. If it's included inside neon itself, even better.

[kjvalencik]

@antonok-edm do you happen to use the the export macro or any other macros for converting arguments? That's the part that I'm most hesitant about pulling into Neon. While there's certainly some ambiguity about how to handle certain data types, converting function arguments is a bigger design space.

RE: Performance. I don't have a good sense of how much serialization is a bottleneck in Neon apps overall. My view is a bit skewed because it's by far the largest cost in the app I build at work. I've tried a few things--including implementing v8.serialize in Rust--but, so far, nothing has beat JSON.

[antonok-edm]

I'm not using any macros at the moment. (code is here if you're curious)

edit: other than register_module!, but that's from neon

[ranile]

I needed this for work so we ended up forking neon-serde and bringing it up to date. It can be found at https://github.com/formbird/neon-serde