Using JAQ with a custom type

[tusharmath]

Thanks for an amazing library in Rust! I plan to add support for JAQ in tailcallhq/tailcall#1321 However there are a few concerns:

I have a "json-like" structure (not serde_json) that I'd like to integrate with JAQ. Is there a way to do it? What components can I reuse vs re-implement?

[wader]

The gojq fork used by fq adds a JQValue interface to make it possible for any type to behave like a jq value, see https://github.com/wader/gojq/blob/fq/gojq.go#L17 (note that some gojq implementations details leaks into the interface). Maybe can be a useful reference how it could work. It would be very interesting to see other jq implementations allowing things like this but it's understandable if it's out of scope for jaq.

[01mf02]

Hi @tusharmath! I have already thought about making the jaq value type more flexible, in order to support use cases like yours. I suppose that an approach like @wader's could work, and this might even allow jaq to be made flexible enough to support more input formats, like fq.

We could, for example, make a Value trait that implements just enough functionality such that the jq operations with dedicated syntax (constant value construction, indexing, iteration, arithmetic operations, and the likes) work on it. I believe that that's roughly what @wader did. This would not allow the core library to work yet, because it makes more assumptions about the value type (such as having a length), but it would be a start.

Can you show us what your value type looks like? That would help fleshing out whether the approach I outlined above would be applicable here.

[01mf02]

(Thanks, @wader, by the way for explaining how you do it in fq.)

[01mf02]

@tusharmath, I created a trait in jaq called ValT that will eventually allow you to use jaq with any kind of data type that implements this trait. However, so far I did not make this trait public, for I will not be able to make breaking changes to it once I release the next version (at least until jaq 2.0). Can you tell me whether your type could implement this trait?

[tusharmath]

Thanks for getting back @01mf02 !

Yes we can implement ValT. I would recommend making the trait public but marking it as unstable and hide it using the unstable feature flag. Folks like us could try it out and give you feedback without expecting a major version increase on breaking changes.

// This trait will be public if 'unstable' feature is enabled, otherwise it's crate-private
#[cfg_attr(feature = "unstable", pub)]
trait ValT {
    
}

// Implement the trait conditionally
#[cfg(feature = "unstable")]
impl ValT for MyStruct {
   
}

[wader]

@ssddOnTop hey! just to expand a bit on on my comment above. To implement some fq features (query then hexdump, binary concat/slice etc) it's important that an instance of something that implements the JQValue interface should "survive" untouched thru an jq expression evaluation if possible. Not sure if jaq aim's to support something like that and if so that would require use of dyn or something? (i'm a rust noob, working on it!)

[ssddOnTop]

@wader You are correct. As mentioned here implementation of ValT can perform all that operations. My PR just adds (de)serializer to Val, it can only be used for type conversions.

I think I should not link the PR with this issue

[wader]

Aha i see 👍

[01mf02]

@tusharmath, I'm happy to hear that you can implement ValT. I plan to release a new stable jaq version this week, and I plan to make the ValT trait public. Given that I plan to make a breaking release (2.0) next, this can integrate potential necessary changes to ValT.

[01mf02]

@wader, the "survival" of jq values is preserved by the introduction of the ValT trait, as it happens independently from ValT. For example, see the implementation of . (identity filter) in jaq: This returns the input value unchanged, so the original value "survives". This particular filter can be implemented without ValT.
Other operations, like e.g. addition that are handled inside of ValT, also always take values "by value" (as opposed to "by reference") and can thus return their original input values; that allows jaq to return the original value when adding null to it.

[wader]

@wader, the "survival" of jq values is preserved by the introduction of the ValT trait, as it happens independently from ValT. For example, see the implementation of . (identity filter) in jaq: This returns the input value unchanged, so the original value "survives". This particular filter can be implemented without ValT. Other operations, like e.g. addition that are handled inside of ValT, also always take values "by value" (as opposed to "by reference") and can thus return their original input values; that allows jaq to return the original value when adding null to it.

Great! that seem to be more than what my fork does atm. I think that will enable some interesting possibilities. The thing that was trickest and broke to most things when adding a JQValue interface to gojq for me has been standard library functions that are implemented with helpers in go for performance, ex things like to_entries, from_entries, split, join and sort, so mostly functions that work with array of values.

[01mf02]

Great! that seem to be more than what my fork does atm. I think that will enable some interesting possibilities. The thing that was trickest and broke to most things when adding a JQValue interface to gojq for me has been standard library functions that are implemented with helpers in go for performance, ex things like to_entries, from_entries, split, join and sort, so mostly functions that work with array of values.

Just to clarify, the interface I defined for values is quite minimal right now and only supports operations for which there exists dedicated syntax in jq (such as .[] or f + g). This excludes any filters that have a name, such as those that you mentioned, i.e. to_entries, ..., sort. However, I think that most filters that work with strings should be implementable with this interface already. For other filters, this interface is not be expressive enough. When I'll work on making the core library (which provides filters like length, from_entries etc.) generic over the value types, I'll probably define a new extended interface, or I'll augment the existing interface. I'm still not quite sure about which route to take. :)