Compatibility matrix and benchmark repo

[codetiger]

Hi Team,
Planning to add a repo to check compatibility and if possible benchmark various implementations of JSONLogic spec. Original idea came from this discussion (jwadhams/json-logic#46) and want to extend the idea.
I am thinking of building a repo with an improved version of the standard test data that can be reused across implementations for testing compatibility and benchmarking.

Repo	Language	Release	Compatibility	Benchmark
https://github.com/Viicos/jsonlogic	Python	0.1.0 (3 Jun 2024)
https://github.com/TotalTechGeek/json-logic-engine	JavaScript
https://github.com/Open-Payments/datalogic-rs	Rust	1.0.6 (3 days ago)

Suggestions invited

[TotalTechGeek]

Let's try to get a list gathered for all JSON Logic interpreters.

These are the projects I've reached out to: https://github.com/orgs/json-logic/discussions/1#discussioncomment-11464610

Note that when testing with json-logic-engine, compatible: true needs to be passed into the constructor, as it tweaks truthiness. In a future release, I might make compatible: true the default.

[codetiger]

Yes, I meant, compatibility can have a list of features that the repo supports from the spec. So users can understand what part of the spec does each engine supports and focuses on.

[josephdpurcell]

I would happily assist with executing benchmarks/testing. It would be very very useful to have some kind of framework by which engines can be plugged in so that this isn't a one-time exercise but something that could be tied into the decision making process by maintainers.

For example, I have a keen interest in seeing jwadhams/json-logic-js#120 be merged. How does it impact performance? I have a theoretical opinion but seeing numbers would be evidence.

I saw rumblings with webpack alternatives that spurred a lot of conflicting opinions and detailed discussion. Having an opinion about the level of rigor for the performance tests would be useful too.

I'm curious what open source projects have done this well?

Those are all some thoughts. If requested I can put some research behind these thoughts.

[TotalTechGeek]

How does it impact performance? I have a theoretical opinion but seeing numbers would be evidence.

While a bit off-topic -- from my experience developing json-logic-engine, adding this capability did not seem to harm overall performance.

Having an opinion about the level of rigor for the performance tests would be useful too.

Full agreement -- it's also possible that we may need to run the same library against multiple configurations.

json-logic-engine is able to detect if a branch is deterministic and pre-compute (or cache in interpreted mode) the result so it doesn't need to perform the work again. Benchmarks here.

So we'd probably want to explore not giving certain interpreters an unfair advantage, or at a bare minimum run the benchmarks with multiple configurations (disableInterpretedOptimization and disableInline flags on json-logic-engine)

[TotalTechGeek]

I also think it should be worked into Actions / CI for a repo in this organization, right now I'm committing accepted proposals and tests to the tests folder.

[pandaGaume]

Hey, adding my old JsonLogig.net as .net candidate https://github.com/pandaGaume/JsonLogic.Net. Need some polish to beeing updated with the latest .Net 8, but normally working out of the box. Let me know if you want to reference this work into the repo.

[TotalTechGeek]

Heya; We definitely do!

[codetiger]

Yes, thanks everyone for the interest.

Am looking for a framework to handle multiple language test execution in an organised manner and collect results and document. Please suggest if you can find any resources for this. Worst case am thinking of triggering individual test code for each implementations directly from GitHub actions.

Meanwhile, the intention is to build and maintain a unified way of testing all implementations. Benchmarking might help each maintainer have their own references during code changes. Also if a particular implementation has different ways of setting up, we can trigger benchmarks with each option so comparison gives a better idea of how the users can choose.

[beeme1mr]

Gherkin may be a decent option. We use it in OpenFeature to ensure all the SDKs behave the same across languages.

https://cucumber.io/docs/gherkin/

[beeme1mr]

I guess the test runner is technically Cucumber, and the configuration is Gherkin. At any rate, Cucomber has solid language support.

[TotalTechGeek]

OH! I love Cucumber / Gherkin -- this toy implementation uses JSON Logic 😂!

In general, I think that Cucumber / Gherkin tends to be overkill for unit tests, and maybe inappropriate for performance testing.

BUT this actually might help settle a concern that was brought up in the chained logic discussions w/r/t non-deterministic testing if that use-case pops up.

[codetiger]

I've created a test-suite repo (private), and added test suite for 2 Rust implementations and one for JS. Please review the project structure and suggest if we can proceed with this or needs some improvements.
Things to improve:

1. The JS implementation has issues in passing rule and data, that needs to be fixed.
2. Convert all report.json from projects and generate a md file for compatibility matrix

[codetiger]

I feel removing cucumber framework makes it much easier in this case. A plain test and benchmark code running as GH Actions sounds much better. A python script that can generate markdown table based on the reports.

Let me know your thoughts.

[gregsdennis]

Please feel free to take these as examples/inspiration:

• JSON Path Comparison
• JSON Schema's Bowtie

[gregsdennis]

JsonLogic from json-everything passes all tests. I've even added some new ones.

That said, there are still some behavioral differences that get reported from time to time.

[codetiger]

I've added Go lang tests into the Compat Tables repo today. Next is python and ruby, I need sometime with C# as it's been a while since I tried C#.