Release of QCheck2

[vch9]

Hi!
Do you have any idea on when you'll release the new QCheck2?
@sir4ur0n and I are currently working on migrating the Tezos qcheck tests to qcheck2 -> here.
That merge request should be a good example of the the "validity" of that new version.

(I'm not trying to push you, we were just wondering 😄)

[c-cube]

That's a good question. The only PR pending is a stylistic one, I think we could give a try to a release with qcheck2. The stability guarantees are going to be weak for a little while though 🙂.

[c-cube]

OK, can submit them both as QCheck(1)+QCheck2 tests.

Awesome, please do!

IIRC @c-cube, didn't we use some of my playing-with-the-function-generator https://github.com/jmid/qcheck-fun
as regression tests?

I don't recall that we used these as regression tests, but I'd welcome a PR that adds them to the repo! Never sure how to integrate them into actual automated tests (maybe nested tests?) but if we could put that into CI it'd be awesome.

[jmid]

OK, I've had a closer look. From what I can tell

• src/core/example/QCheck_runner_test.ml contains a combination of examples and tests (incl. at least one regression and some of the function tests from the above link) - runnable by make example-runner
• src/core/example/alcotest and src/core/example/ounit contain selected variations of these - runnable by make example-test, make example-ounit-test, and make example-alcotest
• test/core/test.ml contains the beginning of a proper test suite, starting with a bunch of internal QCheck2 tests from @sir4ur0n - runnable by make test

The last one fits immediately into a CI hook, whereas the first two have good content accumulated over a couple of years, but morally more of the form "run-these-and-manually-check-the-output-errors-before-releasing, please".
I believe this is because the former are all positive tests - they should be "all green".

To turn the latter into the former - and to add the two false examples above as proper testsuite tests I think
we could benefit from a fails-test. This should test a knowingly false property and ensure that it fails as expected.
The commercial Quviq QuickCheck for Erlang has such a thing: http://quviq.com/documentation/eqc/eqc.html#fails-1
Do you agree? (A quick look at the source code indicates that this may be achievable by adding a FalseTest case to Test.t + a number of adjustments)

[c-cube]

actually, now that I think of it: we could turn these into expect tests. This allows to check for the output shape as well as pass/fail status, and is very low cost to implement.

[c-cube]

fyi: #140

[jmid]

Awesome @c-cube! I've come across expect-tests in the past, but hadn't thought of using them here...
I agree this is more low cost to implement and I'm glad to see our accumulated tests come to good use! 😀👍🏻

[c-cube]

Closed the release PR to opam-repo as the new code is clearly not cooked yet :)

[c-cube]

Indeed! I just merged it. It's a bit finicky because some output is not deterministic, but hopefully it serves us well.

[jmid]

Great! Can I ask why you only diff-test for OCaml pre-4.09? (just curious) 🤔

[c-cube]

On 4.12 some compiler output changed again. But now it's filtered out, maybe I should retry on 4.12

[sir4ur0n]

Hey, what's the current status on releasing QCheck2 please? Now that the String shrinking bug is fixed.

I have rebased https://gitlab.com/tezos/tezos/-/merge_requests/2908 on master and adapted to latest changes, tests still pass - the only noticeable difference is for src/lib_proxy/test/test_fuzzing_light.ml which makes intensive use of recursive generators with lists of lists of string, where the perf diff is noticeable, as we knew. Otherwise this looks good.

[c-cube]

I don't really know. We have a test suite now which (I think) addresses @jmid 's initial findings, and these bugs were fixed. Y'all at Tezos are using QCheck2 in your test suite without problem? If it's the case I'll try to draft a new release soon.

[sir4ur0n]

We currently don't use QCheck2 on master because it's not in Opam, but the branch I have linked uses it and has migrated more than 20 PBT files to QCheck2: all tests succeed.

[sir4ur0n]

Hi @c-cube , I wanted to kindly bump this discussion 😅

No worries if you don't have time, it's just in case it slipped your mind 😄

Is there anything we can do to help get this released?

[vch9]

Maybe I could use generators inside https://gitlab.com/tezos/tezos/-/merge_requests/2908 in failing test in order to see the shrinking resilience.

[c-cube]

More tests is always good :). I'm aware of this issue but right now I'm on vacation :)

[c-cube]

it's released!

[jmid]

I can see 0.18 is available here https://opam.ocaml.org/packages/qcheck/
However both https://github.com/c-cube/qcheck and https://c-cube.github.io/qcheck/ still list 0.17 as the latest.
I think the issue is the release tag which is v0.18 for the latest (in contrast to 0.17, 0.16, 0.15, ... sans v),
which breaks some of the automated updating...

[c-cube]

there is no automation for releases, but I just updated the branch :)

[jmid]

Hm, OK.
On the frontpage of https://github.com/c-cube/qcheck
in the right column it still says "0.17 (latest)" in my Firefox.

If I then press "Releases (26)" and go to https://github.com/c-cube/qcheck/releases
the "v0.18" is styled differently (smaller, white font) and doesn't say "c-cube released this on ..." like for the other releases 🤔