Built-in support for Property Based Tests (aka Fuzzing)

[dubzzz]

Hey 👋,

I'm Nicolas, core maintainer and main contributor to the project called fast-check. The project does property based testing and reaches 650k downloads a month at the moment.

In a nutshell:

Property Based Tests is another way to test code. Instead of writing hard-coded tests (such as "startWith should return true for prefix='t' value='toto'"), we write tests following specs (such as "startWith should return true for any prefix and value=prefix+suffix").

I like to say that "it can detect the unexpected".

It can be pushed even further to detect race conditions, CVEs...

Why it matters?

Well, while classical tests are a must have for test frameworks, property based tests leverage them to next level by making tests able to find bugs earlier. Having such built-in in vitest would probably help the JavaScript community to embrace even more the technique (up to now all JavaScript property frameworks put together are gathering a bit less than 1M downloads per month with 650k for fast-check alone).

Some people asked such support on my side: dubzzz/fast-check#3054

Suggestions of integration

While the libraries can be used as standalone libraries next to any test framework, some binding making them better integrated with test runners can always help. For fast-check, I came up with an integration for Jest working as a standalone package providing an API rewiring Jest and fast-check for a better integration but as I'm starting one for Vitest I'm wondering if the best option would not be to just have it native.

My idea would be to come up with some kind of it.prop (see related topic for the binding between Jest and fast-check).

Alternative way being: third-party package wiring vitest to fast-check such as @fast-check/jest but for vitest or users keep doing it on their side.

👋 I opened the discussion mostly because I believe that providing such test technique directly within Vitest would help a lot the JavaScript community. If the idea of having such built-ins directly within Vitest is something that might be interested for Vitest, then I can probably give an hand and start an integration if I get some guidance and if an API is somehow accepted.

💡 Let me know if you need more context. I linked fast-check so that you can see more in depth what it's about.

[jasikpark]

I've been using fast-check with vitest as much as possible! This could be great!

[dubzzz]

Following multiple requests to have a fuzzing approach natively in fast-check library, I started to dig a little bit more on the subject and thought back about this opened discussion. Actually, many people want fuzzing to be a thing in JavaScript and actually fast-check alone miss some parts of it, as vitest does. A good fuzzer (as the one of Go) requires a collaboration between the test runner and the fuzz system (fast-check is a fuzz system).

My point is: we should probably call the function it.fuzz instead of it.prop.

The normal test mode runs the test in a classical PBT manner. The fuzz mode could be inspired by what Go does with some kind of infinite runs or runs being capped by a time limit. Anyway, so far I implemented on a drafty version for the it.fuzz in test mode in @fast-check/vitest, I'll start making my mind on a real fuzz mode. But from what I see: a closer collaboration with the test runner might be needed at some point to have a dedicated mode.

[sheremet-va]

You can track the progress there: #4076

If you have any questions or suggestions feel free to leave it there.

[dubzzz]

Do you want to call it twice? I don't understand the usecase

In the context of property based testing or fuzzing, the very same function will be executed multiple times (by default 100 times if everything goes well in fast-check). Basically the aim of these techniques is to generate inputs that may detect issues on the code under tests. Even if they are quite efficient to detect problems quickly they tend to require a certain number of runs (a hundred being very safe). As a consequence, the same function will be executed multiple times, so if beforeEach have been provided by the user it looks legit to run them.

Ideally, the handler code should be something like:

   handler: async () => {
    const promiseProp = wrapProp(prop);
    const propertyInstance = (asyncProperty as any)(...(arbitraries as any), promiseProp);
    await assert(
      propertyInstance.beforeEach(theBeforeEachDefinedForVitest).afterEach(theAfterEachDefinedForVitest),
      params
    );
  },

[sheremet-va]

Then you should provide 100 tests, this is how this system works. One custom (renamed to task) is one task, beforeEach/afterEach runs before and after each task. How you populate it is your implementation

[dubzzz]

I just tested e744d9e by relying on v1.0.0-beta.0 of vitest it works perfectly!

---

UPDATE: I just have a typing issue with the output of createChainable. So far, I have not succeeded into typing it in a way the produced function (fuzz) can rely on generics. But I'll retry later. At the moment I have: const fuzz: ChainableFunction<"concurrent" | "only" | "skip" | "todo" | "fails", [name: string, arbitraries: ArbitraryTuple<any[] | [any]>, prop: Prop<any[] | [any]>, params?: FcParameters<any[] | [any]> | undefined], void, {}> while I'd need T to replace some of the any. But honestly I think most of the issue is on my side. Exposing ChainableFunction might help but I'm not even sure, I'll do some tests on my side.

My code today being:

export const fuzz = createChainable(['skip', 'only', 'todo', 'concurrent', 'fails'], function fuzz<
  Ts extends [any] | any[],
  TsParameters extends Ts = Ts,
>(this: any, name: string, arbitraries: ArbitraryTuple<Ts>, prop: Prop<Ts>, params?: FcParameters<TsParameters>): void {...}

[sheremet-va]

I just have a typing issue with the output of createChainable

I would recommend using the new "createTestCollector" utility instead (it implements more APIs like skipIf, but also extend and each which might not be desirable in your cade). Although you will probably have similar issues with types there 🤔

[bertschneider]

Even though it's a somewhat different approach, in the project Jazzer.js we combined libFuzzer with Jest and thought about providing a similar integration with vitest.

Perhaps we could collaborate on a vitest integrations to get the best of both approaches.

Using it.fuzz feels quite natural, so I would encourage using that syntax.