Session.fuzz() options to minimize redundant testcases

[kamakazikamikaze]

Proposal

Continuing from #622, I believe it would be beneficial to add additional parameters to Session.fuzz() for fine-tuning the generation of testcases. The current method of recursive generation may cause BooFuzz to revisit already-sent payloads.

Therefore I would suggest/request the following:

1. An optional min_depth parameter that specifies the minimum combinatorial count of Fuzzable mutations to use in testcases
2. An optional unique_only parameter that modifies the test generation logic to ignore mutation ordering when determining if a testcase has already been visited

I have included example code in the discussion answer that demonstrates how this can behave as well as the performance speedup in doing so.

Use-Case

Our test targets are rate-limited and the pre-/post-testcase callbacks will likely add significant processing time for checks/verification of results. Reducing the quantity of redundant testcases will help us optimize our time and ensure that we can reasonably perform testing in our approaching window. Other users may benefit from the reduction of redundant testcases when they include a large number of primitives in a payload.

Anything else?

The code example provided is a demonstration of the benefit(s) from this request and by no means an expectation of how it should be implemented.

[SR4ven]

Hey @kamakazikamikaze! Sorry for the long delay, things are a bit busy over here at the moment.

I really like you proposal, it would be a great improvement to boofuzz! I'd be happy to review the PR!

Your time measurements are really interesting as well. I always thought some kind of deduplication logic would impose a larger time penalty than the benefit from the skipped test cases. But I guess that highly depends on how high the test case frequency is.

About the min_depth parameter, it's not exactly intuitive so we'll have to document that in detail.

[kamakazikamikaze]

Understood. Since this is something I've developed while at work I've gotta go through some internal review processes for approval to release. In the meantime I'll expand the Session.rst documentation so that it goes into detail on "depth", understanding when (not) to specify min_depth, and copy-paste examples to demonstrate some of their benefits. It'll eventually be submitted as one PR.

EDIT 2022-07-05: Still waiting for Legal to review my request to submit a PR. I'll try to do weekly or bi-weekly updates on the status when I can.

EDIT 2022-07-20: Legal hasn't responded to my message inquiries. Trying to avoid escalating up the chain since they have higher priorities to deal with but I'm prepping an email when it's been long enough of a wait.

[jtpereyda]

Thank you @kamakazikamikaze and good luck on the legal process! I dropped an implementation thought in the linked discussion. I think unique_only doesn't need to be a user-configurable option. I can't think of a situation where it's needed.