ippfuzz: Fuzz test generator for ipptool

[michaelrsweet]

Copied from ippsample issue 72

As part of our discussions during the February 2018 IPP F2F meeting, we discussed ways to introduce fuzz-testing into ipptool.

Two approaches were considered:

1. Adding support to ipptool to randomly generate requests, attributes, values, etc.
2. Adding a program to generate random ipptool test files that can be fed into ipptool.

Since the first approach would still require ipptool to be able to write out a failed test for later diagnosis (what the second approach will do as its primary function), it seems like creating a test file generator is the simpler approach.

The generator should provide options to limit the amount of randomness, for example "always use Get-Printer-Attributes", "always include the following attributes and values", etc. so that "mostly valid" requests can be generated to test specific areas.

The generator should also provide options for the expected responses, for example "expect the following status codes" or "expect the random attributes to be in the unsupported attributes group of the response."

Ideally ippfuzz should be able to automate running ipptool and iterate N times, storing the results in a directory and providing a summary of the test results on stdout.

---

Also, when automating the tool should stop after a certain number of repeated failures (default 3?) - at that point the Printer has probably crashed...

---

When writing the results to a test directory, rename tests and results, e.g. "pass-nnnnnnnn.ext" for passed tests and "fail-nnnnnnnn.ext" for failed tests? Be sure to include the .test file and the XML results plist file.

---

Maybe focus on supporting fuzzing templates which look like ipptool test files? Expand values like:

• "$[ascii(100)]": Random string of exactly 100 characters
• "$[ascii(0,32767)]": Random string of 0 to 32767 characters
• "$[utf8(1000)]": Random Unicode string of 1000 characters
• "$[bytes(10,5000)]": 10 to 5000 random bytes
• "$[integer(10,20)]": Random integer from 10 to 20
• "$[boolean]": Random boolean value
• "$[value1|value2|value3]": randomly choose between the three values
• "$[repeat(10000) ... ]": expand the "..." text 10000 times.

"$[...]" might not be the best choice, consider other markup options.

Need to see what other fuzzing tools do.

---

Resources:

• The Fuzzing Project
• American fuzzy lop
• Scapy

[attah]

Inspired by this, i added fuzzing to my IPP implementation.
I went with LibFuzzer since it was so quick to get running: attah/ppm2pwg@e71a6e7

The people that worked on it has since moved on and via centipede, the efforts were merged into google/fuzztest.
In doing so it has gained functionality for property based testing, which should be good for the guided fuzzing mentioned above. Though i must admit i'm a bit apprehensive about guiding fuzzing too much, as i'd probably be blind to several problematic areas and might guide the fuzzer away from them. Coverage guided fuzzing definitely seems like a good approach, and with fuzztest it sounds like you can combine the approaches as you wish.

So how did LibFuzzer do on my code? It managed to smoke out one pretty bad undefined behavior that could be hit already without a corpus. The decoder itself has now held for something like 8 thread-hours. Not sure what that says though; either i did a good job, or the fuzzer is not very effective. ¯\(ツ)/¯
Fort sure, combining with the other sanitization options will turn up all sorts of things.