Fuzzing with AFL

This tutorial walks through the process of fuzzing a Java program using JQF with the AFL fuzzing engine. AFL specializes in fuzzing binary data, and is therefore suited for testing Java programs that parse arbitrary streams of bytes. In particular, JQF-AFL generates random InputStream objects. If you want to generate test inputs of other types, such as data structures, syntax trees, or any inputs that satisfy specific constraints, use JQF with the Zest engine.

Requirements

JQF's integration with AFL has been tested on MacOS and Ubuntu, though it should work on any other linux-based systems too. If you have a unix-based system where JQF-AFL does not work as intended, please raise an issue. JQF requires at least AFL version 2.31b to work, though something more recent is preferred (JQF has been tested with AFL 2.52b at the time of this writing).

Setup

Setting AFL_DIR

JQF must be able to find AFL! If you installed afl-fuzz via standard repositories you may skip this step. JQF will either:

• look for the program afl-fuzz in the current PATH (type afl-fuzz in the command-line to see if it works; also check the minimum version as mentioned above), or
• look for the program afl-fuzz in the directory specified by the environment variable AFL_DIR.

The most common way to get started with JQF is to type in bash:

# Clone and build AFL
git clone https://github.com/google/afl && (cd afl && make)

# Set AFL directory location
export AFL_DIR=$(pwd)/afl

Install JQF

Unlike with other fuzzing engines (e.g. Zest), the use of JQF+AFL requires that you clone the JQF repository and run the set up scripts. The scripts build the JQF-AFL proxy program that communicates signals between the AFL process and the JVM which runs the test program.

# Clone and build JQF
git clone https://github.com/rohanpadhye/jqf && jqf/setup.sh

### See usage of JQF-AFL
jqf/bin/jqf-afl-fuzz

Write a test driver

A JQF test driver is a JUnit-style test class with the following annotations: @RunWith(JQF.class) on the test class, and @Fuzz on the test method. The test method must be public void. The formal parameters of the test method are the inputs generated by JQF -- when using JQF+AFL, the test method should have exactly one formal parameter of type InputStream.

Here is an example of a test driver that tests the in-built PNG image decoding logic in the JDK ImageIO library:

import javax.imageio.ImageIO;
import javax.imageio.ImageReader;
import javax.imageio.stream.ImageInputStream;
import java.io.FileOutputStream;
import java.io.IOException;
import java.io.InputStream;

import edu.berkeley.cs.jqf.fuzz.Fuzz;
import edu.berkeley.cs.jqf.fuzz.JQF;
import org.junit.Assume;
import org.junit.runner.RunWith;

@RunWith(JQF.class)
public class PngTest {

    @Fuzz /* JQF will generate inputs to this method */
    public void testRead(InputStream input)  {
        // Create parser
        ImageReader reader = ImageIO.getImageReadersByFormatName("png").next();

        // Decode image from input stream
        try {
            reader.setInput(ImageIO.createImageInputStream(input));

            // Bound dimensions to avoid OOM
            Assume.assumeTrue(reader.getHeight(0) <= 256);
            Assume.assumeTrue(reader.getWidth(0) <= 256);

            // Decode first image in the input stream
            reader.read(0); 
        } catch (IOException e) {
            // This exception signals invalid input and not a test failure
            Assume.assumeNoException(e);
        }
    }
}

Save the above file as PngTest.java. You can use javac to compile this file as long as you include JQF and all its dependencies on the classpath. JQF ships with a bash script called classpath.sh which expands to a list of all JARs that it needs. Compile the above file as follows in bash:

javac -cp .:$(jqf/scripts/classpath.sh) PngTest.java # 'jqf' is the directory where JQF is cloned/installed

Fuzzing with jqf-afl-fuzz

The program jqf-afl-fuzz is a wrapper around the program afl-fuzz that comes with AFL. The wrapper allows you to specify the classpath of your Java tests and then just the test class and method instead of having to invoke the JVM yourself. You can always pass additional options to the JVM (e.g. system properties) using the environment variable JVM_OPTS.

Usage: jqf-afl-fuzz [options] TEST_CLASS TEST_METHOD
Options: 
  -c JAVA_CLASSPATH  Classpath used to find your test classes (default is '.')
  -i AFL_INPUT_DIR   Seed inputs for AFL (default is a few seeds of random data)
  -o AFL_OUTPUT_DIR  Where AFL should save fuzz results (default is './fuzz-results')
  -x AFL_DICT        Provide a dictionary to AFL (default is no dictionary)
  -S WORKER_ID       A unique identifier when running in parallel mode
  -T AFL_TITLE       Customize title banner (default is TEST_CLASS#TEST_METHOD)
  -m MEM_LIMIT       Set a memory limit in MB (default is 8192)
  -t TIMEOUT         Set a single-run timeout in milliseconds (default is 10000)
  -v                 Enable verbose logging (in file 'jqf.log')

Let's run jqf-afl-fuzz on our PNG image decoder, while providing as seed inputs the default PNG files that ship with AFL.

jqf/bin/jqf-afl-fuzz -i $AFL_DIR/testcases/images/png/ PngTest testRead
> Performing pilot run...

The wrapper first performs a pilot run without AFL to ensure that you have set-up your classpath and test driver correctly. If there was any problem, you should see an error message here. Otherwise, if all goes well, you'll see:

> Pilot run success! Launching AFL now...

And then the AFL status screen:

Analyzing crashes (and other fuzzed inputs)

You can reproduce one or more inputs found by AFL using the program jqf-repro:

jqf/bin/jqf-repro PngTest testRead fuzz-results/crashes/id:000000*

Run the above program without any arguments to see full usage information (e.g. providing custom classpaths).

The repro script will re-execute the test method with the provided inputs, and print exception stacktraces, if any, on the standard error stream.

Troubleshooting

If the pilot run fails, you should see a stacktrace of the exception that caused this on the screen. Typical reasons for the pilot run failing is providing a wrong or misspelt class name or method name, or providing an incorrect classpath, resulting in a ClassNotFoundException.

You can also use the -v option with jqf-afl-fuzz to make JQF dump logs of certain things in the file jqf.log, such as the bytecode instrumentation of class files. You will see any errors due to instrumentation in this file, along with anything your test classes write to STDOUT or STDERR.

If you cannot see any errors or the error does not seem to be your fault, please open a GitHub issue.

Advanced

Parallel mode

You can also run multiple instances of JQF-AFL in parallel to utilize multiple CPU cores and speed up fuzzing. Run each sequential instance with the option -S <id>, where <id> is some unique identifier for that instance. For example, the following spaws two parallel instances:

jqf/bin/jqf-afl-fuzz -c tests -S id1 PngTest testRead
jqf/bin/jqf-afl-fuzz -c tests -S id2 PngTest testRead

You might have to run the above commands in separate terminals/screens. The output for instance id1 will be in directory fuzz_results/id1 and so on. Remember that you can change the root output directory with the -o option.

A note on multi-threading

Multi-threaded tests can be difficult to work with. If your test method spawns new threads to do work, make sure to synchronize with these threads before returning (e.g. using Thread.join()). If a worker thread continues to run after the test method returns, then its code coverage can spill onto subsequent trials (with different inputs) and the fuzzing session will become unreliable.