Writing an Annotation Processor

What is an annotation processor? It's code that writes code. In fact, you may have already used an annotation processor, such as Immutables or Dagger. For example, you can annotate an interface with @Value.Immutable, and Immutables will generate an implementation.

But what if you want to write (and test) an annotation processor? This project demystifies that process and provides a reference example.

Example

ImmutableProcessor generates a simplified implementation of an immutable interface.

For example, you can create an interface annotated with @Immutable, like this:

package org.example.immutable.example;

import org.example.immutable.Immutable;

@Immutable
public interface Rectangle {

    static Rectangle of(double width, double height) {
        return new ImmutableRectangle(width, height);
    }

    double width();

    double height();

    default double area() {
        return width() * height();
    }
}

...and ImmutableProcessor will generate this implementation:

package org.example.immutable.example;

import javax.annotation.processing.Generated;

@Generated("org.example.immutable.processor.ImmutableProcessor")
class ImmutableRectangle implements Rectangle {

    private final double width;
    private final double height;

    ImmutableRectangle(double width, double height) {
        this.width = width;
        this.height = height;
    }

    @Override
    public double width() {
        return width;
    }

    @Override
    public double height() {
        return height;
    }
}

(A real-world annotation processor would also implement equals(), hashCode(), and toString(), among other things.)

Goals and Non-Goals

Goals

• Demystify the process of writing an annotation processor.
• Include enough features to demonstrate the complexity of writing an annotation processor.
• Provide guidance for how to...
  • ...debug an annotation processor.
  • ...design an annotation processor.
  • ...unit-test an annotation processor.

Non-Goals

• Build an annotation processor that should be used in the real world.
  • I.e., you should use Immutables in the real world.

Quickstart

I just want to jump into the code. Where do I start?

ImmutableLiteProcessor

How do I debug the annotation processor?

• ./gradlew -Dorg.gradle.debug=true --no-daemon :immutable-example:clean :immutable-example:compileJava
  • From there, you can attach a debugger to Gradle.
  • If you use the Gradle Error Prone plugin with JDK 16+, you will also need to add some JVM args.
• You could also debug a test written with Compile Testing.

Where can the generated sources be found?

immutable-example/build/generated/sources/annotationProcessor/java/main

Design

We will start with ImmutableLiteProcessor and work downstream from there:

• ImmutableLiteProcessor implements a single method: void process(TypeElement annotatedElement) throws Exception
• The TypeElement corresponds to a type that is annotated with @Immutable.

Overview

ImmutableLiteProcessor processes an interface annotated with @Immutable in two stages:

1. The modeler stage converts the TypeElement to an ImmutableImpl.
   • The entry point for this stage is ImmutableImpls.
   • The code lives in the org.example.immutable.processor.modeler package.
2. The generator stage converts the ImmutableImpl to source code.
   • The entry point for this stage is ImmutableGenerator.
   • The code lives in the org.example.immutable.processor.generator package.

ImmutableImpl lives in the org.example.immutable.processor.model package:

• Types in this package are @Value.Immutable interfaces that can easily be created directly.
• Types in this package are designed to be serializable (via Jackson).
• Types in this package have corresponding types in the modeler and generator packages.

The implementation of the annotation processor is split into two projects:

• processor contains generic, reusable logic for annotation processing.
• immutable-processor contains logic specific to the @Immutable annotation.

Processing Environment

Annotation processors will use tools that are provided via Java's ProcessingEnvironment:

• The Messager reports compilation errors (and warnings).
• The Filer creates source files.
• Elements and Types provide utilities for working with Element's and TypeMirror's.

ProcessorModule can be used to @Inject these objects (via Dagger).

Incremental Annotation Processing

This annotation processor generates a single output for each input. Thus, it can be configured to support incremental annotation processing.

The following steps are needed to enable incremental annotation processing:

• Use CLASS or RUNTIME retention for the annotation. (The default is CLASS.)
• Use gradle-incap-helper to enable incremental annotation processing.
• Include the originating element when creating a file via Filer.createSourceFile().

Reporting Compilation Errors

Diagnostics is used to report any diagnostics, including compilation errors. It serves as a wrapper around Messager.

The main reason that Diagnostics wraps Messager is to enable error tracking. The error tracker converts the result to Optional.empty() if Diagnostics.add(Diagnostic.Kind.ERROR, ...) is called. This allows processing to continue for non-fatal errors; compilers don't stop on the first error.

See this condensed snippet from ImmutableImpls:

try (Diagnostics.ErrorTracker errorTracker = diagnostics.trackErrors()) {
    // [snip]
    ImmutableImpl impl = ImmutableImpl.of(type, members);
    return errorTracker.checkNoErrors(impl);
}

Upstream Design

So how do we work upstream from ImmutableLiteProcessor to ImmutableProcessor?

These classes rely on generic infrastructure in the org.example.processor.base package:

• interface LiteProcessor
  • Lightweight, simple version of Java's Processor interface
  • Contains a single method: void process(Set<? extends TypeElement> annotations, RoundEnvironment roundEnv) throws Exception
• abstract class IsolatingLiteProcessor<E extends Element> implements LiteProcessor
  • Designed for isolating annotation processors where each output is generated from a single input
  • Contains a single abstract method: void process(E annotatedElement) throws Exception
• abstract class AdapterProcessor implements Processor
  • Adapts a LiteProcessor to a Processor
  • The LiteProcessor is provided via an abstract method: LiteProcessor createLiteProcessor(ProcessingEnvironment processingEnv)

Here is how the annotation processor for @Immutable consumes this infrastructure:

• final class ImmutableLiteProcessor extends IsolatingLiteProcessor<TypeElement>
• final class ImmutableProcessor extends AdapterProcessor

End-to-End Testing

For end-to-end-testing, Google's Compile Testing framework is used.

Setup (JDK 16+)

To use Compile Testing with JDK 16+, add these lines to build.gradle.kts:

tasks.named<Test>("test") {
    // See: https://github.com/google/compile-testing/issues/222
    jvmArgs("--add-exports=jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED")
    jvmArgs("--add-exports=jdk.compiler/com.sun.tools.javac.main=ALL-UNNAMED")
    jvmArgs("--add-exports=jdk.compiler/com.sun.tools.javac.util=ALL-UNNAMED")
}

Compiling the Code

TestCompiler serves as a useful wrapper around the Compile Testing framework.

See this snippet to compile a single source with ImmutableProcessor:

Compilation compilation = TestCompiler.create().compile(sourcePath);

Without TestCompiler, you could also directly compile this source with this snippet:

Compilation compilation = Compiler.javac()
        .withProcessors(new ImmutableProcessor())
        // Suppress this warning: "Implicitly compiled files were not subject to annotation processing."
        .withOptions("-implicit:none")
        .compile(JavaFileObjects.forResource(sourcePath));

Verifying the Compilation

TestCompiler also verifies that the compilation succeeded or failed. By default, it expects that the compilation will succeed.

See this snippet from ImmutableProcessorTest, where a compilation failure is expected:

TestCompiler.create().expectingCompilationFailure().compile(sourcePath);

Compile Testing also provides fluent assertions. Here is the static import to use those assertions:

import static com.google.testing.compile.CompilationSubject.assertThat;

Source files are stored in resource folders:

• Example source files live in the test folder:
  • Rectangle.java
  • ColoredRectangle.java
  • Empty.java
• The expected generated source files live in the generated/test folder:
  • ImmutableRectangle.java
  • ImmutableColoredRectangle.java
  • ImmutableEmpty.java

Unit Testing

Many design decisions were made with testability in mind. Case in point, most classes have a corresponding test class.

Overview

The two-stage design of the annotation processor facilitates testability as well.

More specifically, ImmutableImpl is a pure type that can easily be created directly:

• For testing the modeler stage, an ImmutableImpl (or other model types) can be used as the expected value.
• For testing the generator stage, an ImmutableImpl can be used as the starting point.

TestImmutableImpls provides pre-built ImmutableImpl's that correspond to the examples sources:

• TestImmutableImpls.rectangle()
• TestImmutableImpls.coloredRectangle()
• TestImmutableImpls.empty()

Testability Challenges

Here are the core testability challenges:

• In the modeler stage, it is costly and/or difficult to directly create or mock out the various Element's.
• In the generator stage, it is (somewhat less) costly and/or difficult to directly create or mock out a Filer.
  • The unit tests verify the conversion of a model type (e.g., ImmutableImpl) to source code.
  • The end-to-end tests for this stage do mock out a Filer (via Mockito). See ImmutableGeneratorTest.

modeler Stage

Strategy

The unit testing strategy for the modeler stage is built around custom annotation processors:

1. The custom annotation processor creates a Java object.
2. The annotation processor serializes that Java object to JSON. (Recall that ImmutableImpl is serializable.)
3. The annotation processor writes that JSON to a generated resource file (instead of a generated source file).
4. The test reads and deserializes that resource file to obtain the Java object.
5. The test verifies the contents of the deserialized Java object.

Creating Custom Annotation Processors

TestCompiler.create() has another overload: TestCompiler.create(Class<? extends LiteProcessor> liteProcessorClass)

• It uses TestImmutableProcessor, which uses a custom implementation of LiteProcessor.
• A unit test can create a test implementation of LiteProcessor.
  • See ImmutableImplsTest.TestLiteProcessor for an example.
  • Each test implementation of LiteProcessor also needs to be added to TestProcessorModule.

Generating Resource Files

TestResources saves Java objects to generated resource files and then loads those objects.

See this snippet from ImmutableImplsTest.TestLiteProcessor, which saves the Java object:

@Override
protected void process(TypeElement typeElement) {
    implFactory.create(typeElement).ifPresent(impl -> TestResources.saveObject(filer, typeElement, impl));
}

...and this snippet from ImmutableImplsTest, which loads the Java object and verifies it:

private void create(String sourcePath, ImmutableImpl expectedImpl) throws Exception {
    Compilation compilation = TestCompiler.create(TestLiteProcessor.class).compile(sourcePath);
    ImmutableImpl impl = TestResources.loadObjectForSource(compilation, sourcePath, new TypeReference<>() {});
    assertThat(impl).isEqualTo(expectedImpl);
}