Skip to content

Solutions for instrumenting application flow tracking API calls into an existing code base in a non-invasive way

License

Notifications You must be signed in to change notification settings

GoodGrind/ghostwriter

Repository files navigation

GhostWriter

What is GhostWriter?

GhostWriter helps people to reason about their application, catch and pinpoint bugs quickly without writing any lines of extra code or deploying additional services

Tracing your application flow

See GhostWriter with the tracer runtime in action: Tracer in action

How does it work?

GhostWriter takes a different approach of debugging and bug tracking by writing code instead of you at the creation of the application. When you launch the GhostWriter-augmented final product it catches unexpected errors, tracks variable states and method calls so you or your team can see what’s the problem but more importantly, why is it happening. You can attach any application or tool to the interface of the application and receive the messages that GhostWriter creates.

Can you give me a more technical description?

GhostWriter is a non-invasive solution for adding common application event handler stubs to your Java code. Essentially what you would end up writing by hand if you would like to have proper tracking of your application workflow. After the instrumentation you will have the chance to provide custom handlers for all:

  • method calls, including entering, return and exiting events (some methods are excluded by default, see Configuration options and Excluding methods)

  • state changes of your application, such as value assignment and other side-effectful operations

  • unexpected errors

A "pseudo" before-after setup with GhostWriter (more or less readable decompiled code):

ghostwriter-tracer
ℹ️
You always see and work with your original source. The above picture is just an illustration.

Installation

Just add the necessary dependencies to your application and GhostWriter will take care of instrumenting the ghostwriter-api calls.

There are two ways to use GhostWriter: using annotation processing, or directly calling the instrumentation logic to enhance your classes. The former method is now deprecated, because it only supports Java 7 and 8.

GhostWriter Direct Instrumentation

First, add a tracing as a runtime dependency:

dependencies {
    runtime 'io.ghostwriter:ghostwriter-rt-tracer:0.5.0'
    runtime 'io.ghostwriter:ghostwriter-rt-tracer-slf4j:0.5.0' // Include this as well for SLF4J based logging.
}

Then, add the instrumentation component as a dependency to the build scripts, and integrate it to the compilation:

buildscript {
    dependencies {
        classpath "io.ghostwriter:ghostwriter:0.8.0"
    }
}
task instrument(type: JavaExec) {
    main = 'io.ghostwriter.GhostWriterClassFileTransformer'
    args = [sourceSets.main.java.outputDir.absolutePath]
    // Supply configuration by setting JVM arguments. In this case, disable value assignment tracking.
    jvmArgs = ['-DGHOSTWRITER_TRACE_VALUE_CHANGE=false']
    classpath = buildscript.configurations.classpath
    classpath += sourceSets.main.runtimeClasspath
}
compileJava.finalizedBy instrument

For all configuration options, see Configuration options .

GhostWriter Annotation Processor (Legacy, supports Java 8 and Java 7 only)

You have to choose the dependency that is inline with your Java compiler (JDK) version.

For Java 7 this means ghostwriter-jdk-v7, for Java 8 it is ghostwriter-jdk-v8.

GhostWriter is tested and verified with both Oracle JDK and Open JDK versions.

For detailed instructions keep reading on, if you just want to have a quick overview on a working sample, check here.

Maven

Add another dependency entry to the dependencies section. This means that for JDK8, you will have the following entry in you pom.xml file.

<dependency>
    <groupId>io.ghostwriter</groupId>
    <artifactId>ghostwriter-jdk-v8</artifactId>
    <version>0.7.2</version>
    <scope>compile</scope>
</dependency>

Your are done! Time to recompile your application!

Gradle

The provided snippets work with Gradle 3+. The dependencies of GhostWriter are fetched from Maven Central. So you have to add it to your list of repositories if its not there yet. For a project that uses JDK8, you’ll will have to add the following lines to your build.gradle file.

repositories {
    mavenCentral()
}

dependencies {
    compile "io.ghostwriter:ghostwriter-jdk-v8:0.7.2"
}

Your are done! Time to recompile your application!

For older Java versions (Java 7)

If you are still using an older Java/JDK version such as Java 7, you’ll need a different set of dependencies.

Maven

<dependency>
    <groupId>io.ghostwriter</groupId>
    <artifactId>ghostwriter-jdk-v7</artifactId> // (1)
    <version>0.7.2</version>
    <scope>compile</scope>
</dependency>
  1. Note the use of ghostwriter-jdk-v7.

Gradle

repositories {
    mavenCentral()
}

dependencies {
    compile "io.ghostwriter:ghostwriter-jdk-v7:0.7.2" // (1)
}
  1. Note the use of ghostwriter-jdk-v7

Now recompile your application and if all goes well, you should now have support for plugging in runtime implementations.

Explicitly specifying the compile time annotation

This steps should only be done in case you manually set annotation processors (for whatever reason). By default the compiler should pick up the GhostWriter annotation processor based on the service loader contract.

Maven

To have it explicitly set, you’ll need to add the following lines to your pom.xml.

<build>
     <plugins>
         <plugin>
             <groupId>org.apache.maven.plugins</groupId>
             <artifactId>maven-compiler-plugin</artifactId>
             <version>3.6.0</version>
             <executions>
                 <execution>
                     <id>default-compile</id>
                     <phase>compile</phase>
                     <goals>
                         <goal>compile</goal>
                     </goals>
                     <configuration>
                         <!-- This is how we enable GhostWriter, the rest is more or less boilerplate of Maven -->
                         <annotationProcessors>
                             <annotationProcessor>io.ghostwriter.openjdk.v8.GhostWriterAnnotationProcessor</annotationProcessor> // (1)
                         </annotationProcessors>
                         <source>1.8</source>
                         <target>1.8</target>
                     </configuration>
                 </execution>
             </executions>
        </plugin>
     </plugins>
 </build>
  1. Make sure to use the correct annotation processor, for Java 7 this would be io.ghostwriter.openjdk.v7.GhostWriterAnnotationProcessor

The important part is the specification of the annotation processor using the annotationProcessor tag. The rest is more or less Maven foreplay.

Gradle

In Gradle, that is done by adding the following snippet to your build.gradle file.

compileJava {
    options.compilerArgs = [
            // use the GhostWriter preprocessor to compile Java classes
            "-processor", "io.ghostwriter.openjdk.v8.GhostWriterAnnotationProcessor" // (1)
    ]
}
  1. Make sure to use the correct version, for Java 7 this would be io.ghostwriter.openjdk.v7.GhostWriterAnnotationProcessor

Is it working?

Set the following environmental variable to track what kind of code GhostWriter writes instead of you.

export GHOSTWRITER_VERBOSE=true

You should see something like this:

ghostwriter verbose output

As you can see there are a lot of Note: outputs that dump the instrumented code.

Configuring

Configuration options can be set as compiler options. For example:

subprojects {
    afterEvaluate {
        tasks.withType(JavaCompile) {
            options.compilerArgs.addAll(["-AGHOSTWRITER_EXCLUDE=my.package.SomeClass,my.package.subpackage"]);
        }
    }
}

For all configuration options, see Configuration options .

Selecting a runtime handler

Enhancing your application with GhostWriter is half the battle. You still need that data after all! With the no-operations stubs you won’t get much benefit from GhostWriter, however this is where GhostWriter shines! You can leverage one of the multiple runtime implementations available or roll your own!

Tracing your application - for the times when you don’t have your handy debugger at your disposal and you want to find out exactly what is going on in you application.

Capturing error snapshots - giving you better exceptions by providing the exact and detailed application state that led to the unexpected error and thus helping you battle Heisenbugs!

Do whatever you want! - provide your own solution for handling the data you get!

Controlling instrumentation

In some cases you might be inclined to change the default behaviour of the instrumentation steps. Currently there are 2 ways to do this. If you want to disable an instrumentation steps for you entire project, use the appropriate configuration option otherwise stick to the annotations provided by the API.

Configuration options

From the following configuration options can be set.

Instrumentation task Description Configuration option Default value

Logging

Log the exact steps GhostWriter does to your application along with the pretty printed instrumented code

GHOSTWRITER_VERBOSE

false

Overall instrumentation

Disable or enable the code instrumentation during compile time

GHOSTWRITER_INSTRUMENT

true

Annotated-only mode

GhostWriter will only instrument code that is explicitly marked with an annotation

GHOSTWRITER_ANNOTATED_ONLY

false

Excluding classes and packages

GhostWriter will not instrument code that is excluded. See Excluding classes

GHOSTWRITER_EXCLUDE

none

Excluding methods

GhostWriter will not instrument methods that are excluded. See Excluding methods

GHOSTWRITER_EXCLUDE_METHODS

toString, equals, hashCode, compareTo

Excluding short methods

GhostWriter will not instrument methods with the amount of statement is under or equal to the limit. See Excluding short methods

GHOSTWRITER_SHORT_METHOD_LIMIT

none

Entering and exiting

Event for entering and exiting a method

Not yet supported

true

Returning

Event for returning a value from a function

GHOSTWRITER_TRACE_RETURNING

true

Value change

Event generated by value assignments and changes

GHOSTWRITER_TRACE_VALUE_CHANGE

true

On error

Event generated by an uncaught exception in a method

GHOSTWRITER_TRACE_ON_ERROR

true

Excluding

Excluding classes

You can exclude classes from instrumentation - without modifying the source code - by setting GHOSTWRITER_EXCLUDE to a comma-separated list of package name and class names. For example, the following will exclude the class my.package.SomeClass and all classes in my.package.subpackage:

GHOSTWRITER_EXCLUDE=my.package.SomeClass,my.package.subpackage

Excluding methods

Methods can be also excluded globally by setting GHOSTWRITER_EXCLUDE_METHODS to a comma separated list of method names. For example:

GHOSTWRITER_EXCLUDE_METHODS=toString,equals

Setting this variable will overwrite the default excluded methods. If no methods should be excluded, set this variable to an empty string

GHOSTWRITER_EXCLUDE_METHODS=""

Excluding short methods

Methods can be excluded based on their size, meaning the amount of instruction it contains. By setting GHOSTWRITER_SHORT_METHOD_LIMIT with a valid integer, every method that has less or equal amount of instruction will be excluded.

Annotations

The fine grained instrumentation control is achieved using the annotations provided by the ghostwriter-api module.

Exclude a class

If you have a class, which you don’t want to trace at all, just put the @Exclude annotation on the class declaration itself. This signals to GhostWriter that all methods should be skipped. Usually you would do this for classes that handle sensitive information.

@Exclude
public class ExcludedTopLevelClass {

    // this method won't be traced
    public int meaningOfLife() {
        return 42;
    }

}

Exclude an entire method

By putting the @Exclude annotation on a method GhostWriter completely skips it. Primary use case is to exclude the performance sensitive methods of the application.

@Exclude // the annotation signals the GhostWriter instrumenter to ignore this method
public int excludedMethod() {
    int i = 3;
    // ...
    return i;
}

Exclude a method parameter

Sometimes you just want to ignore some sensitive data (password, credit card number, …​) that passes through you application. You can do so by excluding that specific parameter.

public void login(String userName, @Exclude char[] password) {
        // ...
}

In the above example, the password parameter and its value will not be part of the entering event.

Excluding local variables

Sensitive data can also occur inside method implementations, so you can also apply the exclusion to local variables as well.

public void buyAllTheThings() {
    // ...
    @Exclude String creditCardNumber;
    // ...
}

In annotated-only mode, include a specific class

By default, the @Include annotations are ignored. These annotations are only used if the GHOSTWRITER_ANNOTATED_ONLY environmental variable is set to true. In that case, only classes that are marked with the @Include annotation are instrumented. As before, the @Exclude annotations still behave the same way.

@Include
class MyClass {

   public void myMethod() {
      // this will be instrumented
   }

   @Exclude
   public void myOtherMethod() {
      // this will not be instrumented
   }

}

In annotated-only mode, include a specific method

Assuming that annotated-only mode is enabled (see GHOSTWRITER_ANNOTATED_ONLY), we can opt-in to instrumenting specific methods. By annotating a method of a class, GhostWriter will only instrument that specific method if the class itself is not annotated with @Include.

class BestClassEver {

   public void aMethod() {
      // this will not be instrumented
   }

   @Include
   public void theMethodIWantToTrace() {
      // this will be instrumented
   }

}

Contributing

First and foremost thank you for putting in the effort and time that is needed to contribute!

For smaller changes, just create a pull request and make sure that the automated tests still pass and that your changes are inline with the code quality checks. Providing additional documentation and test coverage is always welcome!

For bigger changes (API, new features, …​) consider opening an issue first so it can be discussed.

Getting help

If you have a quick question or stumble upon a bug feel free to open an issue or ask on Gitter.

FAQ

What about the performance impact?

By default GhostWriter uses no-op stubs, so the performance heavily depends on the runtime implementation you use. The JVM does an awesome job of optimizing the generated code and the end performance depends on your application behaviour as well. In case of performance critical section the instrumentation can be skipped by applying the correct annotation in order to minimize the performance overhead.

What about 3rd party code? Will that have the same stubs instrumented

Only if you compile that yourself. Potentially you can compile your own rt.jar with GhostWriter and have full blown coverage! The general consideration with the compile-time instrumenter implementation is that you should focus on the code that is in your control.

Will it mess with my stack traces? Like referring to line numbers that do not exist in my original source code?

No. The code instrumenter implementation makes sure that it is non-invasive and your stack traces refer to the correct source lines.

Why not a Java agent based solution?

At the end of the day this is about trade-offs and implementation details. With the current approach you get type-safety (the compiler verifies that the instrumented code is correct) and there is no application startup performance penalty. Plus, once you compiled your code, it is only a matter of providing dependencies. Even if you are not in control of specifying how your application/library is used you still have tracing support. Of course, the current implementation also has disadvantages. In the long run both compile-time and run-time implementation will be supported. Depending on your use case (library vs. application), you can pick the one that fits your needs. The acceptance testing infrastructure is in place for verifying the instrumentation steps, so feel free to contribute a solution ;)

I put the tracer related jars into my application’s classpath, yet they are ignored and noop is selected, why?

Ghostwriter uses the Java ServiceLoader to find the tracer implementation. In ghostwriter-api 0.4.0, ghostwriter-tracer 0.3.1 and earlier versions the default context classloader of the current thread was used. In some cases it was not set, and the system classloader was used as a fallback, which most probably did not contain the tracer jars (one example is when the application is deployed as a war into a Wildfly server). This behaviour is changed, and now it uses the classloader which loaded the Ghostwriter api classes.

About

Solutions for instrumenting application flow tracking API calls into an existing code base in a non-invasive way

Topics

Resources

License

Stars

Watchers

Forks

Packages

No packages published

Languages