Java Agent for OpenTracing

NOTE: It is deprecated now. Please migrate to OpenTelemetry Instrumentation.

Automatically instruments 3rd-party libraries in Java applications

What is SpecialAgent?

The SpecialAgent is software that attaches to Java applications, and automatically instruments 3rd-party libraries within. The SpecialAgent uses the OpenTracing API for Integrations that instrument 3rd-party libraries, as well as Trace Exporters that export trace data to OpenTracing Tracer vendors. The architecture of SpecialAgent was specifically designed to include contributions from the community, whereby its platform automates the installation of OpenTracing Integrations written by individual contributors. In addition to Integrations, the SpecialAgent also supports Trace Exporters, which connect an instrumented runtime to OpenTracing-compliant tracer vendors, such as LightStep, Wavefront, or Jaeger. Both the Integrations and the Trace Exporters are decoupled from SpecialAgent -- i.e. neither need to know about SpecialAgent. At its core, the SpecialAgent is itself nothing more than an engine that abstracts the functionality for the automatic installation of Integrations, and their connection to Trace Exporters. A benefit of this approach is that the SpecialAgent intrinsically embodies and encourages community involvement.

Both the Integrations and the Trace Exporters are open-source, and are developed and supported by the OpenTracing community.

The SpecialAgent supports Oracle Java and OpenJDK.

Table of Contents

  1 Introduction
  2 Quick Start
    2.1 Installation
       2.1.1 In Application
           2.1.1.1 Stable
           2.1.1.2 Development
       2.1.2 For Development
           2.1.2.1 Integrations
               2.1.2.1.1 Uncoupled Integrations
               2.1.2.1.2 Coupled Integrations
               2.1.2.1.3 Development of Integration Rules
           2.1.2.2 Trace Exporters
               2.1.2.2.1 Short Name
    2.2 Usage
       2.2.1 Static Attach
       2.2.2 Dynamic Attach
       2.2.3 Static Deferred Attach
  3 Configuration
    3.1 Overview
    3.2 Properties
       3.2.1 Logging
       3.2.2 Integration
       3.2.3 General
    3.3 Selecting the Trace Exporter
    3.4 Disabling Integration Rules
       3.4.1 Disabling All Integration Rules
       3.4.2 Disabling (or enabling) One Integration Rule
       3.4.3 Disabling AgentRules of an Integration Rule
    3.5 Disabling Trace Exporters
    3.6 Including custom Integration Rules
    3.7 Rewritable Tracer
  4 Supported Integrations and Trace Exporters
    4.1 Integrations
    4.2 Trace Exporters
    4.3 Libraries instrumented via other Integrations
  5 Objectives
    5.1 Goals
    5.2 Non-Goals
  6 Definitions
    6.1 Tracer
    6.2 Trace Exporter
    6.3 Integration
    6.4 Integration Rule
  7 Credits
  8 Contributing
  9 License

1 Introduction

This file contains the operational instructions for the use and development of SpecialAgent.

2 Quick Start

The SpecialAgent is a Java Agent that attaches to an application (either statically or dynamically). Once attached, the SpecialAgent it loads its bundled Integrations, which are all enabled by default.

The SpecialAgent is stable -- any exception that occurs during attachment of SpecialAgent will not adversely affect the stability of the target application. It is, however, important to note that SpecialAgent bundles Integrations that are developed by 3rd parties and individual contributors. We strive to assert the stability of SpecialAgent with rigorous integration tests, yet it is still possible that the code in a bundled Integration may result in an exception that is not properly handled, which could potentially destabilize a target application.

2.1 Installation

The Maven build of the SpecialAgent project generates 2 artifacts: main and test. These artifacts can be obtained by downloading directly from Maven's Central Repository, or by cloning this repository and following the Development Instructions.

2.1.1 In Application

The SpecialAgent is contained in a single JAR file. This JAR file is the main artifact built by Maven, and bundles the Integrations from opentracing-contrib for which Integration Rules have been implemented.

To use the SpecialAgent on an application, please download the stable or development main artifact.

The artifact JAR can be provided to an application with the -javaagent:${SPECIAL_AGENT_JAR} vm argument for Static Attach and Static Deferred Attach. The artifact JAR can also be executed in standalone fashion, which requires an argument to be passed for the PID of a target process to which SpecialAgent is to dynamically attach. Please refer to Usage section for usage instructions.

2.1.1.1 Stable

The latest stable release is: 1.7.4

wget -O opentracing-specialagent-1.7.4.jar "https://repo1.maven.org/maven2/io/opentracing/contrib/specialagent/opentracing-specialagent/1.7.4/opentracing-specialagent-1.7.4.jar"

2.1.1.2 Development

The latest development release is: 1.7.5-SNAPSHOT

wget -O opentracing-specialagent-1.7.5-SNAPSHOT.jar "https://oss.sonatype.org/service/local/artifact/maven/redirect?r=snapshots&g=io.opentracing.contrib.specialagent&a=opentracing-specialagent&v=LATEST"

Note: Sometimes the web service call (in the line above) to retrieve the latest SNAPSHOT build fails to deliver the correct download. In order to work around this issue, please consider using the following command (for Linux and Mac OS):

wget -O opentracing-specialagent-1.7.5-SNAPSHOT.jar $(curl -s https://oss.sonatype.org/content/repositories/snapshots/io/opentracing/contrib/specialagent/opentracing-specialagent/1.7.5-SNAPSHOT/ | grep '".*\d\.jar"' | tail -1 | awk -F\" '{print $2}')

2.1.2 For Development

The SpecialAgent is built in 2 passes utilizing different profiles:

1. The default profile is used for development of Integration Rules. It builds and runs tests for each rule, but does not bundle the rules into the main JAR (i.e. opentracing-specialagent-1.7.5-SNAPSHOT.jar).

   To run this profile:

   mvn clean install

   Note: If you skip tests, the assemble profile will display an error stating that tests have not been run. See Convenient One-Liners for quick ways to build and package SpecialAgent.

2. The assemble profile is used to bundle the Integrations into the main JAR (i.e. opentracing-specialagent-1.7.5-SNAPSHOT.jar). It builds each rule, but does not run tests. Once the build with the assemble profile is finished, the main JAR (i.e. opentracing-specialagent-1.7.5-SNAPSHOT.jar) will contain the built rules inside it.

   Note: If you do not run this step, the opentracing-specialagent-1.7.5-SNAPSHOT.jar from the previous step will not contain any Integrations!

   Note: It is important to not run Maven's clean lifecycle when executing the assemble profile, otherwise the Integrations built in with the default profile will be cleared.

   To run this profile:

   mvn -Dassemble install

• For a one-line build command to build SpecialAgent, its rules, run all tests, and create the assemble package:

  mvn clean install && mvn -Dassemble install

Convenient One-Liners

1. Skipping tests when building SpecialAgent.

   mvn -DskipTests clean install

2. Skipping compatibility tests when building SpecialAgent rules.

   mvn -DskipCompatibilityTests clean install

3. Packaging SpecialAgent with rules that skipped test execution.

   mvn -Dassemble -DignoreMissingTestManifest install

2.1.2.1 Integrations

The SpecialAgent supports two kinds of Integrations:

2.1.2.1.1 Uncoupled Integrations

Uncoupled Integrations are those that can be used without SpecialAgent. These Integrations are not coupled to SpecialAgent, and can be used via manual integration in an application.

Uncoupled Integrations are implemented in opentracing-contrib, and do not know about SpecialAgent.

To support Uncoupled Integrations, SpecialAgent requires the implementation of an Integration Rule that bridges the Uncoupled Integrations to SpecialAgent's auto-instrumentation mechanism.

The implementation of Integrations as uncoupled is preferred, as this pattern allows users to instrument their applications manually, if so desired. However, not all 3rd-party libraries can be instrumented to allow manual integration, leaving the alternative pattern: Coupled Integrations.

2.1.2.1.2 Coupled Integrations

Coupled Integrations are those that can only be used with SpecialAgent. These Integrations are coupled to SpecialAgent, and can only be used via automatic installation in an application.

Coupled Integrations are effectively Integration Rules that implement the full scope of the instrumentation of the 3rd-party library, and directly bridge this integration into the SpecialAgent's auto-instrumentation mechanism.

The implementation of Integrations as coupled is discouraged, as this pattern prohibits users from instrumenting their applications manually, if so desired. However, not all 3rd-party libraries can be instrumented to allow manual integration, leaving Coupled Integrations as the only option.

2.1.2.1.3 Development of Integration Rules

For development of Integration Rules, import the opentracing-specialagent-api and test-jar of the opentracing-specialagent.

<properties>
  <special-agent-version>1.7.4</special-agent-version> <!-- 1.7.5-SNAPSHOT -->
</properties>
...
<dependency>
  <!-- Allows you to write Integration Rules for Special Agent -->
  <groupId>io.opentracing.contrib.specialagent</groupId>
  <artifactId>opentracing-specialagent-api</artifactId>
  <version>${special-agent-version}</version>
</dependency>
<dependency>
  <!-- Allows the Integration Rules to use OpenTracing API -->
  <groupId>io.opentracing.contrib.specialagent</groupId>
  <artifactId>opentracing-adapter</artifactId>
  <version>${special-agent-version}</version>
</dependency>
<dependency>
  <!-- Allows the Integration Rules be tested with SpecialAgent's `AgentRunner` -->
  <groupId>io.opentracing.contrib.specialagent</groupId>
  <artifactId>opentracing-specialagent</artifactId>
  <version>${special-agent-version}</version>
  <type>test-jar</type>
  <scope>test</scope>
</dependency>
<dependency>
  <!-- Needed for JUnit tests from IDE -->
  <groupId>org.apache.maven</groupId>
  <artifactId>maven-model</artifactId>
</dependency>

The test-jar is the test artifact that contains the AgentRunner class, which is a JUnit runner provided for testing of the ByteBuddy auto-instrumentation rules. This JAR does not contain Integration Rules themselves, and is only intended to be applied to the test phase of the build lifecycle of a single Integration Rule implementation.

For direction with the development of Integration Rules, please refer to the opentracing-specialagent-api module.

2.1.2.2 Trace Exporters

Trace Exporters integrate with the SpecialAgent via the OpenTracing TracerResolver, which connects the SpecialAgent to a Tracer.

Trace Exporters integrate to the SpecialAgent via the SPI mechanism defined in the TracerResolver, and are therefore not coupled to the SpecialAgent.

Trace Exporters must be provided as "fat JARs" that contain the full set of all classes necessary for operation.

If the Trace Exporter JAR imports any io.opentracing:opentracing-* dependencies, the io.opentracing.contrib:opentracing-tracerresolver, or any other OpenTracing dependencies that are guaranteed to be provided by SpecialAgent, then these dependencies MUST BE excluded from the JAR, as well as from the dependency spec.

Trace Exporters are integrated with the SpecialAgent by specifying a dependency in the <isolatedDependencies> configuration element of the specialagent-maven-plugin in the root POM. For instance, the dependency for the Jaeger Trace Exporter is:

<isolatedDependencies>
...
  <dependency>
    <groupId>io.opentracing.contrib</groupId>
    <artifactId>jaeger-client-bundle</artifactId>
  </dependency>
...
</isolatedDependencies>

2.1.2.2.1 Short Name

Each Trace Exporter integrated with the SpecialAgent must define a Short Name, which is a string that is used to reference the plugin with the -Dsa.exporter=${SHORT_NAME} system property. To provide a Short Name for the Trace Exporter, you must define a Maven property in the root POM with the name matching the artifactId of the Trace Exporter module. For instance, the Short Name for the Jaeger Trace Exporter is defined as:

<properties>
...
  <jaeger-client-bundle>jaeger</jaeger-client-bundle>
...
</properties>

2.2 Usage

The SpecialAgent is used by attaching to a target application. Once attached, the SpecialAgent relies on Java’s Instrumentation mechanism to transform the behavior of the application.

SpecialAgent supports the following attach modes:

Attach Mode	Number of Required Commands to Attach	Plugin and Rule Initialization Timeline
Static Attach	1 (sync)	Before app start (any application)
Dynamic Attach	2 (async)	After app start (any application)
Static Deferred Attach	1 (sync)	After app start (some applications)

2.2.1 Static Attach

With Static Attach, the application is executed with the -javaagent argument, and the agent initialization occurs before the application is started. This mode requires 1 command from the command line.

Statically attaching to a Java application involves the use of the -javaagent vm argument at the time of startup of the target Java application. The following command can be used as an example:

java -javaagent:opentracing-specialagent-1.7.4.jar -jar MyApp.jar

This command statically attaches SpecialAgent to the application in MyApp.jar.

2.2.2 Dynamic Attach

With Dynamic Attach, the application is allowed to start first, afterwhich an agent VM is dynamically attached to the application's PID. This mode requires 2 commands from the command line: the first for the application, and the second for the agent VM.

Dynamically attaching to a Java application involves the use of a running application’s PID, after the application’s startup. The following commands can be used as an example:

1. To obtain the PID of the target application:

   jps

2. To attach to the target PID:

   • For jdk1.8

     java -Xbootclasspath/a:$JAVA_HOME/lib/tools.jar -jar opentracing-specialagent-1.7.4.jar ${PID}

   • For jdk9+

     java -jar opentracing-specialagent-1.7.4.jar ${PID}

Note: Properties that are provided in the command to dynamically attach will be absorbed by the target application. This applies to properties specific to SpecialAgent, such as -Dsa.log.level=FINER, as well as other properties such as -Djava.util.logging.config.file=out.log.

Troubleshooting: If you encounter an exception stating Unable to open socket file, make sure the attaching VM is executed with the same permissions as the target VM.

2.2.3 Static Deferred Attach

With Static Deferred Attach, the application is executed with the -javaagent argument, but the agent initialization is deferred until the application is started. This mode requires 1 command from the command line, and is designed specifically for runtimes that have complex initialization lifecycles that may result in extraneously lengthy startup times when attached with Static Attach.

Static Deferred Attach is currently supported for:

1. Spring WebMVC (1.0 to LATEST).
2. Spring Boot (1.0.0.RELEASE to LATEST).

If the above supported application environment is detected, Static Deferred Attach is automatically activated.

To deactivate Static Deferred Attach, specify the following system property on the command line:

-Dsa.init.defer=false

The following command can be used as an example:

java -javaagent:opentracing-specialagent-1.7.4.jar -Dsa.init.defer=false -jar MySpringBootApp.jar

3 Configuration

3.1 Overview

The SpecialAgent exposes a simple pattern for configuration of SpecialAgent, the Integrations, as well as Trace Exporters. The configuration pattern is based on system properties, which can be defined on the command-line, in a properties file, or in @AgentRunner.Config for JUnit tests:

Configuration Layers

1. Properties passed on the command-line via -D${PROPERTY}=... override same-named properties defined in the subsequent layers.

2. The @AgentRunner.Config annotation allows one to define log level and re/transformation event logging settings. Properties defined in the @Config annotation override same-named properties defined in the subsequent layers.

3. The -Dsa.config=${PROPERTIES_FILE} command-line argument can be specified for SpecialAgent to load property names from a ${PROPERTIES_FILE}. Properties defined in the ${PROPERTIES_FILE} override same-named properties defined in the subsequent layer.

4. The SpecialAgent has a default.properties file that defines default values for properties that need to be defined.

3.2 Properties

The following properties are supported by all Integration Rules:

3.2.1 Logging

• -Dsa.log.level

  Set the logging level for SpecialAgent. Acceptable values are: SEVERE, WARNING, INFO, CONFIG, FINE, FINER, or FINEST, or any numerical log level value is accepted also. The default logging level is set to WARNING.

• -Dsa.log.events

  Set the re/transformation events to be logged: DISCOVERY, IGNORED, TRANSFORMATION, ERROR, COMPLETE. The property accepts a comma-delimited list of event names. By default, the ERROR event is logged (only when run with AgentRunner).

• -Dsa.log.file

  Set the logging output file for SpecialAgent.

3.2.2 Integration

1. Verbose mode

       -Dsa.integration.*.verbose
    -Dsa.integration.${INTEGRATION_NAME_PATTERN}.verbose

   Sets verbose mode for all plugins (i.e. *) or one plugin (i.e. ${INTEGRATION_NAME_PATTERN}). This property can also be set in an AgentRunner JUnit test with the @AgentRunner.Config(verbose=true) for all tests in a JUnit class, or @AgentRunner.TestConfig(verbose=true) for an individual JUnit test method.

   The Java Concurrent API plugin supports verbose mode, which is disabled by default. To enable, set sa.integration.concurrent.verbose=true. In non-verbose mode, parent span context is propagating to task execution (if a parent span context exists). In verbose mode, a parent span is always created upon task submission to the executor, and a child span is created when the task is started.

3.2.3 General

1. Skip fingerprint verification:

       -Dsa.fingerprint.skip

   Tells the SpecialAgent to skip the fingerprint verification when linking Integrations into class loaders. This option allows one to work around an unexpected fingerprint verification failure, which can happen in complex runtimes that do not contain all class definitions on the class path. It must be noted, however, that if the fingerprint verification is disabled, the SpecialAgent will indiscriminately install all plugins regardless of library version compatibility issues, which may lead to NoClassDefFoundError, IllegalAccessError, AbstractMethodError, LinkageError, etc.

3.3 Selecting the Trace Exporter

The SpecialAgent supports OpenTracing-compatible Tracers. There are 2 ways to connect a Tracer to the SpecialAgent runtime:

1. Bundled Trace Exporters

   The SpecialAgent bundles the following Trace Exporters:

   1. Jaeger Trace Exporter
   2. LightStep Trace Exporter
   3. Wavefront Trace Exporter
   4. OpenTelemetry Bridge Trace Exporter
   5. MockTracer

   The -Dsa.exporter=${SHORT_NAME} property specifies which Trace Exporter is to be used. The value of ${SHORT_NAME} is the Short Name of the Trace Exporter, i.e. jaeger, lightstep, wavefront, otel, or mock.

2. External Trace Exporters

   The SpecialAgent allows external Trace Exporters to be attached to the runtime.

   The -Dsa.exporter=${TRACE_EXPORTER_JAR} property specifies the JAR path of the Trace Exporter to be used. The ${TRACE_EXPORTER_JAR} must be a JAR that supplies an implementation of the TracerFactory interface of the TracerResolver project.

NOTE: If a tracer is not specified with the -Dsa.exporter=... property, the SpecialAgent will present a warning in the log that states: Tracer NOT RESOLVED.

3.4 Disabling Integration Rules

Integrations bundled with the SpecialAgent are enabled by default.

Multiple properties to disable or to enable all or individual plugins can be declared via the Configuration Pattern. The processing order of the properties is equal to the order of their declaration.

3.4.1 Disabling All Integration Rules

To disable all Integrations:

sa.integration.*.disable

^{The suffix .disable is interchangeable with .enable=false.}

3.4.2 Disabling (or enabling) One Integration Rule

To disable an individual Integration:

sa.integration.${INTEGRATION_NAME_PATTERN}.disable

^{The suffix .disable is interchangeable with .enable=false.}

Conversely, to enable an individual Integration.

sa.integration.${INTEGRATION_NAME_PATTERN}.enable

^{The suffix .enable is interchangeable with .disable=false.}

Integration Name Pattern

The value of ${INTEGRATION_NAME_PATTERN} represents the name of the Integration Rule, as specified in Integrations ("Integration Rule" column). The ${INTEGRATION_NAME_PATTERN} allows for the use of * and ? characters to match multiple rules simultaneously. For instance:

1. dubbo:2.?
   Matches all Dubbo rules, including dubbo:2.6, and dubbo:2.7.
2. cassandra:driver:*
   Matches all Cassandra Driver rules, including cassandra:driver:3, and cassandra:driver:4.
3. spring:web*
   Matches all Spring WebMVC, Spring WebFlux and Spring WebSocket rules, including spring:webmvc, spring:webflux, and spring:websocket.
4. cassandra:driver
   Matches all Cassandra Driver rules, including cassandra:driver:3 and cassandra:driver:4.

If the version part of the ${INTEGRATION_NAME_PATTERN} does not end with a * or ? character, a * will be appended automatically. Therefore:

1. dubbo:2
   Matches all Dubbo v2 rules, including dubbo:2.6, and dubbo:2.7.
2. cassandra:driver
   Matches all Cassandra Driver rules, cassandra:driver:3, and cassandra:driver:4.
3. spring
   Matches all Spring rules.
4. spring:w
   Does not match any rules.

3.4.3 Disabling AgentRules of an Integration Rule

To disable an individual AgentRule of an Integration:

sa.integration.${INTEGRATION_NAME_PATTERN}#${AGENT_RULE_SIMPLE_CLASS_NAME}.disable

^{The suffix .disable is interchangeable with .enable=false.}

The value of ${AGENT_RULE_SIMPLE_CLASS_NAME} is the simple class name of the AgentRule subclass that is to be disabled.

3.5 Disabling Trace Exporters

All Trace Exporters bundled in SpecialAgent are enabled by default, and can be disabled.

To disable all Trace Exporters:

sa.exporter.*.disable

^{The suffix .disable is interchangeable with .enable=false.}

To disable an individual Trace Exporter:

sa.exporter.${SHORT_NAME}.disable

^{The suffix .disable is interchangeable with .enable=false.}

The value of ${SHORT_NAME} is the Short Name of the plugin, such as jaeger, lightstep, wavefront, otel, or mock.

3.6 Including Custom Integration Rules

Custom Integration Rules can be implemented by following the SpecialAgent Rule API. JARs containing custom Integration Rules can be loaded by SpecialAgent via:

-Dsa.classpath=${JARs}

Here, ${JARs} refers to a File.pathSeparator-delimited (: for *NIX, ; for Windows) string of JARs containing the custom Integration Rules.

3.7 Rewritable Tracer

The Rewritable Tracer allows one to rewrite data in the spans created by Integrations without having to modify the source code.

The Rewritable Tracer is a rules engine that is configured via JSON files that conform to a specification.

For example:

• The following JSON defines a rule for all Integrations to drop all tags in spans matching key literal http.url and value regex .*secret.*.

  {
    "*": [
      {
        "input": {
          "type": "tag",
          "key": "http.url",
          "value": ".*secret.*"
        }
      }
    ]
  }

• The following JSON defines a rule for the jedis Integration to rewrite all logs matching key literal http.method as a tag.

{
  "jedis": [
    {
      "input": {
        "type": "log",
        "key": "http.method",
      },
      "output": {
        "type": "tag"
      }
    }
  ]
}

For a configuration spec and other use-case examples, please refer to the rewrite plugin.

4 Supported Integrations and Trace Exporters

4.1 Integrations

Intrinsically, the SpecialAgent includes support for the instrumentation of the following 3rd-party libraries. Each row refers to an Integration, the Integration Rule, and the minimum and maximum version tested by the build.

For the development of Integration Rules, please refer to the opentracing-specialagent-api module.

Integration (link to impl. of Integration)	Integration Rule (link to impl. of Integration Rule)	Min Version (min supported)	Max Version (max supported)
Akka Actor	akka:actor	2.5.0	LATEST
Akka Http	akka:http	10.1.0	LATEST
Apache Camel	camel	2.24.0	2.24.2
Apache CXF	cxf	3.3.3	LATEST
Apache Dubbo	dubbo:2.6	2.6.2	2.6.7
	dubbo:2.7	2.7.1	LATEST
Apache HttpClient	apache:httpclient	4.4	LATEST
Async Http Client	asynchttpclient	2.7.0	LATEST
AWS SDK	aws:sdk:1	1.11.79	LATEST
	aws:sdk:2	2.1.4	FIXME
Cassandra Driver	cassandra:driver:3	3.0.0	3.7.2
	cassandra:driver:4	4.0.0	LATEST
Couchbase Client	couchbase-client	2.7.3	2.7.13
Dynamic	dynamic (configurable)	*	*
Elasticsearch Client	elasticsearch:client-transport elasticsearch:client-rest	6.4.0 6.4.0	LATEST 6.8.7
Feign	feign	9.0.0	LATEST
Google Http Client	google-http-client	1.19.0	LATEST
Grizzly AsyncHttpClient	grizzly:ahc	1.15	LATEST
Grizzly HTTP Server	grizzly:http-server	2.3.35	LATEST
GRPC	grpc	1.7.3	LATEST
Hazelcast	hazelcast	3.12	3.12.6
Java Concurrent API [java.util.concurrent]	concurrent	1.5	11
Java JDBC API [java.sql]	jdbc (configurable)	3.1	4.3
Java JMS API [javax.jms]	jms	1.1-rev-1	LATEST
Java Servlet API [javax.servlet]	servlet (configurable)	2.3	3.1
Jetty		7.6.21.v20160908	9.2.15.v20160210
Tomcat		7.0.65	9.0.27
Java Thread [java.lang.Thread]	thread	1.0	11
HttpURLConnection [java.net.HttpURLConnection]	httpurlconnection	1.1	11
JAX-RS Client	jax-rs	2.0	LATEST
Jedis Client	jedis	2.7.0	3.3.0
Kafka Client	kafka:client	1.1.0	LATEST
Kafka Streams	kafka:streams	1.1.0	LATEST
Lettuce Client	lettuce	5.0.0.RELEASE	LATEST
MongoDB Driver	mongo:driver	3.9.0	LATEST
Mule 4 Artifact Module	mule:artifact-module:4	4.2.2	LATEST
Mule 4 Core	mule:core:4	4.2.2	LATEST
Neo4j Driver	neo4j:driver	4.0.0	LATEST
Netty	netty	4.1.0	4.1.46.Final
OkHttp	okhttp	3.5.0	LATEST
Play Framework	play	2.6.0	LATEST
Play WS	play:ws	2.0.0	FIXME
Pulsar Client	pulsar:client	2.2.0	FIXME
Pulsar Functions	pulsar-functions (configurable)	2.2.0	FIXME
RabbitMQ Client	rabbitmq:client	5.0.0	LATEST
Reactor	reactor	3.2.3.RELEASE	LATEST
Redisson	redisson	3.11.0	3.11.5
RxJava	rxjava:2	2.1.0	LATEST
	rxjava:3	3.0.0	LATEST
Spring JMS	spring:jms	5.0.0.RELEASE	LATEST
Spring Kafka	spring:kafka	2.2.0.RELEASE	LATEST
Spring Messaging	spring:messaging	5.1.0.RELEASE	5.3.0.RELEASE
Spring RabbitMQ	spring:rabbitmq	2.0.0.RELEASE	LATEST
Spring WebFlux	spring:webflux	5.1.0.RELEASE	LATEST
Spring Boot WebSocket STOMP	spring:websocket	2.1.0.RELEASE	LATEST
Spring [@Async and @Scheduled]	spring:scheduling	5.0.0.RELEASE	LATEST
Spring Web MVC	spring:webmvc	3.0.2.RELEASE	LATEST
Spymemcached	spymemcached	2.11.0	LATEST
Thrift	thrift	0.10.0	0.13.0
Zuul	zuul	1.0.0	2.1.1

4.2 Trace Exporters

Intrinsically, the SpecialAgent includes support for the following Trace Exporters. A demo can be referenced here.

Trace Exporter (link to impl. of trace exporter)	Short Name (-Dsa.exporter=${short_name})
Jaeger Trace Exporter (configuration reference)	jaeger
LightStep Trace Exporter (configuration reference)	lightstep
Wavefront Trace Exporter (configuration reference)	wavefront
OpenTelemetry Bridge Trace Exporter (configuration reference)	otel
MockTracer	mock

4.3 Libraries instrumented via other Integrations

The following libraries are instrumented by other Integration Rules.

Library	Integration Rule (link to impl. of Integration Rule)	Min Version (min supported)	Max Version (max supported)
Hystrix	concurrent	1.5	11
JDBI	jdbc (configurable)	*	*
Ratpack	netty	1.4.0	LATEST
Solr Client	apache:httpclient	4.0.0	LATEST
SparkJava	javax.servlet	2.2	LATEST
Spring Cloud	*	*	*
Spring Data	jdbc (configurable)	*	*
Spring Web	httpurlconnection	*	*
Twilio	apache:httpclient	0.0.1	LATEST

5 Objectives

5.1 Goals

1. The SpecialAgent must allow any Integration available in opentracing-contrib to be automatically installable in applications that utilize a 3rd-party library for which an Integration exists.
2. The SpecialAgent must automatically install the Integration for each 3rd-party library, regardless in which class loader the 3rd-party library is loaded.
3. The SpecialAgent must not adversely affect the runtime stability of the application on which it is intended to be used. This goal applies only to the code in the SpecialAgent, and transitively applies to the code of the Integration made available in opentracing-contrib.
4. The SpecialAgent must support Static Attach and Dynamic Attach for applications running on JVM versions 1.7, 1.8, 9, and 11.
5. The SpecialAgent must implement a lightweight test methodology that can be easily applied to a module that implements Integration for a 3rd-party library. This test must simulate:
   1. Launch the test in a process simulating the -javaagent vm argument that points to the SpecialAgent (in order to test auto-instrumentation functionality).
   2. Elevate the test code to be executed from a custom class loader that is disconnected from the system class loader (in order to test bytecode injection into an isolated class loader that cannot resolve classes on the system classpath).
   3. Allow tests to specify their own Tracer instances via GlobalTracer, or initialize a MockTracer if no instance is specified. The test must provide a reference to the Tracer instance in the test method for assertions with JUnit.
6. The SpecialAgent must provide a means by which Integrations can be configured for use on a target application.

5.2 Non-Goals

1. The SpecialAgent is not designed to modify application code, beyond the installation of Integrations. For example, there is no facility for dynamically augmenting arbitrary code.

6 Definitions

The following terms are used throughout this documentation.

6.1 Tracer

Service provider of the OpenTracing standard, providing an implementation of the io.opentracing.Tracer interface.

Examples:

• Jaeger Tracer
• LightStep Tracer
• Wavefront Tracer

_{Tracers are not coupled to the SpecialAgent.}

6.2 Trace Exporter

A bridge providing automatic discovery of Tracers in a runtime instrumented with the OpenTracing API. This bridge implements the TracerFactory interface of TracerResolver, and is distributed as a single "fat JAR" that can be conveniently added to the classpath of a Java process.

_{Trace Exporters are not coupled to the SpecialAgent.}

6.3 Integration

An OpenTracing Integration for a 3rd-party library, existing as individual repositories in opentracing-contrib.

Examples:

• opentracing-contrib/java-okhttp
• opentracing-contrib/java-jdbc
• opentracing-contrib/java-jms

_{Integrations are not coupled to the SpecialAgent.}

6.4 Integration Rule

A submodule of the SpecialAgent that implements the auto-instrumentation rules for Integrations via the opentracing-specialagent-api. See Integrations for a description of Uncoupled and Coupled Integrations.

Examples:

• rule/okhttp
• rule/jdbc
• [rule/jms-1][jms-1]

_{Integration Rules are coupled to the SpecialAgent.}

7 Credits

Thank you to the following contributors for developing Integrations and Integration Rules:

• Sergei Malafeev
• Jose Montoya
• Przemyslaw Maciolek
• Jianshao Wu
• Gregor Zeitlinger
• @limfriend

Thank you to the following contributors for developing Trace Exporters:

• Carlos Alberto Cortez
• Han Zhang

Thank you to the following developers for filing issues and helping us fix them:

• Louis-Etienne
• Marcos Trejo Munguia
• @kaushikdeb
• @deepakgoenka
• Prometheus
• Randall Theobald

Thank you to the following individuals for all other general contributions to the codebase:

• Daniel Rodriguez Hernandez
• qudongfang
• Pontus Rydin

Finally, thanks for all of the feedback! Please share your comments as an issue!

8 Contributing

Pull requests are welcome. For major changes, please open an issue first to discuss what you would like to change.

Please make sure to update tests as appropriate.

9 License

This project is licensed under the Apache 2 License - see the LICENSE.txt file for details.