Smoke test improvements

[trask]

Improve smoke test harness

• Test against different application servers and different JVMs

Add more smoke tests, starting with the most fragile instrumentation we have:

• Log4j 1 instrumentation (EDIT: removed until after GA)
• Log4j 2 instrumentation (EDIT: removed until after GA)
• java.util.logging instrumentation (EDIT: removed until after GA)
• Logback instrumentation (EDIT: removed until after GA)
  • Need to be careful not to double capture when one logging framework delegates to another
  • JBoss/Wildfly has it's own unique java.util.logging implementation
• gRPC instrumentation
  • gRPC instrumentation is fragile because of the shading conflicts involved with gRPC Context used internally by embedded OpenTelemetry API
• OpenTelemetry API instrumentation
  • OpenTelemetry API instrumentation is fragile because of the shading involved to bridge the user brought OpenTelemetry API and the embedded OpenTelemetry API

[trask]

More things to smoke test:

• Exporter configuration (OTLP, Jaeger, Zipkin, Prometheus)
• B3 Propagator configuration

[trask]

Another thing to improve:

• Don't use logging exporter / log parsing, which is unreliably under concurrent logging

[iNikem]

Based GA requirements I will narrow down the scope of this task.

Smoke test consists of running given application (Spring Boot, Wildlfy) with Otel java auto instrumentation agent attached. Agent should be configured to run given exporter (OTLP, Jaeger, Zipkin) and to send data to locally run Otel Collector (also started by the test harness). Collector should be configured to use file exporter. Test harness should execute requests against the application and verify that all expected data was written by Collector to the export file.

Test harness should be run on all LTS versions of starting from 8 plus on the latest java version if not LTS.

[iNikem]

I have a working prototype, which runs Spring Boot application with java agent attached and the Collector and then makes a request to the app. Collector is configured to use file exporter. Test then reads that file written by the Collector and verifies that it contains expected spans.

My prototype differs from current tests and I want to discuss it before making a PR. First, it uses testcontainers to run both the application and the Collector within Docker containers.

Second, I don't want to actually build test applications on every run. They change quite rarely. I am thinking about having them as separate Gradle builds in our repo, build them separately and publish Docker images to our Bintray organisation.

Your opinions?

[anuraaga]

Just wondering, is there a big advantage in running the test app inside the container instead of directly? If we run them directly, maybe the Gradle build of the test app isn't too slow compared to the rest of the build.

Otherwise, publishing Docker images sounds fine to me too, we may as well use GitHub Packages instead of Bintray

[anuraaga]

Ah, I guess it means the agent can be separated from the test VM. SGTM

[iNikem]

I still don't want to burn electricity to build the same apps again and again. Also, play test app is one of the few remaining sources of Gradle warnings.

[anuraaga]

Well, we're optimistic of Gradle build cache right? :-)

[iNikem]

Well, we're optimistic of Gradle build cache right? :-)

Hm, that's a good point. But I still don't like Gradle warnings :)

[anuraaga]

Makes sense - but yeah let's try out GitHub Packages to reduce external dependencies.

https://docs.github.com/en/packages/using-github-packages-with-your-projects-ecosystem/configuring-docker-for-use-with-github-packages