Explain difference between OTel agent and Micrometer instrumentations

[marcingrzejszczak]

The suggested approach is to use Micrometer Observation bridge since all Spring projects are using that for instrumentation. One can then use Micrometer OTLP Meter Registry to be OTLP compatible. For Tracing, Micrometer Tracing is suggested with either Brave (once this projects is released in GA) or OTel bridge, in order to be OTLP compliant.

Usage of Agent is discouraged due to various restrictions:

• OpenTelemetry Java Instrumentation project contains multiple alpha JARs (e.g. check the release notes)
• You can't create a native image when using the agent
• Spring Boot OTel configuration is maintained by the Spring Boot team, other auto configurations may not be compatible with Spring Boot supported solutions
• Agent instrumentation is buggy (e.g. issue)

[bclozel]

Thanks Marcin for this suggestion. Rather than trying to compare technologies (we don't usually do that in our reference docs), it would make sense to clear up some points that could confuse developers:

• If your application is using the Observation API, I guess produced metrics and traces won't be exported if you're using the OTel agent?
• If you are using the micrometer bridge, the observation metadata described in Spring reference docs apply (see Spring for GraphQL or Spring Framework). If you're using the OTel agent, its semantics apply instead
• The Spring team does not provide support for the OTel agent in general, you should reach out to the OTel community

Spring projects could also consider echoing this by saying that the metrics/traces described in their reference docs only apply if Micrometer is used, not if a third party instrumentation agent is used.

[famaridon]

Hello,

It's currently quite challenging to understand how to configure Spring Boot with OpenTelemetry, as there are three different solutions available:

1. OpenTelemetry JavaAgent
2. OpenTelemetry Spring Boot Starter
3. Spring Boot Configuration (Micrometer OTel Bridge)

The first two solutions are documented within the OpenTelemetry community. However, the OpenTelemetry Spring Boot Starter can be particularly confusing.

The third solution, using Spring Boot's configuration with the Micrometer OTel Bridge, is currently not clearly explained. Additionally, Spring Boot currently does not support logging to OpenTelemetry, but this feature is expected to be available in version 3.4.0 (see issue #40961).

I believe it would be very useful for Spring Boot users who want to use (or switch to) OpenTelemetry to have a dedicated chapter explaining how to configure logs, traces, and metrics with Spring Boot.

The appropriate place for this documentation would likely be in the Production-ready Features section, where an OpenTelemetry chapter could be added.

[bclozel]

@famaridon it would be nice if you could elaborate on what's not clearly explained. Features like logs/traces/metrics are explained separately, I don't think we intend to have a specific variant of all of these for OpenTelemetry. This section and the following ones should apply.

However, the OpenTelemetry Spring Boot Starter can be particularly confusing.

As for solutions 1) and 2), do we agree that docs for those belong in OpenTelemetry itself?

[vpavic]

Not so long ago I opened #40340 asking for reference manual to mention OpenTelemetry's own Spring Boot starter and I still strongly believe that should be the case, because as OpenTelemetry Support section mentions, Spring Boot’s actuator module includes basic support for OpenTelemetry whereas OpenTelemetry's own support in many ways offers more and could be considered substantial.

IMO the point is to raise awareness of the existence of several competing solutions for setting up OpenTelemetry instrumentation of Spring Boot based applications by simply mentioning them and let the users chose whatever solution fits their projects the best. I know that I would've liked to have learned earlier about the OpenTelemetry's own Spring Boot starter.

Additionally, this issue is another example of presenting the choice as it's only OpenTelemetry Java agent vs Spring Boot's auto-config but that's simply not the case, and for many the 3rd option (that is, OpenTelemetry's own Spring Boot starter) is the true sweet spot.

[bclozel]

@vpavic this issue is another proof that this matter is really about opinion - there are already 3 very different opinions listed in this issue about the various existing solutions. Listing the OpenTelemetry starter in the list of 3rd party starters seems to be the safest choice.

We'll try and make things clearer in the first OpenTelemetry section I linked above. At least disambiguating existing solutions should help, but we shouldn't highlight differences or compare them.

[vpavic]

I fully agree the choice is opinion based. Again, I'm not asking for highlighting differences or including comparisons into the reference but rather to simply mention the existence of OpenTelemetry's own solutions in that section in order to make users aware of the choice.

[famaridon]

I agree with you.

Can we simply provide a sample in the logger section on how to configure the logger with OpenTelemetry, using the content from #40961?

[bclozel]

I added a small section mentioning the available solutions for OpenTelemetry support. I guess the reverse is not true, since the official OpenTelemetry page only mentions the agent and the OTel starter, but not the variant supported by the Spring team. This obviously leads to issues like open-telemetry/opentelemetry.io#5023

Now about the logging docs, I think #41825 is covering what @famaridon is looking for.