Adding custom ObservationHandler does not preserve the default ObservationHandlers

[violetagg]

Spring Boot version

3.0.4

Reproducible example

https://github.com/violetagg/netty-observation-repro

Problem description

I would like to use Reactor Netty tracing capabilities in Spring Boot application.
For this purpose I need to reuse the ObservationRegistry created by Spring Boot and to add ObservationHandlers provided by Reactor Netty.

Observations:

1. If I only reuse the ObservationRegistry without adding the ObservationHandlers provided by Reactor Netty - the tracing information is correctly visualised but the span tags are duplicated and one and the same Timer object is created every time.

2. If I reuse the ObservationRegistry and add ObservationHandlers provided by Reactor Netty - the tracing information is NOT correctly visualised.

3. If I reuse the ObservationRegistry, add ObservationHandlers provided by Reactor Netty and in addition add all default ObservationHandlers provided by Spring Boot - the tracing information is correctly visualised, the span tags are not duplicated and one and the same Timer object is created only once. Reactor Netty tracing capabilities are working as expected.

Desired solution

Spring Boot should preserve the default ObservationHandlers when custom ObservationHandlers are added to the configuration.

[violetagg]

/cc @tmarback @marcingrzejszczak

[wilkinsona]

See also #34399.

[violetagg]

yep, because of the mentioned issue, in the example I added @Order when declaring the various ObservationHandlers

[marcingrzejszczak]

I think that the default observation handlers should not have conditional on missing bean

[wilkinsona]

That doesn't feel very Boot-like to me. It would leave users who want to have the auto-configured handlers back off and replaced with their own with a problem. They'd have to exclude ObservationAutoConfiguration which would lose more than just the default observation handlers.

[marcingrzejszczak]

If you don't want the default tracing handler you register your own tracing handler before the default one because only one can be used. Same for metrics. We are grouping all handlers for exactly this reason - that you can put your own handlers before the existing ones.

Another option for @violetagg would be never to extend from the ones that we provide but delegate to them (that way the types would not be the same and @ConditionalOnMissingBean wouldn't have effect).

[violetagg]

If you don't want the default tracing handler you register your own tracing handler before the default one because only one can be used. Same for metrics. We are grouping all handlers for exactly this reason - that you can put your own handlers before the existing ones.

Yes we do exactly that our tracing handlers are implemented in a way that io.micrometer.tracing.handler.TracingObservationHandler#supportsContext handles only our Reactor Netty implementation, everything else should be handled by the default configuration in Spring Boot.

The scenario here is that we do want the default Spring Boot tracing for the server. What we want to enable is the tracing for Reactor Netty HttpClient only.

Another option for @violetagg would be never to extend from the ones that we provide but delegate to them (that way the types would not be the same and @ConditionalOnMissingBean wouldn't have effect).

[marcingrzejszczak]

The scenario here is that we do want the default Spring Boot tracing for the server. What we want to enable is the tracing for Reactor Netty HttpClient only.

Yeah so all of your custom handlers wouldn't extend from the default ones, they would delegate to them. So you would never hit the @ConditionalOnMissingBean problem

[violetagg]

The scenario here is that we do want the default Spring Boot tracing for the server. What we want to enable is the tracing for Reactor Netty HttpClient only.

Yeah so all of your custom handlers wouldn't extend from the default ones, they would delegate to them. So you would never hit the @ConditionalOnMissingBean problem

So we have to delegate to the default in both cases when in Spring Boot application and when in standalone Reactor Netty.
For Spring Boot we will delegate to the handlers created by Spring Boot.
In the standalone case to what should we delegate?

[marcingrzejszczak]

So if you have 3 custom types that extend default, sending and receiving then you should in all three cases not extend but delegate to these 3. That way in Boot you will have 6 handlers registered but yours should be first (in order - your sending or receiving then your default and then the boot ones). In the case of standalone you need to do the same:

new FirstMatchingComposite...(

• your sending
• your receiving
• your default
• micrometer sending
• micrometer receiving
• micrometer default
  )

[violetagg]

@marcingrzejszczak and I tried the variant with delegation. (reactor/reactor-netty@e02e722)
Although technically this is possible it requires 3 TracingObservationHandlers which do not have functional logic but just delegate and this should be a way of implementing custom TracingObservationHandler and registering it as a Bean for every library not just Reactor Netty.

On the other hand if default TracingObservationHandlers are not conditional and we rely on the io.micrometer.tracing.handler.TracingObservationHandler#supportsContext then if the custom TracingObservationHandler supports this context it will be the one that will be chosen otherwise the default TracingObservationHandlers. (as FirstMatchingCompositeObservationHandler is used when configuring the ObservationRegistry)

Basically the variant with delegation is a workaround to have the default TracingObservationHandlers in place.