Kafka Streams tracing no longer working after upgrading to Spring Boot 3 + Micrometer Tracing

[j-tim]

Context

At the moment I'm migrating some Spring Boot 2.7.x, Spring for Apache Kafka applications to Spring Boot 3.

I followed the migration guide and Spring Kafka documentation regarding migrating from Spring Cloud Sleuth to Micrometer Tracing

• https://docs.spring.io/spring-kafka/reference/html/#observation-gen
• https://github.com/spring-cloud/spring-cloud-release/wiki/Spring-Cloud-2022.0-Release-Notes
• https://github.com/micrometer-metrics/tracing/wiki/Spring-Cloud-Sleuth-3.1-Migration-Guide

Distributed tracing works fine for:

• producing to Kafka (by enabling observation on the KafkaTemplate)
• and consuming from Kafka (by enabling observation via the container properties of the KafkaListenerContainerFactory)

But for Kafka streams application (using the KafkaStreamBrancher in my minimal, reproducible, sample) the tracing is no longer working after the upgrade.

In what version(s) of Spring for Apache Kafka are you seeing this issue?

Spring for Apache Kafka: 3.0.4

Migrating from:

• Spring Boot 2.7.9
  • Spring Cloud: 2021.0.6
  • Spring for Apache Kafka: 2.8.11
  • Spring Cloud Sleuth

to:

• Spring Boot 3.0.4
  • Spring Cloud: 2022.0.1
  • Spring for Apache Kafka: 3.0.4
  • Micrometer Tracing (Open Telemetry + Zipkin)

Describe the bug

For the Kafka streams application the tracing is no longer working after the upgrade.

Spring Boot 2.7.9, Spring Kafka and Spring Cloud Sleuth:

Spring Boot 3.0.4, Spring Kafka and Micrometer Tracing (OTEL)

Expected behavior

I expect the traces of my Kafka Streams application are reported in this case to Zipkin.

Sample to reproduce the issue

I put some effort in creating a GitHub repository with a minimal, reproducible, sample to reproduce the issue. The README of the sample project contains all the step to reproduce the issue.

Branches in the example repository:

• main: branch that contains the example to reproduce the issue with Spring Boot 3.0.4, Spring Kafka and Micrometer Tracing
• spring-boot-2.7-spring-cloud-sleuth: branch that contains a fully working example based on Spring Boot 2.7.9 and Spring Cloud Sleuth

Did I overlook something in the documentation to enable tracing for my Kafka Streams application? Or is tracing for Kafka Stream applications not supported yet using Spring for Apache Kafka and Micrometer Tracing?

[garyrussell]

Spring is not involved with instrumenting Kafka Streams at all - Spring is only used to set up the stream; there is no runtime involvement.

My (limited) understanding is that Sleuth uses brave to instrument the streams producer/consumer.

I don't know if it's practical for this project to replace the sleuth functionality for Streams - I'll have to defer that to the Micrometer team.

cc/ @marcingrzejszczak

[marcingrzejszczak]

If there's an option for Spring Kafka to instrument Kafka Streams by any means then that would be great (although I don't know if that's feasible?).

[garyrussell]

How does Sleuth do it?

[marcingrzejszczak]

As you said - it does it via Brave's components (https://github.com/spring-cloud/spring-cloud-sleuth/blob/c795675cf93c666c3c13f9bbb9eadbde1f1aa775/spring-cloud-sleuth-autoconfigure/src/main/java/org/springframework/cloud/sleuth/autoconfig/brave/instrument/messaging/BraveKafkaStreamsAutoConfiguration.java).

Here you can see how they exactly do it https://github.com/openzipkin/brave/tree/master/instrumentation/kafka-streams

[garyrussell]

I think all we can do here is explain how to set the KafkaClientSupplier on the StreamsBuilderFactoryBean, as is done by Sleuth here:

https://github.com/spring-cloud/spring-cloud-sleuth/blob/c795675cf93c666c3c13f9bbb9eadbde1f1aa775/spring-cloud-sleuth-autoconfigure/src/main/java/org/springframework/cloud/sleuth/autoconfig/brave/instrument/messaging/BraveKafkaStreamsAutoConfiguration.java#L106

[marcingrzejszczak]

Not really cause that's just delegating to what brave does. What if someone uses open telemetry or another tracer?

[garyrussell]

What does Sleuth do in those cases?

The only extension point that Kafka Streams provides is the client supplier; so each library needs to provide its own implementation to decorate the producer/consumer; there is nothing else that this project can do.

[marcingrzejszczak]

Sleuth only supports the dedicated Brave instrumentation.

[j-tim]

Thanks, Gary and Marcin for your responses.
I understand just delegating to brave is not the way to go.

Based on your suggestions I try to implement a custom Kafka client supplier.
To set the custom KafkaClientSupplier on the StreamsBuilderFactoryBean:

@Bean
public StreamsBuilderFactoryBeanCustomizer streamsBuilderFactoryBeanCustomizer() {
   return factoryBean -> factoryBean.setClientSupplier(new TracingKafkaClientSupplier());
}

The TracingKafkaClientSupplier extends the DefaultKafkaClientSupplier.

Can you give me some hints on how to implement this custom supplier?
Should this custom supplier use the ObservationRegistry of Micrometer? Or should I directly use the OpenTelemetry API
like in this example: https://github.com/ppatierno/kafka-opentelemetry/blob/main/kafka-streams-wrap/src/main/java/io/ppatierno/kafka/opentelemetry/Streams.java#L42

Would it not make more sense if Spring or Micrometer will provide a 'Tracing Kafka client supplier' that will be auto-configured or registered when observation is enabled like how it's done for the KafkaTemplate and KafkaListenerContainer?

[garyrussell]

Unfortunately, Kafka Streams is a black box; Spring has no way to instrument the underlying producers/consumers that are created. Nor does it have access to scope of a consume or produce operation.

So there is no way to provide the functionality that Spring can provide where we have control over the consumption and production of records.

The instrumentation would have to be done internally by the kafka-clients.

[artembilan]

See this one as an alternative solution: https://github.com/openzipkin/brave/blob/2ef59ea0e687d51e445bb5144c27e373fa73b7d7/instrumentation/kafka-clients/src/main/java/brave/kafka/clients/TracingProducer.java.

It feels like a dedicated library as a kafka-micrometer should emerge instead where an Observation wrapping and handling can be implemented down to that ObservationKafkaClientSupplier.
Something similar to what we have now for JDBC: https://jdbc-observations.github.io/datasource-micrometer/docs/current/docs/html/#getting-started-installation.

[marcingrzejszczak]

Ok I see so this has to be done on the kafka level. We already have metrics for Kafka (https://github.com/micrometer-metrics/micrometer/tree/main/micrometer-core/src/main/java/io/micrometer/core/instrument/binder/kafka) so we would need to add observability support, right?

[artembilan]

Sounds legit, @marcingrzejszczak .
👍

[marcingrzejszczak]

Added an issue in Micrometer - micrometer-metrics/micrometer#3713 I guess we can close this in favor of that one?

[artembilan]

I think we will keep this one as a documentation type to mention in the future how to instrument Kafka Streams with an observability when it is ready on a common library.