OpenTelemetry and smallrye reactive kafka #39449

itatdcer · 2024-03-14T19:26:32Z

Describe the bug

I'm getting an error at startup .

I've already tried without success:

mp.messaging.incoming.data.tracing-enabled=false

Same for outgoing topic. The strangest thing about this is that I haven't been able to reproduce it locally (even in prod mode) and it only happens on a prod environment.

ERROR [io.qua.run.Application] (main) Failed to start application (with profile [prod]): java.lang.RuntimeException: Failed to start quarkus
	at io.quarkus.runner.ApplicationImpl.doStart(Unknown Source)
	at io.quarkus.runtime.Application.start(Application.java:101)
	at io.quarkus.runtime.ApplicationLifecycleManager.run(ApplicationLifecycleManager.java:111)
	at io.quarkus.runtime.Quarkus.run(Quarkus.java:71)
	at io.quarkus.runtime.Quarkus.run(Quarkus.java:44)
	at io.quarkus.runtime.Quarkus.run(Quarkus.java:124)
	at io.quarkus.runner.GeneratedMain.main(Unknown Source)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
	at io.quarkus.bootstrap.runner.QuarkusEntryPoint.doRun(QuarkusEntryPoint.java:62)
	at io.quarkus.bootstrap.runner.QuarkusEntryPoint.main(QuarkusEntryPoint.java:33)
Caused by: jakarta.enterprise.inject.CreationException: Error creating synthetic bean [Le6zQbzkojAYO_OiKIQWJf4lGa4]: io.opentelemetry.sdk.autoconfigure.spi.ConfigurationException: Unexpected configuration error
	at io.opentelemetry.api.OpenTelemetry_Le6zQbzkojAYO_OiKIQWJf4lGa4_Synthetic_Bean.doCreate(Unknown Source)
	at io.opentelemetry.api.OpenTelemetry_Le6zQbzkojAYO_OiKIQWJf4lGa4_Synthetic_Bean.create(Unknown Source)
	at io.opentelemetry.api.OpenTelemetry_Le6zQbzkojAYO_OiKIQWJf4lGa4_Synthetic_Bean.create(Unknown Source)
	at io.quarkus.arc.impl.AbstractSharedContext.createInstanceHandle(AbstractSharedContext.java:119)
	at io.quarkus.arc.impl.AbstractSharedContext$1.get(AbstractSharedContext.java:38)
	at io.quarkus.arc.impl.AbstractSharedContext$1.get(AbstractSharedContext.java:35)
	at io.quarkus.arc.impl.LazyValue.get(LazyValue.java:32)
	at io.quarkus.arc.impl.ComputingCache.computeIfAbsent(ComputingCache.java:69)
	at io.quarkus.arc.impl.ComputingCacheContextInstances.computeIfAbsent(ComputingCacheContextInstances.java:19)
	at io.quarkus.arc.impl.AbstractSharedContext.get(AbstractSharedContext.java:35)
	at io.opentelemetry.api.OpenTelemetry_Le6zQbzkojAYO_OiKIQWJf4lGa4_Synthetic_Bean.get(Unknown Source)
	at io.opentelemetry.api.OpenTelemetry_Le6zQbzkojAYO_OiKIQWJf4lGa4_Synthetic_Bean.get(Unknown Source)
	at io.quarkus.arc.impl.InstanceImpl.getBeanInstance(InstanceImpl.java:325)
	at io.quarkus.arc.impl.InstanceImpl.getInternal(InstanceImpl.java:309)
	at io.quarkus.arc.impl.InstanceImpl.get(InstanceImpl.java:190)
	at io.quarkus.arc.runtime.BeanContainerImpl.beanInstance(BeanContainerImpl.java:26)
	at io.quarkus.opentelemetry.runtime.tracing.intrumentation.InstrumentationRecorder.setupVertxTracer(InstrumentationRecorder.java:43)
	at io.quarkus.deployment.steps.OpenTelemetryProcessor$setupVertx126580776.deploy_0(Unknown Source)
	at io.quarkus.deployment.steps.OpenTelemetryProcessor$setupVertx126580776.deploy(Unknown Source)
	... 13 more
Caused by: io.opentelemetry.sdk.autoconfigure.spi.ConfigurationException: Unexpected configuration error
	at io.opentelemetry.sdk.autoconfigure.AutoConfiguredOpenTelemetrySdkBuilder.build(AutoConfiguredOpenTelemetrySdkBuilder.java:427)
	at io.quarkus.opentelemetry.runtime.OpenTelemetryRecorder$1.apply(OpenTelemetryRecorder.java:65)
	at io.quarkus.opentelemetry.runtime.OpenTelemetryRecorder$1.apply(OpenTelemetryRecorder.java:51)
	at io.opentelemetry.api.OpenTelemetry_Le6zQbzkojAYO_OiKIQWJf4lGa4_Synthetic_Bean.createSynthetic(Unknown Source)
	... 32 more
Caused by: java.lang.IllegalStateException: GlobalOpenTelemetry.set has already been called. GlobalOpenTelemetry.set must be called only once before any calls to GlobalOpenTelemetry.get. If you are using the OpenTelemetrySdk, use OpenTelemetrySdkBuilder.buildAndRegisterGlobal instead. Previous invocation set to cause of this exception.
	at io.opentelemetry.api.GlobalOpenTelemetry.set(GlobalOpenTelemetry.java:107)
	at io.opentelemetry.sdk.autoconfigure.AutoConfiguredOpenTelemetrySdkBuilder.maybeSetAsGlobal(AutoConfiguredOpenTelemetrySdkBuilder.java:476)
	at io.opentelemetry.sdk.autoconfigure.AutoConfiguredOpenTelemetrySdkBuilder.build(AutoConfiguredOpenTelemetrySdkBuilder.java:408)
	... 35 more
Caused by: java.lang.Throwable
	at io.opentelemetry.api.GlobalOpenTelemetry.set(GlobalOpenTelemetry.java:115)
	at io.opentelemetry.api.GlobalOpenTelemetry.get(GlobalOpenTelemetry.java:85)
	at io.smallrye.reactive.messaging.kafka.tracing.KafkaOpenTelemetryInstrumenter.create(KafkaOpenTelemetryInstrumenter.java:41)
	at io.smallrye.reactive.messaging.kafka.tracing.KafkaOpenTelemetryInstrumenter.createForSource(KafkaOpenTelemetryInstrumenter.java:27)
	at io.smallrye.reactive.messaging.kafka.impl.KafkaSource.<init>(KafkaSource.java:230)
	at io.smallrye.reactive.messaging.kafka.KafkaConnector.getPublisher(KafkaConnector.java:234)
	at io.smallrye.reactive.messaging.kafka.KafkaConnector_Subclass.getPublisher$$superforward(Unknown Source)
	at io.smallrye.reactive.messaging.kafka.KafkaConnector_Subclass$$function$$9.apply(Unknown Source)
	at io.quarkus.arc.impl.AroundInvokeInvocationContext.proceed(AroundInvokeInvocationContext.java:73)
	at io.quarkus.arc.impl.AroundInvokeInvocationContext.proceed(AroundInvokeInvocationContext.java:62)
	at io.quarkus.smallrye.reactivemessaging.runtime.DuplicatedContextConnectorFactoryInterceptor.intercept(DuplicatedContextConnectorFactoryInterceptor.java:32)
	at io.quarkus.smallrye.reactivemessaging.runtime.DuplicatedContextConnectorFactoryInterceptor_Bean.intercept(Unknown Source)
	at io.quarkus.arc.impl.InterceptorInvocation.invoke(InterceptorInvocation.java:42)
	at io.quarkus.arc.impl.AroundInvokeInvocationContext.perform(AroundInvokeInvocationContext.java:30)
	at io.quarkus.arc.impl.InvocationContexts.performAroundInvoke(InvocationContexts.java:27)
	at io.smallrye.reactive.messaging.kafka.KafkaConnector_Subclass.getPublisher(Unknown Source)
	at io.smallrye.reactive.messaging.kafka.KafkaConnector_ClientProxy.getPublisher(Unknown Source)
	at io.smallrye.reactive.messaging.providers.impl.ConfiguredChannelFactory.createPublisher(ConfiguredChannelFactory.java:188)
	at io.smallrye.reactive.messaging.providers.impl.ConfiguredChannelFactory.register(ConfiguredChannelFactory.java:142)
	at io.smallrye.reactive.messaging.providers.impl.ConfiguredChannelFactory.initialize(ConfiguredChannelFactory.java:112)
	at io.smallrye.reactive.messaging.providers.impl.ConfiguredChannelFactory_ClientProxy.initialize(Unknown Source)
	at java.base/java.util.Iterator.forEachRemaining(Iterator.java:133)
	at java.base/java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1845)
	at java.base/java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:762)
	at io.smallrye.reactive.messaging.providers.extension.MediatorManager.start(MediatorManager.java:250)
	at io.smallrye.reactive.messaging.providers.extension.MediatorManager_ClientProxy.start(Unknown Source)
	at io.quarkus.smallrye.reactivemessaging.runtime.SmallRyeReactiveMessagingLifecycle.onApplicationStart(SmallRyeReactiveMessagingLifecycle.java:53)
	at io.quarkus.smallrye.reactivemessaging.runtime.SmallRyeReactiveMessagingLifecycle_Observer_onApplicationStart_qTrMuLFyQ1IvGfeSxRVitl6CCBQ.notify(Unknown Source)
	at io.quarkus.arc.impl.EventImpl$Notifier.notifyObservers(EventImpl.java:346)
	at io.quarkus.arc.impl.EventImpl$Notifier.notify(EventImpl.java:328)
	at io.quarkus.arc.impl.EventImpl.fire(EventImpl.java:82)
	at io.quarkus.arc.runtime.ArcRecorder.fireLifecycleEvent(ArcRecorder.java:155)
	at io.quarkus.arc.runtime.ArcRecorder.handleLifecycleEvents(ArcRecorder.java:106)
	at io.quarkus.deployment.steps.LifecycleEventsBuildStep$startupEvent1144526294.deploy_0(Unknown Source)
	at io.quarkus.deployment.steps.LifecycleEventsBuildStep$startupEvent1144526294.deploy(Unknown Source)
	... 13 more

Expected behavior

No error at startup

Actual behavior

No response

How to Reproduce?

No response

Output of `uname -a` or `ver`

No response

Output of `java -version`

17

Quarkus version or git rev

3.8.2

Build tool (ie. output of `mvnw --version` or `gradlew --version`)

No response

Additional information

No response

The text was updated successfully, but these errors were encountered:

quarkus-bot · 2024-03-14T19:26:36Z

/cc @Ladicek (smallrye), @alesj (kafka), @brunobat (opentelemetry), @cescoffier (kafka), @jmartisk (smallrye), @ozangunalp (kafka), @phillip-kruger (smallrye), @radcortez (opentelemetry,smallrye)

brunobat · 2024-03-14T21:16:31Z

This bug should be filed in the smallrye reactive messaging repo: https://github.com/smallrye/smallrye-reactive-messaging
For some weird reason, in your prod env, the static init is kicking earlier here:
https://github.com/smallrye/smallrye-reactive-messaging/blob/570f8e802500b662b47f48e6cae53f0c5dc13422/smallrye-reactive-messaging-kafka/src/main/java/io/smallrye/reactive/messaging/kafka/tracing/KafkaOpenTelemetryInstrumenter.java#L41

We should not static init the instrumenters, the OpenTelemetry object should be injected because it's created here:

quarkus/extensions/opentelemetry/deployment/src/main/java/io/quarkus/opentelemetry/deployment/OpenTelemetryProcessor.java

Line 115 in 079d8d0

    
           SyntheticBeanBuildItem openTelemetryBean(OpenTelemetryRecorder recorder, OTelRuntimeConfig oTelRuntimeConfig) {

CC @ozangunalp

ozangunalp · 2024-03-15T07:50:08Z

There is also the issue with tracing-enabled=false config not working for you. When not enabled that instrumenter isn't created at all.

itatdcer · 2024-03-15T07:57:25Z

This bug should be filed in the smallrye reactive messaging repo: https://github.com/smallrye/smallrye-reactive-messaging For some weird reason, in your prod env, the static init is kicking earlier here: https://github.com/smallrye/smallrye-reactive-messaging/blob/570f8e802500b662b47f48e6cae53f0c5dc13422/smallrye-reactive-messaging-kafka/src/main/java/io/smallrye/reactive/messaging/kafka/tracing/KafkaOpenTelemetryInstrumenter.java#L41

We should not static init the instrumenters, the OpenTelemetry object should be injected because it's created here:

quarkus/extensions/opentelemetry/deployment/src/main/java/io/quarkus/opentelemetry/deployment/OpenTelemetryProcessor.java

Line 115 in 079d8d0

SyntheticBeanBuildItem openTelemetryBean(OpenTelemetryRecorder recorder, OTelRuntimeConfig oTelRuntimeConfig) {

CC @ozangunalp

Should I open the same issue there or is it something that the quarkus team will address?

ozangunalp · 2024-03-15T08:34:12Z

@itatdcer I am creating the issue upstream but I'd like to keep this one for tracking purposes.

I remember discussing this back then with @kabir, I think we still need a failover for creating the global OpenTelemetry instance because Reactive Messaging may run in an environment where it is not available as a CDI bean.

brunobat · 2024-03-15T09:59:08Z

@itatdcer I am creating the issue upstream but I'd like to keep this one for tracking purposes.

I remember discussing this back then with @kabir, I think we still need a failover for creating the global OpenTelemetry instance because Reactive Messaging may run in an environment where it is not available as a CDI bean.

Makes sense.

brunobat · 2024-03-15T10:01:42Z

Not sure if I understood correctly, @itatdcer... You say that when you set:
mp.messaging.incoming.data.tracing-enabled=false
The instrumenter is still instantiated and throws that error?... I would expect that the instrumenter would not be instantiated in that case.

itatdcer · 2024-03-15T11:12:50Z

Not sure if I understood correctly, @itatdcer... You say that when you set: mp.messaging.incoming.data.tracing-enabled=false The instrumenter is still instantiated and throws that error?... I would expect that the instrumenter would not be instantiated in that case.

That's correct. Assuming that the configuration is a runtime one (I can't confirm it on quarkus documentation), I've tried that without success.

For now, I've changed the build time configuration:
quarkus.otel.enabled=false
and bypassed the error.

edeweerd1A · 2024-04-25T14:23:19Z

It would be nice to have this fix back ported to Quarkus 3.8 LTS. I know it should be done upstream, but I was wondering.

cescoffier · 2024-04-28T11:49:45Z

@brunobat @ozangunalp Do you think it can be backported to 3.8?

ozangunalp · 2024-04-28T12:15:06Z

I'll check that, it needs a patch release to reactive messaging.

ozangunalp · 2024-09-02T12:16:57Z

Backported to 3.8 #42158

itatdcer added the kind/bug Something isn't working label Mar 14, 2024

quarkus-bot bot added area/kafka area/smallrye area/tracing labels Mar 14, 2024

cescoffier assigned ozangunalp Mar 15, 2024

ozangunalp mentioned this issue Mar 15, 2024

GlobalOpenTelemetry instance recreated in Reactive Messaging channels smallrye/smallrye-reactive-messaging#2539

Closed

ozangunalp closed this as completed Sep 2, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

OpenTelemetry and smallrye reactive kafka #39449

OpenTelemetry and smallrye reactive kafka #39449

itatdcer commented Mar 14, 2024 •

edited by brunobat

Loading

quarkus-bot bot commented Mar 14, 2024

brunobat commented Mar 14, 2024 •

edited

Loading

ozangunalp commented Mar 15, 2024

itatdcer commented Mar 15, 2024

ozangunalp commented Mar 15, 2024

brunobat commented Mar 15, 2024

brunobat commented Mar 15, 2024

itatdcer commented Mar 15, 2024

edeweerd1A commented Apr 25, 2024

cescoffier commented Apr 28, 2024

ozangunalp commented Apr 28, 2024

ozangunalp commented Sep 2, 2024

OpenTelemetry and smallrye reactive kafka #39449

OpenTelemetry and smallrye reactive kafka #39449

Comments

itatdcer commented Mar 14, 2024 • edited by brunobat Loading

Describe the bug

Expected behavior

Actual behavior

How to Reproduce?

Output of uname -a or ver

Output of java -version

Quarkus version or git rev

Build tool (ie. output of mvnw --version or gradlew --version)

Additional information

quarkus-bot bot commented Mar 14, 2024

brunobat commented Mar 14, 2024 • edited Loading

ozangunalp commented Mar 15, 2024

itatdcer commented Mar 15, 2024

ozangunalp commented Mar 15, 2024

brunobat commented Mar 15, 2024

brunobat commented Mar 15, 2024

itatdcer commented Mar 15, 2024

edeweerd1A commented Apr 25, 2024

cescoffier commented Apr 28, 2024

ozangunalp commented Apr 28, 2024

ozangunalp commented Sep 2, 2024

itatdcer commented Mar 14, 2024 •

edited by brunobat

Loading

Output of `uname -a` or `ver`

Output of `java -version`

Build tool (ie. output of `mvnw --version` or `gradlew --version`)

brunobat commented Mar 14, 2024 •

edited

Loading