Duplicated logs with Elastic APM Agent - Quarkus wrongly importing log4j2 configuration from agent

[alejavigo]

Describe the bug

Using any APM agent starting in version 1.30.0 will make Quarkus to duplicate all its logs. Opened a jira in Elastic and they found the agent modification that started the behaviour but explain that it's Quarkus responsability to not add any log config from agent to main app:
elastic/apm-agent-java#3563

My hypothesis of what's happening here is that quarkus wrongly picks up the log4j2 configuration used by the agent internally. The agent runs in its own classloader with it's own isolated copy of log4j2 and an isolated configuration file.
Quarkus seems to somehow use the thread context-classloaders to pick up logging configurations. It even does so wrongly for elastic apm agent internal threads, which should be none of its business.

The problem doesn't occur anymore if we don't set the thread context classloaders (you can test with elastic/apm-agent-java#3575). However, this is nothing we will change in our agent because we set the context classloader for a reason.

Instead I'd recommend you to open an issue at the Quarkus repo: Quarkus shouldn't care about context-classloaders and their log4j2-configs for javaagents.

Maybe related to:
#7640
#2152

Expected behavior

Normal both quarkus app and agent logs

Actual behavior

All app logs get duplicated, second one with different format.

How to Reproduce?

Explained here:
elastic/apm-agent-java#3563

Output of uname -a or ver

No response

Output of java -version

No response

Quarkus version or git rev

No response

Build tool (ie. output of mvnw --version or gradlew --version)

No response

Additional information

No response

[quarkus-bot]

/cc @dmlloyd (logging)

[dmlloyd]

Quarkus does not use log4j2 for logging. I think the most likely possibility is that the log4j2 in the APM agent is configured, installing a JUL Handler as a part of its setup, causing the Quarkus log manager (based on JUL) to log once via JUL and to unknowingly delegate to log4j again via the injected handler (I don't see how TCCL would be involved in this though).

I'll continue investigating this but I think that this is likely a natural consequence of having two logging systems running in the same JVM. We might be able to provide a workaround of some kind but not a fix, if this is indeed caused by log4j2 injecting a redirection handler.

[dmlloyd]

To add on to this, it could be a good idea to introduce an implementation of the APM agent's logger which uses System.Logger.

[dmlloyd]

It looks like my initial evaluation is incorrect; there is actually a bug in the log manager itself.

I've opened LOGMGR-350 to track it.

[dmlloyd]

jboss-logging/jboss-logmanager#469