[Bug] Agent startup failure with JBoss EAP

[andrewazores]

2023-11-27 18:57:54:554 +0000 [cryostat-agent-main] ERROR io.cryostat.agent.Agent - Agent startup failure
java.lang.RuntimeException: org.openjdk.jmc.rjmx.ConnectionException caused by javax.management.JMRuntimeException: Failed to load MBeanServerBuilder class org.jboss.as.jmx.PluggableMBeanServerBuilder: java.lang.ClassNotFoundException: org.jboss.as.jmx.PluggableMBeanServerBuilder
	at io.cryostat.agent.MainModule.provideJvmId(MainModule.java:307)
	at io.cryostat.agent.MainModule_ProvideJvmIdFactory.provideJvmId(MainModule_ProvideJvmIdFactory.java:42)
	at io.cryostat.agent.MainModule_ProvideJvmIdFactory.get(MainModule_ProvideJvmIdFactory.java:34)
	at io.cryostat.agent.MainModule_ProvideJvmIdFactory.get(MainModule_ProvideJvmIdFactory.java:12)
	at dagger.internal.DoubleCheck.get(DoubleCheck.java:47)
	at io.cryostat.agent.MainModule_ProvideCryostatClientFactory.get(MainModule_ProvideCryostatClientFactory.java:66)
	at io.cryostat.agent.MainModule_ProvideCryostatClientFactory.get(MainModule_ProvideCryostatClientFactory.java:15)
	at dagger.internal.DoubleCheck.get(DoubleCheck.java:47)
	at io.cryostat.agent.MainModule_ProvideRegistrationFactory.get(MainModule_ProvideRegistrationFactory.java:74)
	at io.cryostat.agent.MainModule_ProvideRegistrationFactory.get(MainModule_ProvideRegistrationFactory.java:13)
	at dagger.internal.DoubleCheck.get(DoubleCheck.java:47)
	at io.cryostat.agent.DaggerAgent_Client$ClientImpl.registration(DaggerAgent_Client.java:257)
	at io.cryostat.agent.Agent.main(Agent.java:67)
	at io.cryostat.agent.Agent.lambda$agentmain$2(Agent.java:153)
	at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: org.openjdk.jmc.rjmx.ConnectionException caused by javax.management.JMRuntimeException: Failed to load MBeanServerBuilder class org.jboss.as.jmx.PluggableMBeanServerBuilder: java.lang.ClassNotFoundException: org.jboss.as.jmx.PluggableMBeanServerBuilder
	at org.openjdk.jmc.rjmx.internal.RJMXConnection.connect(RJMXConnection.java:345)
	at io.cryostat.agent.shaded.io.cryostat.core.net.JFRJMXConnection.attemptConnect(JFRJMXConnection.java:412)
	at io.cryostat.agent.shaded.io.cryostat.core.net.JFRJMXConnection.connect(JFRJMXConnection.java:368)
	at io.cryostat.agent.shaded.io.cryostat.core.net.JFRJMXConnection.getJvmId(JFRJMXConnection.java:171)
	at io.cryostat.agent.MainModule.provideJvmId(MainModule.java:302)
	... 14 more
Caused by: javax.management.JMRuntimeException: Failed to load MBeanServerBuilder class org.jboss.as.jmx.PluggableMBeanServerBuilder: java.lang.ClassNotFoundException: org.jboss.as.jmx.PluggableMBeanServerBuilder
	at java.management/javax.management.MBeanServerFactory.checkMBeanServerBuilder(MBeanServerFactory.java:502)
	at java.management/javax.management.MBeanServerFactory.getNewMBeanServerBuilder(MBeanServerFactory.java:535)
	at java.management/javax.management.MBeanServerFactory.newMBeanServer(MBeanServerFactory.java:316)
	at java.management/javax.management.MBeanServerFactory.newMBeanServer(MBeanServerFactory.java:273)
	at org.openjdk.jmc.rjmx.internal.SyntheticAttributeRepository.<init>(SyntheticAttributeRepository.java:78)
	at org.openjdk.jmc.rjmx.internal.MCMBeanServerConnection.<init>(MCMBeanServerConnection.java:80)
	at org.openjdk.jmc.rjmx.internal.RJMXConnection.connect(RJMXConnection.java:336)
	... 18 more
Caused by: java.lang.ClassNotFoundException: org.jboss.as.jmx.PluggableMBeanServerBuilder
	at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:581)
	at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
	at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:527)
	at java.management/javax.management.MBeanServerFactory.loadBuilderClass(MBeanServerFactory.java:444)
	at java.management/javax.management.MBeanServerFactory.checkMBeanServerBuilder(MBeanServerFactory.java:487)
	... 24 more

[ebaron]

Came across this VisualVM issue: https://stackoverflow.com/a/38455790. Adding -Djboss.platform.mbeanserver=true fixed it for us too.

[andrewazores]

Maybe we should set up a project wiki to track specific bugs and workarounds like that.

[ebaron]

Came across this VisualVM issue: https://stackoverflow.com/a/38455790. Adding -Djboss.platform.mbeanserver=true fixed it for us too.

I might have spoken too soon. I think I just missed the exception in the logs the first time.

[andrewazores]

I can do some deeper digging to see if it's feasible to replace all of the Agent's internal uses of various MBeans with direct accesses, but IIRC there was something that we're using from the FlightRecorderMXBean which the jdk.jfr package doesn't have in its API. At the very least this is likely to break the Smart Triggers functionality which explicitly depends on reading metrics from MBeans.

[FranciscoMeloJr]

Hello, this jira came to my attention today April 2nd by @leticia Konno.
Which EAP version is that specifically? I remember there was some discussion about agent usage and MODULES_OPTS, and for 7.4+ I'd try moving the agent to MODULE_OPTS so it doesn't start too early and see if that resolves it.
I explain this here: https://access.redhat.com/solutions/7021605

[andrewazores]

I can do some deeper digging to see if it's feasible to replace all of the Agent's internal uses of various MBeans with direct accesses, but IIRC there was something that we're using from the FlightRecorderMXBean which the jdk.jfr package doesn't have in its API. At the very least this is likely to break the Smart Triggers functionality which explicitly depends on reading metrics from MBeans.

This was fixed in #280 .

---

@FranciscoMeloJr I can't recall which EAP version we tested at the time I filed this bug report. @ebaron @sefroberg do you know? We had a lot of discussions back then about MODULES_OPTS and things as well and IIRC we did come up with some form of workaround.

[ebaron]

I think the problem we wre facing with EAP 7.4+ with MODULE_OPTS was due to jboss-modules trying to load jars from the manifest's Class-Path entry. That's been removed with #311, so I would expect this to work now, but I haven't tested it myself yet.