Upcoming changes to JMX metrics gatherer

[breedx-splk]

Component(s)

receiver/jmx

Describe the issue you're reporting

Hi collector folks! Greetings from the Java instrumentation group.

We have a java contrib component commonly called the JMX Metrics Gatherer which allows users to choose from a set of preconfigured out-of-the-box frameworks and/or provide custom configurations for their own JMX MBeans. This component runs as a stand-alone jar file, connects to a JVM, and exports metrics. We believe it is bundled or otherwise launched from the jmxreceiver in this repository.

For some time, we (otel java sig) have had a confusing and limiting situation which involves two components with similar goals but notably different implementations and configuration: jmx auto-instrumentation via the agent, and the above-mentioned jmx-metrics gatherer.

We want to begin unifying these separate implementations into something more cohesive. Furthermore, the jmx-metrics gatherer is written in and configured with groovy, and there is little interest in maintaining that going forward. So, the approach we discussed today in the Java SIG meeting today is this -- we will create a new module in java-contrib (name is tbd) that will essentially be a new version of the jmx-metrics gatherer. This will be written in Java and will use the same yaml based configuration as the java agent instrumentation. Once that module is published and we think it is ready to be used widely, we will remove the existing jmx-metrics module and stop publishing updates to it.

So this issue is really to:

1. let collector contrib maintainers/contributors know that this change is coming (timeline tbd)
2. solicit feedback and provoke discussion about the approach described above
3. learn more from collector folks about how widely used the jmx receiver is and any other existing challenges with it.

Thanks!

[github-actions]

Pinging code owners:

• receiver/jmx: @rmfitzpatrick

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[SylvainJuge]

For example for the items 2) and 3), we already know the following:

The current JMX gatherer (contrib) implementation also includes some pre-defined sets of metrics which are selected through the otel.jmx.target.system (doc)

One of the goals is being able to re-implement the current set of metrics with the YAML configuration instead of the groovy definitions, so ideally it could be a drop-in replacement for people relying on it.

In practice though, end-users are also allowed to provide their own groovy metrics definitions with otel.jmx.groovy.script config option, for this use-case a migration from groovy to yaml will be necessary. This "breaking change" is the main reason that makes us believe creating a new component is better than changing the current implementation.

In other terms, for those pre-defined metrics, do we have any idea if using the otel.jmx.groovy.script is frequent or not ?

[BinaryFissionGames]

From a cursory glance at the code, I don't think otel.jmx.groovy.script is configurable through the receiver. The jmx gatherer config is completely generated by the receiver in this function, and I don't see any reference to it.

opentelemetry-collector-contrib/receiver/jmxreceiver/receiver.go

Line 171 in 2d5b62d

func (jmx *jmxMetricReceiver) buildJMXMetricGathererConfig() (string, error) {

So if otel.jmx.target.system is still supported in some form, it seems like it might not necessarily be a breaking change?

[hughesjj]

From a cursory glance at the code, I don't think otel.jmx.groovy.script is configurable through the receiver.

It's not ... it was actually specifically taken out due to security concerns (it basically allows the collector to run an arbitrary groovy script -> essentially arbitrary code execution)

This proposed code change would allow customers of the collector to change what's queried, an in general be a huge usability increase

[breedx-splk]

So if otel.jmx.target.system is still supported in some form, it seems like it might not necessarily be a breaking change?

Not directly related to the collector or the receiver, but we have to still assume that existing users are launching the jar file through some other mechanism, in which case it's still "breaking" to them. Just sayin'.