1.13 Migration Guide

This guide helps users upgrading from Micrometer 1.12.x to Micrometer 1.13.x by explaining changes users should be aware of and potentially take action on. See also the release page for included changes.

Prometheus Java client 0.x to 1.x upgrade

The micrometer-registry-prometheus module has upgraded to the Prometheus Java client 1.x version, which required some breaking changes from the prior 0.x version. The breaking changes may not affect all users depending on whether you are directly interacting with the API in PrometheusMeterRegistry or a framework is handling that configuration for you. As a fallback to ease upgrading, the previous code based on the Prometheus Java client 0.x is available in a new module named micrometer-registry-prometheus-simpleclient (deprecated). The plan is to eventually remove support for the micrometer-registry-prometheus-simpleclient so it should only be used as an interim solution until you can adapt to the necessary changes in micrometer-registry-prometheus related to the upgrade to Prometheus Java client 1.x.

If you want to migrate to the Prometheus Java client 1.x with Micrometer 1.13.0:

- runtimeOnly 'io.micrometer:micrometer-registry-prometheus:1.12.6'
+ runtimeOnly 'io.micrometer:micrometer-registry-prometheus:1.13.0'

If you want to stay on the Prometheus Java client 0.x but you want to migrate to Micrometer 1.13.0:

- runtimeOnly 'io.micrometer:micrometer-registry-prometheus:1.12.6'
+ runtimeOnly 'io.micrometer:micrometer-registry-prometheus-simpleclient:1.13.0'

Framework compatibility

Frameworks that integrate with Micrometer and specifically the Micrometer Prometheus registry will need to update their support to adapt to the changes in micrometer-registry-prometheus for the Prometheus Java client 1.x upgrade.

Spring Boot

Spring Boot offers dependency management as a feature, and best compatibility with integrations is ensured when you let Spring Boot manage the version of dependencies rather than override them (especially to new minor/major versions), including Micrometer. For example, Spring Boot's auto-configuration for the Prometheus registry will not work if you override the managed version of Micrometer to 1.13.0 when using Spring Boot 3.2 with micrometer-registry-prometheus (note that Spring Boot 3.2.x manages the version of Micrometer to 1.12.x), but it will work if you switch to the module for the Prometheus Java client 0.x (micrometer-registry-prometheus-simpleclient). Alternatively, upgrading to Spring Boot 3.3.x will result in the Micrometer version being managed to 1.13.x and auto-configuration working with either micrometer-registry-prometheus or micrometer-registry-prometheus-simpleclient (or both).

Package change

The micrometer-registry-prometheus module now uses the base package io.micrometer.prometheusmetrics rather than the previous io.micrometer.prometheus. If you are using any of the classes in the module in your code, you will need to update the import of the class.

For example

import io.micrometer.prometheus.PrometheusMeterRegistry;

should be replaced to

import io.micrometer.prometheusmetrics.PrometheusMeterRegistry;

and so on.

API changes

PrometheusMeterRegistry

Some types exposed in the micrometer-registry-prometheus module's API were from the Prometheus Java client and they have changed. Accordingly, the API in PrometheusMeterRegistry has changed. Specifically, where a CollectorRegistry was used before, now a PrometheusRegistry is. The constructor that took an io.prometheus.client.exemplars.ExemplarSampler for exemplar support now takes a io.prometheus.metrics.tracer.common.SpanContext instead.

For example creating a PrometheusMeterRegistry using the Prometheus Java client 0.x can look like this:

import io.micrometer.prometheus.*;

PrometheusMeterRegistry prometheusMeterRegistry = new PrometheusMeterRegistry(
            PrometheusConfig.DEFAULT,
            new CollectorRegistry(),
            Clock.SYSTEM,
            new DefaultExemplarSampler(new DemoSpanContextSupplier())
);

the same with the Prometheus Java client 1.x can look like this:

import io.micrometer.prometheusmetrics.*;

PrometheusMeterRegistry prometheusMeterRegistry = new PrometheusMeterRegistry(
            PrometheusConfig.DEFAULT,
            new PrometheusRegistry(),
            Clock.SYSTEM,
            new DemoSpanContext()
);

HistogramFlavor

HistogramFlavor as well as "native" support for VictoriaMetrics were removed since Prometheus Java client 1.x prevents changing the histogram bucket label (le vs. vmrange). On the other hand, according to the VictoriaMetrics docs, VictoriaMetrics can be used as drop-in replacement for Prometheus and should be able to scrape targets that Prometheus can.

Prometheus Config Support

Micrometer supports the new config mechanism of Prometheus Java client 1.x. You can read more about this in our docs. From migration perspective this can be an important detail because of the constructor of PrometheusMeterRegistry that took an io.prometheus.client.exemplars.ExemplarSampler for exemplar support now takes a io.prometheus.metrics.tracer.common.SpanContext instead which can prevent you customizing certain behaviors, like disabling Exemplars for _count. With the Prometheus Config support, you can modify the exemplar- and exporter properties of the Prometheus Java client 1.x, for example, you can disable exemplars on _count:

PrometheusConfig config = new PrometheusConfig() {
    @Override
    public String get(String key) {
        return null;
    }

    @Override
    public Properties prometheusProperties() {
        Properties properties = new Properties();
        properties.setProperty("io.prometheus.exporter.exemplarsOnAllMetricTypes", "false");
        return properties;
    }
};

PrometheusMeterRegistry registry = new PrometheusMeterRegistry(config, new PrometheusRegistry(), Clock.SYSTEM, spanContext);

Scrape output changes

Because of behavioral differences and extra validation in Prometheus Java client 1.x (micrometer-registry-prometheus), there are changes in the scrape output compared to Prometheus Java client 0.x (micrometer-registry-prometheus-simpleclient).

LongTaskTimer

LongTaskTimer is now registered as a GaugeHistogram type metric with the Prometheus registry rather than a Histogram/Summary type. The _count and _sum time series are now _gcount and _gsum and they will no longer have _active and _duration respectively appended to the base metric name. See the following sample.

Old Client (0.x) (micrometer-registry-prometheus-simpleclient)

# HELP spring_security_authorizations_active_seconds
# TYPE spring_security_authorizations_active_seconds summary
spring_security_authorizations_active_seconds_active_count{application="my-app",spring_security_authentication_type="n/a",spring_security_authorization_decision="unknown",spring_security_object="request",} 0.0
spring_security_authorizations_active_seconds_duration_sum{application="my-app",spring_security_authentication_type="n/a",spring_security_authorization_decision="unknown",spring_security_object="request",} 0.0

New Client (1.x) (micrometer-registry-prometheus)

# HELP spring_security_authorizations_active_seconds
# TYPE spring_security_authorizations_active_seconds gaugehistogram
spring_security_authorizations_active_seconds_gcount{application="my-app",spring_security_authentication_type="n/a",spring_security_authorization_decision="unknown",spring_security_object="request"} 0
spring_security_authorizations_active_seconds_gsum{application="my-app",spring_security_authentication_type="n/a",spring_security_authorization_decision="unknown",spring_security_object="request"} 0.0

Info vs. Gauge type

Gauge (Micrometer) suffixed with .info is now registered as Prometheus Info type metric rather than Prometheus Gauge type. This does not effect the name of the time series (what you query) but the type and the names in # TYPE and # HELP are different now. See the following sample.

Old Client (0.x) (micrometer-registry-prometheus-simpleclient)

# TYPE jvm_info gauge
# HELP jvm_info JVM version info
jvm_info{runtime="OpenJDK Runtime Environment",vendor="BellSoft",version="17.0.8.1+1-LTS"} 1.0

New Client (1.x) (micrometer-registry-prometheus)

# TYPE jvm info
# HELP jvm JVM version info
jvm_info{runtime="OpenJDK Runtime Environment",vendor="BellSoft",version="17.0.8.1+1-LTS"} 1

As you can see, the time-series (jvm_info{...}) did not change but the type (gauge -> info) and the names (jvm_info -> jvm) did.

Invalid Meter suffixes

Name suffixes are validated by Prometheus Java client 1.x, the following suffixes are now invalid and will be removed from the Meter names by PrometheusMeterRegistry: .info, .total, .created, .bucket (or _info, _total, _created, _bucket). These suffixes are used by Prometheus Java client 1.x and in certain situations they will be appended to the name. This can have some interesting consequences:

• ✅ If you create a Micrometer Gauge named test.info, the name of the Prometheus time series will be test_info since the registry removes but the Prometheus client adds back the _info suffix (see the previous section about Info vs. Gauge type).
• ✅ If you create a Micrometer Counter named test.total, the name of the Prometheus time series will be test_total since the registry removes but the Prometheus client adds back the _total suffix.
• ⚠️ If you create a Micrometer Counter named test.info, the name of the Prometheus time series will be test_total since the registry removes the _info suffix and the Prometheus client adds the _total suffix for counters.
• ⚠️ If you create a Micrometer Gauge named test.total, the name of the Prometheus gauge will be test since the registry removes the _total suffix and the Prometheus client does not add _info suffix for gauges.
• ⚠️ Similarly, _created and _bucket suffixes will be removed, otherwise the the Prometheus client will fail.

As you can see from the above, using .total, .created, and .bucket suffixes should be avoided to avoid unexpected outputs. A valid scenario could be using the .info suffix with Gauge (and only with Gauge) if you want the Prometheus type to be info instead of gauge (see the previous section about info vs. gauge type). For example this sample: registry.counter("test.info").increment(); produces the following output using the new Prometheus Java client (1.x) (micrometer-registry-prometheus):

# TYPE test counter
# HELP test
test_total 1.0

Please notice that the name of the registered Counter was test.info while the name of the time series is test_total.

Single character names are no longer valid

Names are validated by Prometheus Java client 1.x and using single character Meter names (e.g.: registry.counter("c").increment();) will result in an exception on Prometheus Java client side:

Exception in thread "main" java.lang.IllegalArgumentException: 'c': Illegal metric name. The metric name contains unsupported characters Call PrometheusNaming.sanitizeMetricName(name) to avoid this error.

On the other hand, single character Tag names and values (Prometheus Labels) are still valid: registry.counter("test", "a", "b").increment();.

Number format

The number format of _count and _bucket (Summary/Histogram) was changed in Prometheus Java client 1.x. Since these values are always integers, the fraction was dropped. Theoretically this should not cause issues, the server that scrapes the output should be able to parse both. See the following example using Micrometer Timer (Timer.builder("test").register(registry).record(100, MILLISECONDS);).

Old Client (0.x) (micrometer-registry-prometheus-simpleclient)

# TYPE test_seconds summary
# HELP test_seconds
test_seconds_count 1.0
test_seconds_sum 0.1

New Client (1.x) (micrometer-registry-prometheus)

# TYPE test_seconds summary
# HELP test_seconds
test_seconds_count 1
test_seconds_sum 0.1

Please notice the value of test_seconds_count changing from 1.0 to 1.

You can experience the same behavior with _bucket if you enable .publishPercentileHistogram():

Old Client (0.x) (micrometer-registry-prometheus-simpleclient)

# TYPE test_seconds histogram
# HELP test_seconds
test_seconds_bucket{le="0.001"} 0.0
[...]

New Client (1.x) (micrometer-registry-prometheus)

# TYPE test_seconds histogram
# HELP test_seconds
test_seconds_bucket{le="0.001"} 0
[...]

Trailing commas were removed (Prometheus Text Format only)

The Prometheus Text Format (but not the OpenMetrics format) contained trailing commas (,) in the label list of every time series that contained labels. These trailing commas were removed in Prometheus Java client 1.x. Theoretically this should not cause issues, the server that scrapes the output should be able to parse both especially in case of modern Prometheus servers since they ask for the OpenMetrics format where trailing commas were missing originally so nothing needed to be removed. See the following example using a Micrometer Counter (registry.counter("test.counter", "a", "b").increment();).

Old Client (0.x) (micrometer-registry-prometheus-simpleclient) using Prometheus Text Format

# HELP test_counter_total
# TYPE test_counter_total counter
test_counter_total{a="b",} 1.0

New Client (1.x) (micrometer-registry-prometheus) using Prometheus Text Format

# HELP test_counter_total
# TYPE test_counter_total counter
test_counter_total{a="b"} 1.0

Please notice the end of the time series changing from {a="b",} 1.0 to {a="b"} 1.0.

Exemplar value for _count and _bucket

The value of the Exemplar used in _count (Summary/Histogram) was changed. In the old client (0.x) (micrometer-registry-prometheus-simpleclient), the value was always 1.0 since for every recording, the counter was incremented by 1. In the new client (1.x) (micrometer-registry-prometheus), the Exemplar value of _count is the recorded value. For example in the case of recording latency, if the last recorded (and sampled) value was 250ms (0.25s), the Exemplar value is 0.25 (recorded latency) instead of 1.0 (counter increment). See the following example using Micrometer Timer (Timer.builder("test").register(registry).record(250, MILLISECONDS);).

Old Client (0.x) (micrometer-registry-prometheus-simpleclient)

# TYPE test_seconds summary
# HELP test_seconds  
test_seconds_count 1.0 # {span_id="321",trace_id="123"} 1.0 1715384800.092
test_seconds_sum 0.25

New Client (1.x) (micrometer-registry-prometheus)

# TYPE test_seconds summary
# HELP test_seconds  
test_seconds_count 1 # {span_id="321",trace_id="123"} 0.25 1715384800.092
test_seconds_sum 0.25

Please notice the value of Exemplar changing from 1.0 to 0.25 in # {span_id="321",trace_id="123"} 0.25.

Histogram vs. Summary

In the Prometheus text and in the OpenMetrics formats, a Summary can have count, sum, and a set of quantiles while a Histogram can have count, sum and it must have at least one bucket (it cannot have quantiles). Previously, Micrometer and the old Prometheus Java Client (0.x) (micrometer-registry-prometheus-simpleclient) allowed creating Histograms with quantiles, so this worked:

Timer.builder("test")
    .publishPercentiles(0.99)
    .publishPercentileHistogram()
    .register(registry)
    .record(Duration.ofMillis(10));

likewise, this too (since SLO means extra histogram bucket):

Timer.builder("test")
    .publishPercentiles(0.99)
    .serviceLevelObjectives(Duration.ofMillis(100))
    .register(registry)
    .record(Duration.ofMillis(10));

but unfortunately neither of these produced a valid output. They produced something which was neither a Summary (since it had buckets) nor a Histogram (since it had quantiles). This was ok for the Prometheus Server (it did not drop the data) but this is not ok for the new Prometheus Java Client (1.x) (micrometer-registry-prometheus). Trying to do this will result in an exception. Because of this Micrometer will favor histogram (Prometheus Histogram) over quantiles (Prometheus Summary) and it will ignore the quantiles if a histogram is also requested. This means that both of the examples above produces a valid Histogram that does not contain quantiles.

There is still a case where you can get into a scenario where the new Prometheus Java Client (1.x) throws an exception: if you want to have quantiles and histograms within the same "Metric Family" (same name but different set of tags), for example:

Timer.builder("test")
    .tags("index", "1")
    .serviceLevelObjectives(Duration.ofMillis(100))
    .register(registry)
    .record(Duration.ofMillis(10));
Timer.builder("test")
    .tags("index", "2")
    .publishPercentiles(0.99)
    .register(registry)
    .record(Duration.ofMillis(10));

will result in

Exception in thread "main" java.lang.ClassCastException: class io.prometheus.metrics.model.snapshots.SummarySnapshot$SummaryDataPointSnapshot cannot be cast to class io.prometheus.metrics.model.snapshots.HistogramSnapshot$HistogramDataPointSnapshot (io.prometheus.metrics.model.snapshots.SummarySnapshot$SummaryDataPointSnapshot and io.prometheus.metrics.model.snapshots.HistogramSnapshot$HistogramDataPointSnapshot are in unnamed module of loader 'app')

This is because the Prometheus type is defined on the level of the "Metric Family" should different set of tags (Prometheus labels) within the same name must have the same type. In this case the name is "test" so everything under this name must have the same type (Summary OR Histogram in this case), if the type is different while the name is the same, the new Prometheus Java Client (1.x) will throw an exception similar to the one above.