-
Notifications
You must be signed in to change notification settings - Fork 990
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Prometheus requires that all meters with the same name have the same set of tag keys #3508
Comments
Some of my findings related to this issue: At bootstrap, the As you can see in the bindTo method, there is a scheduler
which will be firing every ~30 seconds to look for new metrics to bind. So, the first time this scheduled timer runs (~30 seconds after bootstrap) it will find the query-tracking and query-routing consumer metrics and it will try to bind them but now both query-tracking and query-routing consumer metrics have an extra tag named topic (which holds the topic name) and when it tries to bind these new metrics the |
I assume the extra tag appears after the consumer is assigned partitions from the topic, which happens after initialization. |
I believe I have encountered the same issue in a slightly different context: We have a bunch of components producing similarly structured counters, and for a given counter name, if all calls use the same set of labels, everything works as expected: //provided a valid MeterRegistry registry
registry.counter("my_metric", Set.of(
Tag.of("some_label", "value_1"),
Tag.of("other_label", "value_2")
)).increment(1);
/* Someplace else */
registry.counter("my_metric", Set.of(
Tag.of("some_label", "value_2"),
Tag.of("other_label", "value_3")
)).increment(1); However, if some calls use a different set of labels, like, e.g.: //provided a valid MeterRegistry registry
registry.counter("my_metric", Set.of(
Tag.of("some_label", "value_1"),
Tag.of("other_label", "value_2")
)).increment(1);
/* Someplace else */
registry.counter("my_metric", Set.of(
Tag.of("some_label", "value_1")
)).increment(1); Some of those sets will be absent from the Prometheus endpoint but not from the MeterRegistry itself (i.e. in Spring they do show up in |
I think this is a duplicate of #877 |
Seems like a duplicate of #877 |
Describe the bug
While I was investigating a missing kafka metric in our monitoring system I found out it was caused by our application (spring boot using micrometer) not publishing the metric
kafka.consumer.fetch.manager.records.consumed.total
.The application has two kafka consumers, lets call them query-routing and query-tracking consumers, and they are configured via
@KafkaListener
annotation and each kafka consumer has it's own instance ofConcurrentKafkaListenerContainerFactory
.The query-router consumer is configured as:
And the query-tracking consumer is configured as:
The MeterRegistryConfigurator bean configuaration is set as:
The
@KafkaListener
for each consumer is set asand
When the application starts up, going to the actuator/prometheus endpoing I can see the metric for both consumers:
But a few seconds later there is a new call to
io.micrometer.core.instrument.binder.kafka.KafkaMetrics#checkAndBindMetrics
which will remove a set of metrics (includingkafka.consumer.fetch.manager.records.consumed.total)
Going again to actuator/prometheus will only show the metric for the query-routing consumer:
As you can see above the metric for the query-tracking consumer is gone. As the log says, The meter you are attempting to register has keys [client_id, kafka_version, spring_id, topic]. The issue is I cannot find where is this metric with a topic key being registered which will trigger io.micrometer.core.instrument.binder.kafka.KafkaMetrics#checkAndBindMetrics which will remove the metric for the query-tracking consumer.
Environment
To Reproduce
How to reproduce the bug:
Please check description above
Expected behavior
I would expect the kafka metrics for both consumers to be published into spring actuator, even if both metrics have different set of tags.
Additional context
The text was updated successfully, but these errors were encountered: