Kafka consumer offsets not working after broker restart

[arthurh4]

Hello,

I'm getting this strange behavior recently that when we restart a broker the kafka lag exporter stops pulling consumer offset correctly.

For example we had the following crash during a restart:
2019-12-13 21:53:58,784 ERROR c.l.k.ConsumerGroupCollector$ akka://kafka-lag-exporter/user/consumer-group-collector-nycp-main - Supervisor RestartSupervisor saw failure: A failure occurred while retrieving offsets. Shutting down. java.lang.Exception: A failure occurred while retrieving offsets. Shutting down. at com.lightbend.kafkalagexporter.ConsumerGroupCollector$CollectorBehavior.$anonfun$collector$1(ConsumerGroupCollector.scala:188)

And then the threads restarts automaticly without errors:

2019-12-13 21:53:59,998 INFO o.a.kafka.common.utils.AppInfoParser - Kafka version: 2.3.0
2019-12-13 21:53:59,998 INFO o.a.kafka.common.utils.AppInfoParser - Kafka commitId: fc1aaa116b661c8a .
2019-12-13 21:53:59,998 INFO o.a.kafka.common.utils.AppInfoParser - Kafka startTimeMs: 1576274039998 .
2019-12-13 21:53:59,998 INFO c.l.k.ConsumerGroupCollector$ akka://kafka-lag-exporter/user/consumer-group-collector-nycp-main - Collecting offsets .
2019-12-13 21:54:05,553 INFO org.apache.kafka.clients.Metadata - [Consumer clientId=consumer-327, groupId=kafkalagexporter] Cluster ID: rCxAZjktT4CsAXv85w9dUw .
2019-12-13 21:54:14,108 INFO c.l.k.ConsumerGroupCollector$ akka://kafka-lag-exporter/user/consumer-group-collector-nycp-main - Updating lookup tables .
2019-12-13 21:54:14,114 INFO c.l.k.ConsumerGroupCollector$ akka://kafka-lag-exporter/user/consumer-group-collector-nycp-main - Reporting offsets .
2019-12-13 21:54:14,132 INFO c.l.k.ConsumerGroupCollector$ akka://kafka-lag-exporter/user/consumer-group-collector-nycp-main - Clearing evicted metrics
2019-12-13 21:54:14,132 INFO c.l.k.ConsumerGroupCollector$ akka://kafka-lag-exporter/user/consumer-group-collector-nycp-main - Polling in 60 seconds

However on the prometheus endpoint the consumer offsets stays the same and never get updated. I tried to play with the configs without any luck ? any ideas ? Also Could we just crash the pod entirely if it fails to pull the offsets from kafka ?

[seglo]

Thanks for reporting this issue. I was able to reproduce and resolve this bug in #154

[seglo]

Fixed in 0.6.3 https://github.com/lightbend/kafka-lag-exporter/releases/tag/v0.6.3