Consumer group client does not back-off / retry for possible error scenarios

[dotnwat]

Versions

Sarama	Kafka	Go
0189d59	2.4

Configuration

config.Version = sarama.V2_4_0_0
config.Consumer.Return.Errors = true
config.Consumer.Offsets.Initial = sarama.OffsetNewest

Logs

2020/06/02 10:39:50 Initializing new client
2020/06/02 10:39:50 client/metadata fetching metadata for all topics from broker localhost:9092
2020/06/02 10:39:50 Connected to broker at localhost:9092 (unregistered)
2020/06/02 10:39:50 client/brokers registered new broker #1 at 0.0.0.0:9092
2020/06/02 10:39:50 Successfully initialized new client
2020/06/02 10:39:50 client/metadata fetching metadata for [sanfrancisco] from broker localhost:9092
2020/06/02 10:39:50 client/metadata found some partitions to be leaderless
2020/06/02 10:39:50 client/metadata retrying after 250ms... (3 attempts remaining)
2020/06/02 10:39:50 client/metadata fetching metadata for [sanfrancisco] from broker localhost:9092
2020/06/02 10:39:50 client/coordinator requesting coordinator for consumergroup sfo-consumer-group from localhost:9092
2020/06/02 10:39:50 client/coordinator coordinator for consumergroup sfo-consumer-group is #1 (0.0.0.0:9092)
2020/06/02 10:39:50 Connected to broker at 0.0.0.0:9092 (registered as #1)
2020/06/02 10:39:50 ProcessingLoop error:  kafka server: The broker is still loading offsets after a leader change for that offset's topic partition.

Problem Description

When calling Consume on a consumer group client the Kafka broker is returning the error COORDINATOR_LOAD_IN_PROGRESS = 14 when joining the group, but Sarama does not appear to implement the back-off / retry logic for this error condition. It only does back-off / retry for NOT_COORDINATOR = 16 (see here: https://github.com/Shopify/sarama/blob/master/consumer_group.go#L227).

This case is handled in kafka java reference client (see: https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/clients/consumer/internals/AbstractCoordinator.java#L609).

Generally this error COORDINATOR_LOAD_IN_PROGRESS and NOT_COORDINATOR and COORDINATOR_NOT_AVAILABLE are all valid errors from any of the group membership APIs.

Sarama handles the load in progress condition for fetching offsets, but when we experience this error via join group, the client doesn't retry as expected.

[dnwe]

Closing as dupe of #2058 believed to be fixed (via #2214) in v1.33.0 and newer