consumer: batch record number >2*math.MaxUint16 leads to error decoding packet: invalid array length

[july2993]

Versions

Sarama Version:
1.18
Kafka Version:
2.0.0
Go Version:
1.11

Configuration

What configuration values are you using for Sarama and Kafka?
sarama

    sarama.MaxResponseSize = 1 << 30
    sarama.MaxRequestSize = 1 << 30
    config.Producer.MaxMessageBytes = 1 << 30
    config.Producer.Partitioner = sarama.NewManualPartitioner
    config.Producer.RequiredAcks = sarama.WaitForAll

Logs

When filing an issue please provide logs from Sarama and Kafka if at all
possible. You can set sarama.Logger to a log.Logger to capture Sarama debug
output.

sarama
maybe cause by here https://github.com/Shopify/sarama/blob/master/real_decoder.go#L88

error decoding packet        : invalid array length

Problem Description

while consume, meet the invalid array length error.

after change, the producer(set config.Producer.Flush.MaxMessages = math.MaxUint16), the consumer would'not meet the invalid array length.

i print the tmp at maybe cause by here https://github.com/Shopify/sarama/blob/master/real_decoder.go#L88, it will not bigger than the MaxMessages config in producer

i think the problem is using the default Flush config, the batch record number may bigger than 2
x math.MaxUint16, and when consuming, sarama treat it invalid array length if record number > 2 x math.MaxUint16.

if kafka has no limit for batch records num, can we remove the length limit?
or better to change the default config of Flush to limit MaxMessages

[yehaotong]

I also meet this proble when use kafka-exporter to scrape kafka metrics, and logging: Cannot get consumer group: kafka: error decoding packet: invalid array length

[dnwe]

@yehaotong that's a different issue you were seeing — invalid array length is a general encoding/decoding error

This issue is specifically talking about the batch record numbers:

i think the problem is using the default Flush config, the batch record number may bigger than 2
x math.MaxUint16, and when consuming, sarama treat it invalid array length if record number > 2 x math.MaxUint16.

[github-actions]

Thank you for taking the time to raise this issue. However, it has not had any activity on it in the past 90 days and will be closed in 30 days if no updates occur.
Please check if the main branch has already resolved the issue since it was raised. If you believe the issue is still valid and you would like input from the maintainers then please comment to ask for it to be reviewed.