Clarification on sarama.MaxResponseSize limit

[4578395263256]

I don't think the comment here is necessarily correct: https://github.com/Shopify/sarama/blob/master/sarama.go#L44
as I have observed receiving a 170MB fetch response that is hitting the MaxResponseSize limit. It seems like socket.request.max.bytes is a limit only on the request size and not the response size. Also this: https://issues.apache.org/jira/browse/KAFKA-2063 seems to indicate that you can't actually control the fetch response size.

Also just curious as to whether there is some protocol limitation that absolutely prevents a response from being larger than 2GB?

[eapache]

Hmm, you may be right, I'll dig into this more tomorrow.

Also just curious as to whether there is some protocol limitation that absolutely prevents a response from being larger than 2GB?

Yes, every message starts with a signed 32-bit length field, so a message cannot ever contain more than 2^31 bytes (2GiB). See https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol#AGuideToTheKafkaProtocol-CommonRequestandResponseStructure