Differentiate between client and server version of ErrMessageSizeTooLarge

[wvanbergen]

Right now, the Kafka broker can return ErrMessageSizeTooLarge if a message is deemed to large based on the broker configuration, but we also return it client-side in the producer based on the sarama configuration: https://github.com/Shopify/sarama/blob/53f2a54a5cfd662728648b20eb26e4a8e19b4887/async_producer.go#L255

For debugging purposes, it would be nice to be able to differentiate between the client and server-side version.

cc @andremedeiros @kvs @eapache

[eapache]

There are a lot of cases where we return kafka server errors directly:

• https://github.com/Shopify/sarama/blob/53f2a54a5cfd662728648b20eb26e4a8e19b4887/consumer.go#L366
• https://github.com/Shopify/sarama/blob/53f2a54a5cfd662728648b20eb26e4a8e19b4887/async_producer.go#L255
• https://github.com/Shopify/sarama/blob/53f2a54a5cfd662728648b20eb26e4a8e19b4887/client.go#L225
• https://github.com/Shopify/sarama/blob/53f2a54a5cfd662728648b20eb26e4a8e19b4887/client.go#L253
• https://github.com/Shopify/sarama/blob/53f2a54a5cfd662728648b20eb26e4a8e19b4887/client.go#L275
• https://github.com/Shopify/sarama/blob/53f2a54a5cfd662728648b20eb26e4a8e19b4887/client.go#L312
• https://github.com/Shopify/sarama/blob/53f2a54a5cfd662728648b20eb26e4a8e19b4887/client.go#L351
• https://github.com/Shopify/sarama/blob/53f2a54a5cfd662728648b20eb26e4a8e19b4887/client.go#L506
• https://github.com/Shopify/sarama/blob/53f2a54a5cfd662728648b20eb26e4a8e19b4887/client.go#L513
• https://github.com/Shopify/sarama/blob/53f2a54a5cfd662728648b20eb26e4a8e19b4887/client.go#L540

At the time it seemed unnecessary to define a new local error when the kafka-provide value had the exact right meaning, but I guess this could lead to confusion...

[wvanbergen]

This case is a bit more confusing because it is determined by two configuration settings that live in different places that could be different from each other. This made debugging this error a lot harder.

Many of the other cases where we do this are not really influenced by different configuration settings, and we return what the broker would return anyway based on our best locally cached knowledge. I am not against a generic solution, but we jumped in the rabbit hole of adding context to these errors before without much luck.

Note: we also have ErrMessageTooLarge: https://github.com/Shopify/sarama/blob/master/errors.go#L37-L38

[eapache]

Ah you're right, that's legitimately confusing. Especially when you throw the consumer version of the error into the mix.

[AusIV]

I don't know that I have much more to add than a "me too", but I lost some critical time because this error is outright misleading. The error message:

kafka server: Message was too large, server rejected it to avoid allocation error.

It didn't even occur to me to look at the producer limits when the message specifically says the error was rejected by the server, and I knew the server limits were higher than the messages I was sending.

[dnwe]

Fixed by #2628 🎉