"kafka: response did not contain all the expected topic/partition blocks" out of nowhere? when changing compression

[Dieterbe]

Hi, I think if I interpret the kafka docs, i should be allowed to switch back and forth between none/gzip/snappy compression to the same topic/partition, right?

anyway when I try this, first i use no compression, than kill my producer, restart it with snappy or gzip compression, send data to the same topic/partition, i get this in the logs:

[Sarama] 2016/07/08 21:51:24 consumer/broker/0 added subscription to mdm/0
[Sarama] 2016/07/08 21:51:24 kafka: error while consuming mdm/0: kafka: response did not contain all the expected topic/partition blocks
[Sarama] 2016/07/08 21:51:24 consumer/broker/0 abandoned subscription to mdm/0 because kafka: response did not contain all the expected topic/partition blocks
[Sarama] 2016/07/08 21:51:26 consumer/mdm/0 finding new broker
[Sarama] 2016/07/08 21:51:26 client/metadata fetching metadata for [mdm] from broker kafka:9092
[Sarama] 2016/07/08 21:51:26 consumer/broker/0 added subscription to mdm/0
[Sarama] 2016/07/08 21:51:26 kafka: error while consuming mdm/0: kafka: response did not contain all the expected topic/partition blocks
[Sarama] 2016/07/08 21:51:26 consumer/broker/0 abandoned subscription to mdm/0 because kafka: response did not contain all the expected topic/partition blocks
[Sarama] 2016/07/08 21:51:28 consumer/mdm/0 finding new broker
[Sarama] 2016/07/08 21:51:28 client/metadata fetching metadata for [mdm] from broker kafka:9092
[Sarama] 2016/07/08 21:51:28 consumer/broker/0 added subscription to mdm/0
[Sarama] 2016/07/08 21:51:28 kafka: error while consuming mdm/0: kafka: response did not contain all the expected topic/partition blocks
[Sarama] 2016/07/08 21:51:28 consumer/broker/0 abandoned subscription to mdm/0 because kafka: response did not contain all the expected topic/partition blocks
[Sarama] 2016/07/08 21:51:30 consumer/mdm/0 finding new broker
[Sarama] 2016/07/08 21:51:30 client/metadata fetching metadata for [mdm] from broker kafka:9092
[Sarama] 2016/07/08 21:51:30 consumer/broker/0 added subscription to mdm/0
[Sarama] 2016/07/08 21:51:30 kafka: error while consuming mdm/0: kafka: response did not contain all the expected topic/partition blocks
[Sarama] 2016/07/08 21:51:30 consumer/broker/0 abandoned subscription to mdm/0 because kafka: response did not contain all the expected topic/partition blocks

Sarama Version: latest git master
Kafka Version: 0.10
Go Version: 1.7rc1

any idea? thanks!

[eapache]

Hi, I think if I interpret the kafka docs, i should be allowed to switch back and forth between none/gzip/snappy compression to the same topic/partition, right?

Off the top of my head, https://issues.apache.org/jira/browse/KAFKA-1499 suggests that you should, and that it will all end up uniform on the broker.

kafka: response did not contain all the expected topic/partition blocks

That is... surprising. It sounds like you've hit some very peculiar bug in either Sarama or upstream Kafka. I just did a quick skim of our produce-response-parsing code and didn't spot anything, so I would suggest you take a look at the actual broker logs when you try this. Is the broker logging any exceptions? If the Sarama log is accurate then the broker is sending a malformed response and you'll need to take it up with upstream.

[Dieterbe]

Thanks @eapache
i took a copy of all ZK and kafka logs when stopping the nonecompression producer, then restarted producer with snappy enabled, confirmed the errors appearing in my consumer, but some diff -r'ing confirms nothing new appeared in either the ZK or kafka logs :( no exceptions either.

FWIW here's all the kafka, zk and supervisor logs https://gist.github.com/Dieterbe/7142236d6828b7bb078c6f0172b543c4 and this is how I'm running it: https://github.com/raintank/raintank-docker/tree/master/kafka

if it still looks like an upstream issue to you, i'll take this to https://issues.apache.org/jira/browse/KAFKA/

[eapache]

Ya, I'm not sure what the problem is still. You might try inspecting the actual traffic on the wire using a tool like Wireshark to see if the Sarama error message is correct and the broker responses are actually missing the relevant blocks? I definitely suspect an upstream issue, though I have no proof.

[Dieterbe]

https://issues.apache.org/jira/browse/KAFKA-3961

[Dieterbe]

so i found out i can trigger this behavior without going from none->gzip or none->snappy. just using gzip/snappy directly works too. i tried sniffing with wireshark but it's hard to inspect the tcp stream, it looks pretty garbled. probably due to the compression.

[eapache]

Is #720 the explanation?

[Dieterbe]

Hey @eapache don't have time to run the test right now but it seems very likely! glad we're on to something!

[eapache]

Yes, this is the same as #720.

[Dieterbe]

using the latest sarama i'm still seeing this when trying to use snappy:

[Sarama] 2016/09/20 08:57:37 kafka: error while consuming mdm/1: kafka: response did not contain all the expected topic/partition blocks
[Sarama] 2016/09/20 08:57:37 consumer/broker/0 abandoned subscription to mdm/1 because kafka: response did not contain all the expected topic/partition blocks

i use config.Version = sarama.V0_10_0_0 btw. we manually manage offsets and use sarama.NewClient and sarama.NewConsumerFromClient

[eapache]

It's possible my fix in #735 missed a case or something, but it worked in my simple tests so I'm not sure what might be going on. It would be helpful if you could put some additional logging in that flow and see where the issue is.