Consumer group example

[dhawal55]

Can someone add example code on how to implement a consumer group using OffsetManager? I looked at https://github.com/wvanbergen/kafka, but it is uses zookeeper. New version of Kafka allows to store consumer offset in Kafka and OffsetManager seems to enable it. An example implementation will be very helpful.

[wvanbergen]

wvanbergen/kafka#72 is almost done, and should serve as a good example.

[dhawal55]

It doesn't answer my question. The pull request you have improves the original implementation (which uses zookeeper for managing consumer offset) and add logic to also use zookeeper for managing partition metadata as well.
I'm looking for an implementation that does not use zookeeper and instead uses Kafka for managing consumer offset. Zookeeper does not scale well for writes as per this post: https://cwiki.apache.org/confluence/display/KAFKA/Committing+and+fetching+consumer+offsets+in+Kafka

[wvanbergen]

That's exactly what that pull request does. It will still need zookeeper for coordination of instances though.

[dhawal55]

Sorry, I was still looking at ConsumerGroup. How do I fetch the offset? In the example, you start with OffsetNewest. I want to fetch the offset (if available) and start from there.

[wvanbergen]

The OffsetManager takes care of this. When I start a partition consumer, the PartitionOffsetManager will return the next offset to consume. In case there is no stored offset, it will use the initial offset as configured. See #461

[dhawal55]

Thank you so much. I will play with a bit to get more familiar. Is there any plan to merge it with Shopify/sarama?

[wvanbergen]

Nope, because of the Zookeeper dependency.

A future version of Kafka is supposed to have all the APIs to be able to implement a high-level consumer. At that point we will probably add it to sarama.

[jan4984]

as kafka 0.9 is released now, Zookeeper dependency is removed.
do we have any plan for it now?

[wvanbergen]

Yup, it's in scope of sarama now. However, we don't have any immediate plans to start working on it. Contributions are definitely welcomed!