Cannot trigger a rebalance without calling poll

[emilk]

I am trying to implement exactly-once delivery using several consumers (running in different instances). Each message should only end up in exactly ONE consumer. When I start a new consumer I want to trigger an automatic rebalance so that the new consumer gets assigned a few partitions. The existing consumers commits on rebalance, so that the new consumer does not read the same messages (avoiding duplicates).

The problem is that calling subscribe does NOT trigger a rebalance. The only method I have found which reliably triggers a rebalance is a blocking (None timeout) call to poll. The problem is that this will read max.partition.fetch.bytes worth of messages that the existing consumers have already processed, thus creating duplicates.

Is there any other way to trigger a rebalance?

[benesch]

Interesting! Upstream suggests that you pause the consumer, call poll until the rebalance, and then unpause it: confluentinc/librdkafka#2394

Unfortunately, we don't currently expose pause, but we could! PRs welcome.