[kafka] One consumer for multiple kafka topics

[koderoff]

I have several topics which are being populated by a separate app. Is there a way to consume from all of them by one consumer?

[Sevavietl]

As far as I know, you cannot do this using enqueue.

php-rdkafka (based on librdkafka) on which php-enqueue/rdkafka is based provides low-level consuming for such cases:

$queue = $rk->newQueue();

$topic1 = $rk->newTopic("topic1");
$topic1->consumeQueueStart(0, RD_KAFKA_OFFSET_BEGINNING, $queue);
$topic1->consumeQueueStart(1, RD_KAFKA_OFFSET_BEGINNING, $queue);

$topic2 = $rk->newTopic("topic2");
$topic2->consumeQueueStart(0, RD_KAFKA_OFFSET_BEGINNING, $queue);

while (true) {
    $msg = $queue->consume(1000);
}

Enqueue library implements only high-level consuming.

[Steveb-p]

According to example present in the docs: https://arnaud-lb.github.io/php-rdkafka/phpdoc/rdkafka.examples-high-level-consumer.html it is possible to use multiple topics in high level consumer, so we might look into allowing this in some form or another.

[Sevavietl]

@Steveb-p nice point, my bad.

From the source code, I see no other problems to implement this in the rdkafka transport except than:

1. RdKafkaConsumer::getQueue() should return Queue and in the current implementation Queue is the topic itself (RdKafkaTopic).
2. As transport can be configured with DSN only, as I understand, don't you know if there any conventions for specifying multiple topics there? I have never used this approach myself.

What do you think about mentioned caveats?

Thank you in advance for the answers.

[Steveb-p]

@Sevavietl I wasn't considering what changes to existing code should be made for it to work, or if it even makes sense considering other implementations (of enqueue queues). To be frank, I was almost sure one cannot use multiple topics in high level consumer myself.

As far as 2. is concerned, dsn should be perfectly capable of handling it. Specifically, php-enqueue/dsn can work with arrays like so:
https://github.com/php-enqueue/dsn/blob/5d011c648395307c740d8c0e420f8a9faebce72f/Tests/DsnTest.php#L474-L483

All that would be required is to always treat topic as array and convert single topic into an array if passed.

There might be some more configuration changes that need to be done, I just skimmed through what came to my mind.

[Sevavietl]

@Steveb-p thank you for the prompt response. I would be glad to help with this functionality, will try to create the PR

[Steveb-p]

@Sevavietl did you have time to look into this? Or can I take it over?

[Sevavietl]

@Steveb-p I have looked into, but have not implemented it into PR yet, so if you want you can take it over.
Or, better yet, maybe we can discuss the changes we need to implement this.
I do not know if here is the best place for a long discussion...

[Steveb-p]

@Sevavietl I’m available on php-enqueue’s gitter most od the time :)

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[Steveb-p]

Hi stale-bot, be quiet please ;)

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[diguin]

I know I'm a little late for the party, but I just faced the same problem. I noticed that you are able to use a regex as a queue/topic name. You just have to prefix it with ^

[harley84]

+1, having the ability to subscribe to multiple topics at once would be very helpful

[fmiqbal]

sry, any update on this ?