Multiple Kafka topics for publish events

[LudovitVarga]

According to documentation RIG supports only one Kafka topic to publish messages. It would be good idea to support more topics to publish events.

[kevinbader]

I assume you're referring to PROXY_KAFKA_REQUEST_TOPIC.

Currently, RIG supports only one topic because a frontend (or external client) is not supposed to know anything about the topics used by the backend and therefore cannot select a topic through the request it sends. Could you please elaborate on the use case you have in mind?

[LudovitVarga]

frontend (or external client) is not supposed to know anything about the topics used by the backend

I agree and that is why we have proxies that are aware of this information. If RIG can act as proxy than why RIG can't have this information about different topics? Currently you define in configuration, url that acts as endpoint to publish messages to Kafka. So why you can't provide multiple endpoints for multiple topics? For frontend is this single entry point, same as in case you have http proxy.

[kevinbader]

How would you configure which events go to which topic (and partition)?

[LudovitVarga]

Partition can be defined same way as in current solution. Topic can be specified as new parameter in json configuration of endpoints where now you have parameter "target=kafka".

[shafqatevo]

Yes, I consider this essential too. This will open up a lot of flexibility for clients in an event-driven architecture.