Producers/Consumers to support specific topics within a topology to access

[jeqo]

Is your feature request related to a problem? Please describe.

Producers/Consumer:

    consumers:
      - principal: "User:App0"
      - principal: "User:App1"
    producers:
      - principal: "User:App3"
      - principal: "User:App4"

Get access to all the topics within a topology.

Describe the solution you'd like

Support defining specific topics, similar to Kafka Streams:

   consumers:
     - principal: "User:App0"
     - principal: "User:App1"
        topics:
          read:
            - "topicA"
            - "topicB"

    producers:
      - principal: "User:App3"
      - principal: "User:App4"
        topics:
          write:
            - "topicC"
            - "topicD"

Not sure if this is supported in any other way.

[purbon]

Hi @jeqo thanks for your proposal. this should be already possible using the #336 - custom roles feature. What do you think?

[jeqo]

Oh, I love the idea of custom roles feature!

Though, I still see value on extending producer/consumer abstractions to have the ability to filter the roles they work with. It also align these two with how streams and connectors behave.

wdyt?

[akselh]

I guess you know that you can already define consumers and producers under a specific topic @jeqo? This will achieve pretty much the same thing, right?

[jeqo]

Aha! I didn't know about this feature, thanks @akselh!

So this should be possible:

    topics:
      - name: bar
        consumers:
          - principal: "User:App0"
          - principal: "User:App1"
        producers:
          - principal: "User:App3"
          - principal: "User:App4"

That's it then. Will close the this.

One more reason to work on this issue: #358 ;-)