Support specification of a consumer group #283
Conversation
|
|
||
| if (this.consumerGroup != defaultConsumerGroup) | ||
| { | ||
| await this.EnsureConsumerGroupsExistAsync(); |
There was a problem hiding this comment.
Should we consider doing the Consumer Group Creation in the OrchestrationServiceCreateAsync call? since its a resource being created?
There was a problem hiding this comment.
I think that would not work beause the event hub itself is dynamically created (and sometimes deleted and recreated for certain recovery scenarios).
|
Well see discussion : #292 I tried with the new setting : EventHubsConsumerGroup I added : AzureFunctionsJobHost__extensions__durableTask__EventHubsConsumerGroup Furthermore, wouldn't is be more logical to have the taskhubname as the EventHubsConsumerGroup ? |
What type of authentication are you using? A SAS connection string or a managed identity?
Perhaps, yes, I can see that being a little simpler. We still need a configuration setting because this feature is not stable enough at this point to have it be always enabled. |
I am using a managed identity for everythingf |
Did you assign "Azure Event Hubs Data Owner" role? If not, it could be that the app does not have sufficient permissions to create the consumer group. |
Yes I did |
|
|
Hi @sebastianburckhardt, my team is looking to use this feature to support multiple task hubs. I see that this PR has been open for a while. Do you have any updates on this? |
|
We don't currently have a time estimate for providing this feature. The PR is unfortunately no longer up to date with the latest version of the event hubs SDK we are using now. Also, supporting multiple task hubs on the same EH namespace has some issues even when using consumer groups (e.g. the traffic is still redundantly delivered to all workers), and we were not able to fully automate the management of consumer groups for these scenarios, so we are a bit reluctant to add this feature since it will increase the support surface. |
As suggested by one of our users, this PR adds a feature to support specifying a consumer group other than the current default (
$Default) in the options.Specifying separate consumer groups makes it possible to run multiple task hubs on the same EH namespace without having them steal messages from each other (and avoid the chaos that ensues).
Notes about this design:
to make this possible there is now a string option
EventHubsConsumerGroupinNetheriteOrchestrationServiceSettings(and host.json). The default for this setting is$Defaultwhich is also the default consumer group that event hubs already provides, and which is being used currently by all applications. Applications that do not explicitly change that setting will continue to function the same way as they do prior to this PR.If set to a value other than
$Default, Netherite uses that consumer group instead of the default. It will also create this consumer group (which is an Azure resource that is visible in the portal, under the tab for the event hub) if it does not already exist.Note: consumer groups are not automatically deleted, and some plans limit the number of consumer groups (lowest tier allows only a single consumer group). So if using the non-default consumer group, it is possible to get into a situation where Netherite cannot execute because it cannot create the consumer group. In that case users would have to explicitly delete some consumer groups (or perhaps just the entire EH namespace which is less complicated).
Note that even if two task hubs that use the same EH namespace have different consumer groups, each message is still 'double delivered' since it is delivered to both task hubs (one of them will ignore it but it still consumes network and some CPU resources and counts towards the throughput limit). So, if both task hubs are heavily active at the same time, it may be a better idea to continue to use separate EH namespaces.