How to set up the eventhubs ? (can run only 1 function app at a time on netherite) #292

well0549 · 2023-07-25T09:44:11Z

well0549
Jul 25, 2023

I have a collection of 12 Function apps that communicate with queues.....
Each function app is in it's own service plan and has about 5 activities.

A service bus message is used as a trigger, the trigger starts the orchestration, the activities run and the message is put into the next queue.

This runs great on azure storage but unfortunately not on Netherite......

I created an eventhub just as described in the documentation. And I put the connection string in the EventHubsConnection in the host.json....

If i do this for one function, it works ok.
If I set the storageprovider to netherite for all the functions it doesn't work anymore.....
I get all kind of weird messages (out or order/cannot create orchestration)

Am i missing something ?
Should there be an eventhub-namespace per Function ?
Or maybe create an eventhub per function ? And use that connection as EventHubsConnection so it's different for every function ?

Tried several things but cannot see to get it as relieable as AzuraStorage

If i change the storageprovider back to Azure Storage it works again.

well0549 · 2023-07-25T09:54:49Z

well0549
Jul 25, 2023
Author

Some additional information.......

Each orchestration starts with a trigger from a queue and then the first activity in the orcvestration has the same name for all of the function apps

0 replies

jbarntish · 2023-07-25T13:38:43Z

jbarntish
Jul 25, 2023

You might double check that your hubName in host.json is different across all of your func apps. I have accidentally deployed our DF app to two separate app services before, both with the same hubName, which caused all kinds of craziness.

"extensions": {
        "durableTask": {
            "hubName": "yourapp1",

From Durable Function Task Hubs

The name is what differentiates one task hub from another when there are multiple task hubs in a shared storage account. If you have multiple function apps sharing a shared storage account, you must explicitly configure different names for each task hub in the host.json files. Otherwise the multiple function apps will compete with each other for messages, which could result in undefined behavior, including orchestrations getting unexpectedly "stuck" in the Pending or Running state.

0 replies

well0549 · 2023-07-25T14:25:27Z

well0549
Jul 25, 2023
Author

You might double check that your hubName in host.json is different across all of your func apps. I have accidentally deployed our DF app to two separate app services before, both with the same hubName, which caused all kinds of craziness.
"extensions": {
        "durableTask": {
            "hubName": "yourapp1",
From Durable Function Task Hubs

The name is what differentiates one task hub from another when there are multiple task hubs in a shared storage account. If you have multiple function apps sharing a shared storage account, you must explicitly configure different names for each task hub in the host.json files. Otherwise the multiple function apps will compete with each other for messages, which could result in undefined behavior, including orchestrations getting unexpectedly "stuck" in the Pending or Running state.

Well i have set a different name for each eventhub.....

But not in the host.json directly. But in the appsettings of the function app.....
i have set:

AzureFunctionsJobHost__extensions__durableTask__hubName
AzureFunctionsJobHost__extensions__durableTask__storageProvider__EventHubsConnectionName
AzureFunctionsJobHost__extensions__durableTask__storageProvider__partitionCount
AzureFunctionsJobHost__extensions__durableTask__storageProvider__StorageConnectionName
AzureFunctionsJobHost__extensions__durableTask__storageProvider__type (this is Netherite or AzureStorage)

I know the hubname is set correct and is unique for that function app.
Each function app has it's own storage account.

Not sure if it makes a difference if it is defined in the Application settings or in the host.json directly

1 reply

jbarntish Jul 25, 2023

Might try the host.json specifically just to double-check. Netherite may not be looking at the env vars.

well0549 · 2023-07-25T19:38:53Z

well0549
Jul 25, 2023
Author

keep getting these kind of errors

2023-07-25T19:35:53Z [Error] EventHubsProcessor partitions/4 received wrong packet, #19 instead of #9
2023-07-25T19:35:53Z [Error] EventHubsProcessor partitions/4(1) encountered an exception while processing packets : System.InvalidOperationException: EventHubs Out-Of-Order Packet
at DurableTask.Netherite.EventHubsTransport.BlobBatchReceiver1.ReceiveEventsAsync(Byte[] guid, IEnumerable1 hubMessages, CancellationToken token, Nullable1 nextPacketToReceive)+MoveNext() in /_/src/DurableTask.Netherite/TransportLayer/EventHubs/BlobBatchReceiver.cs:line 73 at DurableTask.Netherite.EventHubsTransport.BlobBatchReceiver1.ReceiveEventsAsync(Byte[] guid, IEnumerable1 hubMessages, CancellationToken token, Nullable1 nextPacketToReceive)+System.Threading.Tasks.Sources.IValueTaskSource<System.Boolean>.GetResult()
at DurableTask.Netherite.EventHubsTransport.EventHubsProcessor.Microsoft.Azure.EventHubs.Processor.IEventProcessor.ProcessEventsAsync(PartitionContext context, IEnumerable1 packets) in /_/src/DurableTask.Netherite/TransportLayer/EventHubs/EventHubsProcessor.cs:line 436 at DurableTask.Netherite.EventHubsTransport.EventHubsProcessor.Microsoft.Azure.EventHubs.Processor.IEventProcessor.ProcessEventsAsync(PartitionContext context, IEnumerable1 packets) in //src/DurableTask.Netherite/TransportLayer/EventHubs/EventHubsProcessor.cs:line 436
2023-07-25T19:35:53Z [Error] Part04 !!! Encountered exception while processing events in IEventProcessor.ProcessEventsAsync: System.InvalidOperationException: EventHubs Out-Of-Order Packet
at DurableTask.Netherite.EventHubsTransport.BlobBatchReceiver1.ReceiveEventsAsync(Byte[] guid, IEnumerable1 hubMessages, CancellationToken token, Nullable1 nextPacketToReceive)+MoveNext() in /_/src/DurableTask.Netherite/TransportLayer/EventHubs/BlobBatchReceiver.cs:line 73 at DurableTask.Netherite.EventHubsTransport.BlobBatchReceiver1.ReceiveEventsAsync(Byte[] guid, IEnumerable1 hubMessages, CancellationToken token, Nullable1 nextPacketToReceive)+System.Threading.Tasks.Sources.IValueTaskSource<System.Boolean>.GetResult()
at DurableTask.Netherite.EventHubsTransport.EventHubsProcessor.Microsoft.Azure.EventHubs.Processor.IEventProcessor.ProcessEventsAsync(PartitionContext context, IEnumerable1 packets) in /_/src/DurableTask.Netherite/TransportLayer/EventHubs/EventHubsProcessor.cs:line 436 at DurableTask.Netherite.EventHubsTransport.EventHubsProcessor.Microsoft.Azure.EventHubs.Processor.IEventProcessor.ProcessEventsAsync(PartitionContext context, IEnumerable1 packets) in //src/DurableTask.Netherite/TransportLayer/EventHubs/EventHubsProcessor.cs:line 436 terminatePartition=True

and these as well

2023-07-25T19:35:56Z [Warning] LoadMonitor ignored 1 packets for different taskhub
2023-07-25T19:35:56Z [Error] LoadMonitor encountered an exception while processing packets : System.NullReferenceException: Object reference not set to an instance of an object.
at DurableTask.Netherite.EventHubsTransport.LoadMonitorProcessor.Microsoft.Azure.EventHubs.Processor.IEventProcessor.ProcessEventsAsync(PartitionContext context, IEnumerable`1 packets)

2 replies

jbarntish Jul 25, 2023

Another thing worth trying: when your task hub(s) get into weird states, you may have to do a hard reset. Task Hub Restart. Option 4 has saved me numerous times.

well0549 Jul 25, 2023
Author

Did a clean start, deleted the blob containers, did a redeploy, same issue.....
Now i get an exception like.....

System.OperationCanceledException: Client request was cancelled because host is shutting down.
at DurableTask.Netherite.Client.PerformRequestWithTimeout(IClientRequestEvent request) in //src/DurableTask.Netherite/OrchestrationService/Client.cs:line 259
at DurableTask.Netherite.Client.CreateTaskOrchestrationAsync(UInt32 partitionId, TaskMessage creationMessage, OrchestrationStatus[] dedupeStatuses) in //src/DurableTask.Netherite/OrchestrationService/Client.cs:line 469
at DurableTask.Netherite.NetheriteOrchestrationService.DurableTask.Core.IOrchestrationServiceClient.CreateTaskOrchestrationAsync(TaskMessage creationMessage, OrchestrationStatus[] dedupeStatuses) in //src/DurableTask.Netherite/OrchestrationService/NetheriteOrchestrationService.cs:line 594
at DurableTask.Core.TaskHubClient.InternalCreateOrchestrationInstanceWithRaisedEventAsync(String orchestrationName, String orchestrationVersion, String orchestrationInstanceId, Object orchestrationInput, IDictionary2 orchestrationTags, OrchestrationStatus[] dedupeStatuses, String eventName, Object eventData, Nullable1 startAt) in //src/DurableTask.Core/TaskHubClient.cs:line 614
at Microsoft.Azure.WebJobs.Extensions.DurableTask.DurableClient.Microsoft.Azure.WebJobs.Extensions.DurableTask.IDurableOrchestrationClient.StartNewAsync[T](String orchestratorFunctionName, String instanceId, T input) in D:\a_work\1\s\src\WebJobs.Extensions.DurableTask\ContextImplementations\DurableClient.cs:line 210
at SNFC.DurableHelper.CreateOrchestration(String functionName, String activity, ClaimPointer originalClaim, IDurableOrchestrationClient starter, String servicebusMessageId, String orchtestrationName) in D:\a\1\s\SNFC\DurableComponents\DurableHelper.cs:line 177
at SNFC.DurableHelper.CreateOrchestration(String functionName, String activity, ClaimPointer originalClaim, IDurableOrchestrationClient starter, String servicebusMessageId, String orchtestrationName) in D:\a\1\s\SNFC\DurableComponents\DurableHelper.cs:line 194
at SNFCDurableOnRamp.DurableOnRamp.OnRampTrigger(ServiceBusReceivedMessage[] queuedMessages, IDurableOrchestrationClient starter) in D:\a\1\s\SNFCDurableOnramp\DurableOnramp.cs:line 265

Funny thing is, if i have only 1 function on Netherite and the rest on Azure storage it works......

And i know the taskhub is picked up because i see the name in the storage account

well0549 · 2023-07-26T14:01:14Z

well0549
Jul 26, 2023
Author

Well tried everything but this error just keeps popping up. DO i need a different eventhub for every Function App?. (i have 15 in total)
I am sure the taskhub parameters are picked up because the tables are created. And dthe blob is also created with the correct name. But if i use the Netherite provider for more than 1 function app it all breaks down.

5 replies

davidmrdavid Aug 4, 2023
Maintainer

Hi @well0549, sorry for the delayed response, I'm just now catching this thread.
I'm trying to understand your setup a bit better, especially this piece:

But if i use the Netherite provider for more than 1 function app it all breaks down.

Are you saying that if you set up Netherite for 1 app it works for you? If it works for 1 app but when enabled on multiple, then it does sound like a potential configuration issue.

cc/ @sebastianburckhardt

davidmrdavid Aug 4, 2023
Maintainer

If you can provide me with your appname and the timeframe in UTC when it was using netherite, we may be able to look at our telemetry to understand what may have been happening.

well0549 Aug 5, 2023
Author

see below

well0549 Aug 5, 2023
Author

btw thanks for looking in to this.

sebastianburckhardt Nov 17, 2023
Maintainer

yes, each function app needs to use its own namespace, as documented here:
https://microsoft.github.io/durabletask-netherite/#/connectionnames

well0549 · 2023-07-27T06:26:52Z

well0549
Jul 27, 2023
Author

Giving up on Netherite

3 replies

well0549 Aug 5, 2023
Author

Well i did a test again this morning... Around UTC 2023-08-05T07:25:45Z

The setup is as follows,

I have several Function apps (13), each in it's own service plan.
The flow is as follows.

A function app gets data from somewhere, convert it to a claim check and writes it to an onramp queue
A function app "durableonramp" reads the queue and inspects the claim.
Based on the routing slip (found in the claim) it will write the claim to a new queue
A function app is listening to this queue and will process the message
When th eprocessing is done, the routing slip is adjusted
As long as there are routing itemsThe message is written to the onramp queue and we start again from step 2

If i switch onbly the durable onramp function app to netherite. it seems to work just fine
I did several small runs.

Then at UTC 2023-08-05T07:37:58Z I switched another function app to netherite. As soon as i drop new messages, everything stops working.
The first functionapp "durableonramp" seems to work but the second one "durablexls2csv" doesn't work

If i have only one function app switchet to netherite I see some warnings the log:

2023-08-05T07:45:10Z [Warning] Error response [fed6c7d5-5730-44cd-b440-5bb05353ea77] 409 There is already a lease present. (00.0s)

But if i switch the second one to Netherite I start seeing :

2023-08-05T07:45:34Z [Error] EventHubsProcessor partitions/9 received wrong packet, #3 instead of #0
2023-08-05T07:45:34Z [Error] EventHubsProcessor partitions/9(3) encountered an exception while processing packets : System.InvalidOperationException: EventHubs Out-Of-Order Packet
at DurableTask.Netherite.EventHubsTransport.BlobBatchReceiver1.ReceiveEventsAsync(Byte[] guid, IEnumerable1 hubMessages, CancellationToken token, Nullable1 nextPacketToReceive)+MoveNext() in /_/src/DurableTask.Netherite/TransportLayer/EventHubs/BlobBatchReceiver.cs:line 73 at DurableTask.Netherite.EventHubsTransport.BlobBatchReceiver1.ReceiveEventsAsync(Byte[] guid, IEnumerable1 hubMessages, CancellationToken token, Nullable1 nextPacketToReceive)+System.Threading.Tasks.Sources.IValueTaskSource<System.Boolean>.GetResult()
at DurableTask.Netherite.EventHubsTransport.EventHubsProcessor.Microsoft.Azure.EventHubs.Processor.IEventProcessor.ProcessEventsAsync(PartitionContext context, IEnumerable1 packets) in /_/src/DurableTask.Netherite/TransportLayer/EventHubs/EventHubsProcessor.cs:line 436 at DurableTask.Netherite.EventHubsTransport.EventHubsProcessor.Microsoft.Azure.EventHubs.Processor.IEventProcessor.ProcessEventsAsync(PartitionContext context, IEnumerable1 packets) in /_/src/DurableTask.Netherite/TransportLayer/EventHubs/EventHubsProcessor.cs:line 436
2023-08-05T07:45:34Z [Error] Part09 !!! Encountered exception while processing events in IEventProcessor.ProcessEventsAsync: System.InvalidOperationException: EventHubs Out-Of-Order Packet

Now as a kind of test I switched the durable onramp back to azure storage. And the xls2csv to netherite..
But i could not make it work again.....

Switched everything back to azure storage and everything works again.....

well0549 Aug 5, 2023
Author

I use the same eventhub for all my function apps (not sure if that was clear)

well0549 Aug 5, 2023
Author

I deleted the Event hub and started completely new.....

I created a new evenhubs namespace and first updated the xls2csv function app to use netherite storage.
Everything went fine.

I could see the requests going up in the eventhubs namespace.

Then i updated the durableonramp to use netherite. And submitted a batch of blobs.
immediately everything stopped working

i saw errrors like these:
2023-08-05T10:09:15Z [Error] EventHubsProcessor partitions/8(1) encountered an exception while processing packets : System.InvalidOperationException: EventHubs Out-Of-Order Packet
at DurableTask.Netherite.EventHubsTransport.BlobBatchReceiver1.ReceiveEventsAsync(Byte[] guid, IEnumerable1 hubMessages, CancellationToken token, Nullable1 nextPacketToReceive)+MoveNext() in /_/src/DurableTask.Netherite/TransportLayer/EventHubs/BlobBatchReceiver.cs:line 73 at DurableTask.Netherite.EventHubsTransport.BlobBatchReceiver1.ReceiveEventsAsync(Byte[] guid, IEnumerable1 hubMessages, CancellationToken token, Nullable1 nextPacketToReceive)+System.Threading.Tasks.Sources.IValueTaskSource<System.Boolean>.GetResult()
at DurableTask.Netherite.EventHubsTransport.EventHubsProcessor.Microsoft.Azure.EventHubs.Processor.IEventProcessor.ProcessEventsAsync(PartitionContext context, IEnumerable1 packets) in /_/src/DurableTask.Netherite/TransportLayer/EventHubs/EventHubsProcessor.cs:line 436 at DurableTask.Netherite.EventHubsTransport.EventHubsProcessor.Microsoft.Azure.EventHubs.Processor.IEventProcessor.ProcessEventsAsync(PartitionContext context, IEnumerable1 packets) in //src/DurableTask.Netherite/TransportLayer/EventHubs/EventHubsProcessor.cs:line 436
2023-08-05T10:09:15Z [Error] Part08 !!! Encountered exception while processing events in IEventProcessor.ProcessEventsAsync: System.InvalidOperationException: EventHubs Out-Of-Order Packet
at DurableTask.Netherite.EventHubsTransport.BlobBatchReceiver1.ReceiveEventsAsync(Byte[] guid, IEnumerable1 hubMessages, CancellationToken token, Nullable1 nextPacketToReceive)+MoveNext() in /_/src/DurableTask.Netherite/TransportLayer/EventHubs/BlobBatchReceiver.cs:line 73 at DurableTask.Netherite.EventHubsTransport.BlobBatchReceiver1.ReceiveEventsAsync(Byte[] guid, IEnumerable1 hubMessages, CancellationToken token, Nullable1 nextPacketToReceive)+System.Threading.Tasks.Sources.IValueTaskSource<System.Boolean>.GetResult()
at DurableTask.Netherite.EventHubsTransport.EventHubsProcessor.Microsoft.Azure.EventHubs.Processor.IEventProcessor.ProcessEventsAsync(PartitionContext context, IEnumerable1 packets) in /_/src/DurableTask.Netherite/TransportLayer/EventHubs/EventHubsProcessor.cs:line 436 at DurableTask.Netherite.EventHubsTransport.EventHubsProcessor.Microsoft.Azure.EventHubs.Processor.IEventProcessor.ProcessEventsAsync(PartitionContext context, IEnumerable1 packets) in //src/DurableTask.Netherite/TransportLayer/EventHubs/EventHubsProcessor.cs:line 436 terminatePartition=True

well0549 · 2023-08-10T13:32:48Z

well0549
Aug 10, 2023
Author

Could not find this button until now, i reopened the discussion.

0 replies

well0549 · 2023-08-15T07:30:21Z

well0549
Aug 15, 2023
Author

I think this is realted to : #283

However when i tried that it didn't create the consumer groups.

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

How to set up the eventhubs ? (can run only 1 function app at a time on netherite) #292

{{title}}

Replies: 8 comments 11 replies

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

How to set up the eventhubs ? (can run only 1 function app at a time on netherite) #292

well0549 Jul 25, 2023

Replies: 8 comments · 11 replies

well0549 Jul 25, 2023 Author

jbarntish Jul 25, 2023

well0549 Jul 25, 2023 Author

jbarntish Jul 25, 2023

well0549 Jul 25, 2023 Author

jbarntish Jul 25, 2023

well0549 Jul 25, 2023 Author

well0549 Jul 26, 2023 Author

davidmrdavid Aug 4, 2023 Maintainer

davidmrdavid Aug 4, 2023 Maintainer

well0549 Aug 5, 2023 Author

well0549 Aug 5, 2023 Author

sebastianburckhardt Nov 17, 2023 Maintainer

well0549 Jul 27, 2023 Author

well0549 Aug 5, 2023 Author

well0549 Aug 5, 2023 Author

well0549 Aug 5, 2023 Author

well0549 Aug 10, 2023 Author

well0549 Aug 15, 2023 Author

well0549
Jul 25, 2023

Replies: 8 comments 11 replies

well0549
Jul 25, 2023
Author

jbarntish
Jul 25, 2023

well0549
Jul 25, 2023
Author

well0549
Jul 25, 2023
Author

well0549 Jul 25, 2023
Author

well0549
Jul 26, 2023
Author

davidmrdavid Aug 4, 2023
Maintainer

davidmrdavid Aug 4, 2023
Maintainer

well0549 Aug 5, 2023
Author

well0549 Aug 5, 2023
Author

sebastianburckhardt Nov 17, 2023
Maintainer

well0549
Jul 27, 2023
Author

well0549 Aug 5, 2023
Author

well0549 Aug 5, 2023
Author

well0549 Aug 5, 2023
Author

well0549
Aug 10, 2023
Author

well0549
Aug 15, 2023
Author