Keep orchestrator actions sorted, to avoid out-of-order events in OrchestrationRuntimeState

[sebastianburckhardt]

When debugging a failing test in Connor's integration tests Azure/azure-functions-durable-extension#1979 today I noticed an oddity: on completed OrchestrationWorkItems, any events sent by the orchestration appeared in reverse order in the runtime state. I tracked this down to TaskOrchestrationDispatcher, which returns the orchestrator actions as follows:

public IEnumerable<OrchestratorAction> OrchestratorActions => this.orchestratorActionsMap.Values;

but this is actually nondeterministic (and happened to be in reverse order for me) because the map is just a plain dictionary:

this.orchestratorActionsMap = new Dictionary<int, OrchestratorAction>();

I am a bit surprised that this has not been noticed before and has not caused more issues. The particular test that detected this was an entity test, which was immune to this problem while I used the AzureStorage backend (thanks to the MessageSorter) but as soon as I used Netherite the problem became apparent.

The fix is trivial, use a SortedDictionary instead.

[cgillum]

I'm not sure I understand the issue. I wouldn't expect the list of actions in a single execution step to require any particular ordering. Is this issue specific to entities?

[sebastianburckhardt]

It is definitely an issue for entities (in this case, on orchestrator sending signals to an entity), because our programming model guarantees in-order delivery of signals.

Whether or not it is an issue for orchestrators when no entities are involved is not super clear to me. It may not cause malfunctions we are aware of, but it is certainly surprising (e.g. two executions of a deterministic orchestrator on identical input could produce a different OrchestrationRuntimeState) and I would prefer that we keep things nicely ordered and deterministic. Especially since it doesn't cost anything.