[pack] Enable ordering of RpcLog and InvocationResponse messages

[brettsam]

Note -- I have a few things left to cleanup, but this is ready for initial review while I make sure all tests pass CI.

resolves #9238

This issue was introduced back when the GrpcChannel pipeline was refactored back in #8281. This change changed some behavior such that every message that came from the worker to the host via grpc was quickly queued on the ThreadPool for the fastest possible processing.

However, there are two message types that require ordering -- RpcLog and InvocationResponse:

• If these are out of order, you'll end up with out-of-order logs, which makes debugging and tracing difficult
• Once we receive an InvocationResponse, we consider that invocation complete and remove the tracking context we had been using. This means that if we process a log after this, we have no information that the RpcLog needs and we just drop it.

This change special-cases these "InvocationMessages" and puts them through a per-invocation dispatcher. Internally this sticks these messages into a Channel and processes them via a background Task. This processing ensures they are processed in order, at the expense of running as fast-as-possible on all Threads.

This is currently hidden behind both a stamp-level setting and a FeatureFlag. Once in, we can enable for entire stamps and gather perf numbers, etc, then slowly roll out once we're comfortable. Customers that want to try sooner can set the flag themselves for their app.

Once we're confident in the change, we'll remove this completely and everyone will use this new dispatcher method.

Pull request checklist

• My changes do not require documentation changes
  • Otherwise: Documentation issue linked to PR
• My changes should not be added to the release notes for the next release
  • Otherwise: I've added my notes to release_notes.md
• My changes do not need to be backported to a previous version
  • Otherwise: Backport tracked by issue/PR #issue_or_pr
• My changes do not require diagnostic events changes
  • Otherwise: I have added/updated all related diagnostic events and their documentation (Documentation issue linked to PR)
• I have added all required tests (Unit tests, E2E tests)

[brettsam]

/azp run

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[samir080890]

Hello Team,

Could you please confirm whether the GitHub issue mentioned below will be addressed as part of this ongoing resolution? If so, kindly provide an estimated time of arrival (ETA).

GitHub Issue: #9238

Thank you.