Fix flakey HTTP/2 tests

[JamesNK]

Addresses #39520

[JamesNK]

Might also address #39477 & #39479 & #39492

These all started failing at the same time.

[JamesNK]

Could I get a review? 🙏 I'd like to merge this to stop any more flakey tests from popping up from this issue.

[BrennanConroy]

Could you explain how this fixes the issue?

[JamesNK]

I talked with @halter73 when he made this change. I believe awaiting ThreadPoolAwaitable.Instance before sending preface and settings means the test loses the SynchronizationContext so the Http2Connection request processing runs inline.

The problem that this PR fixes is ThreadPoolAwaitable.Instance was nested in a method. When returning from that method the context is re-acquired before SendPreface/SendSettings. Moving ThreadPoolAwaitable.Instance so it is in a method before SendPreface & SendSettings fixes this.

Correct me if I'm wrong @halter73 😄

[halter73]

Awaiting ThreadPoolAwaitable.Instance should lose the current SynchronizationContext even for callers several async methods up stack.

It's different than ConfigureAwait(false) because it's not just returning a custom awaitable that doesn't restore the sync context. It's also posting the continuation using ThreadPool.UnsafeQueueUserWorkItem which loses the entire execution context. Up stack callers may still try to restore the sync context from the execution context, but there shouldn't be any sync context to restore.

[JamesNK]

This PR definitely fixes the flakey test. Before 5000 executions would result in 30ish failures. After 5000 executions all pass.

Not having ThreadPoolAwaitable.Instance in a nested method has definitely had an impact.

[halter73]

I was wrong with my last comment. Task captures the ExecutionContext when it's constructed so it uses that ExecutionContext rather than the ExecutionContext of the thread it's completed on (thankfully thinking about this more).

This makes me want to return a ConfiguredTaskAwaitable from InitializeConnectionAsync so the entire tests run without the sync context like I thought was happening before, but it doesn't have to be this PR. E.g.

protected ConfiguredTaskAwaitable InitializeConnectionAsync(RequestDelegate application, int expectedSettingsCount = 3, bool expectedWindowUpdate = true)
{
    return InitializeConnectionInnerAsync(application, expectedSettingsCount, expectedWindowUpdate).DefaultTimeout().ConfigureAwait(false);
}