http.sys accept loop - mitigate against break due to possible conflicting IO callbacks

[mgravell]

Context: #54251

System.InvalidOperationException: NativeOverlapped cannot be reused for multiple operations after .NET 6 to .NET 8 upgrade

This manifests as a fatal exception in the http.sys "accept" loop, which can either:

1. fault in purely managed code, stopping the message pump (MessagePump.ExecuteAsync)
2. fault during an IOCP callback, potentially killing the process

The underlying cause appears (hypothesis only) to be a duplicate callback, perhaps similar/related to the "large certificate" scenario (and mitigations) that impacted win8, and in particular this check somehow running both a sync and IOCP version of IOCompleted - this would break our expectations of a single caller into IOCompleted.

IOCompleted is intended to set the outcome of a value-task source; some unguarded SetResult/SetException (especially the final catch-all handler) mean that faults can indeed propagate, but this should not IMO be considered the main issue; rather, this is a symptom (albeit a fatal symptom) of being in an unexpected state, i.e. that we attempted to set the result while it wasn't actively pending.

To investigate and mitigate, this PR:

1. strengthens the SetResult / SetException aspects of IOCompleted, so that any fault becomes transient rather than fatal
2. adds logging of any such failure
3. adds tracking of what completions we're expecting at any point
4. logs any deviations from those expectations

This should a: help resolve the critical failures, and b: improve our understanding via logging if it does recur.

The use of PreAllocatedOverlapped on a reused AsyncAcceptContext means that we do not have the ability to change the IOCP callback state per call, which makes it impossible to do a reliable assert, but if we can prove the hypothesis, one additional mitigation option might be to remove the use of PreAllocatedOverlapped, using public System.Threading.NativeOverlapped* AllocateNativeOverlapped (System.Threading.IOCompletionCallback callback, object? state, object? pinData) per-accept rather than public System.Threading.NativeOverlapped* AllocateNativeOverlapped (System.Threading.PreAllocatedOverlapped preAllocated), with a different state per usage; this will have some additional overhead, but would allow stronger correctness guarantees. This would be an area to investigate if a: the logging shows that this is indeed what is happening, and b: we are unable to mitigate in other ways (similar to HttpSysListener.SkipIOCPCallbackOnSuccess).

Suggested additional testing here should include "large client certificates" (or just "large headers"?) to see if we can force the same failure mode, although this could be ambitious.

Historic context: prior to net6, there was a separate AsyncAcceptContext and TaskCompletionSource<> per accept; the use of TCS (with TrySet...) means any double-set would have not been observed, and the separate AsyncAcceptContext means that each IOCP callback could not ever get confused and try triggering the wrong state.

[mgravell]

/azp run

[mgravell]

(test failures are file copy problems)

[azure-pipelines]

Azure Pipelines successfully started running 3 pipeline(s).

[mgravell]

@halter73 @BrennanConroy @JamesNK @davidfowl @Tratcher could do with some eyeballs here (production fault)

[amcasey]

CI failure looks like #53294 (i.e. known)

[davidfowl]

This looks good.

[mgravell]

/backport to release/8.0

[github-actions]

Started backporting to release/8.0: https://github.com/dotnet/aspnetcore/actions/runs/8204527822

[github-actions]

@mgravell backporting to release/8.0 failed, the patch most likely resulted in conflicts:

$ git am --3way --ignore-whitespace --keep-non-patch changes.patch

Applying: investigate #54251 (more details will be in PR)
Using index info to reconstruct a base tree...
M	src/Servers/HttpSys/src/AsyncAcceptContext.cs
M	src/Servers/HttpSys/test/FunctionalTests/Listener/Utilities.cs
Falling back to patching base and 3-way merge...
Auto-merging src/Servers/HttpSys/test/FunctionalTests/Listener/Utilities.cs
Auto-merging src/Servers/HttpSys/src/AsyncAcceptContext.cs
CONFLICT (content): Merge conflict in src/Servers/HttpSys/src/AsyncAcceptContext.cs
error: Failed to merge in the changes.
hint: Use 'git am --show-current-patch=diff' to see the failed patch
Patch failed at 0001 investigate #54251 (more details will be in PR)
When you have resolved this problem, run "git am --continue".
If you prefer to skip this patch, run "git am --skip" instead.
To restore the original branch and stop patching, run "git am --abort".
Error: The process '/usr/bin/git' failed with exit code 128

Please backport manually!

[github-actions]

@mgravell an error occurred while backporting to release/8.0, please check the run log for details!

Error: git am failed, most likely due to a merge conflict.

[mgravell]

/azp run

[azure-pipelines]

Azure Pipelines successfully started running 3 pipeline(s).