Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[HTTP/3] Test bug (flaky) - premature connection close - need some synchro on app level #55508

Closed
CarnaViire opened this issue Jul 12, 2021 · 8 comments · Fixed by #56134 or #57955
Closed

[HTTP/3] Test bug (flaky) - premature connection close - need some synchro on app level #55508

CarnaViire opened this issue Jul 12, 2021 · 8 comments · Fixed by #56134 or #57955
Assignees
@CarnaViire
Labels
area-System.Net.Quic disabled-test The test is disabled in source code against the issue test-run-core Test failures in .NET Core test runs
Milestone
7.0.0

Comments

@CarnaViire
Copy link
Member

I've seen it (rarely) happening on GetAsync_LargeHeader_Success, but it is most probably a generic HTTP/3 test issue (I suspect LoopbackServer)

System.Net.Http.Functional.Tests.SocketsHttpHandlerTest_HttpClientHandlerTest_Headers_Http3_MsQuic.GetAsync_LargeHeader_Success(headerName: "Content-Security-Policy", headerValueLength: 4618) [FAIL]
        System.Net.Http.HttpRequestException : An error occurred while sending the request.
        ---- System.Net.Quic.QuicOperationAbortedException : Operation aborted.
        Stack Trace:
          C:\Users\knatalia\dev\git\runtime\src\libraries\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\Http3RequestStream.cs(290,0): at System.Net.Http.Http3RequestStream.SendAsync(CancellationToken cancellationToken)
          C:\Users\knatalia\dev\git\runtime\src\libraries\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\Http3RequestStream.cs(298,0): at System.Net.Http.Http3RequestStream.SendAsync(CancellationToken cancellationToken)
          C:\Users\knatalia\dev\git\runtime\src\libraries\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\Http3Connection.cs(214,0): at System.Net.Http.Http3Connection.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
          C:\Users\knatalia\dev\git\runtime\src\libraries\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\HttpConnectionPool.cs(795,0): at System.Net.Http.HttpConnectionPool.TrySendUsingHttp3Async(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)      
          C:\Users\knatalia\dev\git\runtime\src\libraries\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\HttpConnectionPool.cs(871,0): at System.Net.Http.HttpConnectionPool.DetermineVersionAndSendAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
          C:\Users\knatalia\dev\git\runtime\src\libraries\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\HttpConnectionPool.cs(901,0): at System.Net.Http.HttpConnectionPool.SendAndProcessAltSvcAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)   
          C:\Users\knatalia\dev\git\runtime\src\libraries\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\HttpConnectionPool.cs(920,0): at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
          C:\Users\knatalia\dev\git\runtime\src\libraries\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\RedirectHandler.cs(30,0): at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
          C:\Users\knatalia\dev\git\runtime\src\libraries\System.Net.Http\src\System\Net\Http\HttpClient.cs(527,0): at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
          C:\Users\knatalia\dev\git\runtime\src\libraries\System.Net.Http\tests\FunctionalTests\HttpClientHandlerTest.Headers.cs(223,0): at System.Net.Http.Functional.Tests.HttpClientHandlerTest_Headers.<>c__DisplayClass8_0.<<GetAsync_LargeHeader_Success>b__1>d.MoveNext()
          --- End of stack trace from previous location ---
          C:\Users\knatalia\dev\git\runtime\src\libraries\Common\tests\System\Threading\Tasks\TaskTimeoutExtensions.cs(63,0): at System.Threading.Tasks.TaskTimeoutExtensions.WhenAllOrAnyFailed(Task[] tasks)
          C:\Users\knatalia\dev\git\runtime\src\libraries\Common\tests\System\Threading\Tasks\TaskTimeoutExtensions.cs(82,0): at System.Threading.Tasks.TaskTimeoutExtensions.WhenAllOrAnyFailed(Task[] tasks)
          C:\Users\knatalia\dev\git\runtime\src\libraries\Common\tests\System\Net\Http\GenericLoopbackServer.cs(38,0): at System.Net.Test.Common.LoopbackServerFactory.<>c__DisplayClass5_0.<<CreateClientAndServerAsync>b__0>d.MoveNext()
          --- End of stack trace from previous location ---
          C:\Users\knatalia\dev\git\runtime\src\libraries\Common\tests\System\Net\Http\Http3LoopbackServer.cs(100,0): at System.Net.Test.Common.Http3LoopbackServerFactory.CreateServerAsync(Func`3 funcAsync, Int32 millisecondsTimeout, GenericLoopbackOptions options)
          C:\Users\knatalia\dev\git\runtime\src\libraries\System.Net.Http\tests\FunctionalTests\HttpClientHandlerTest.Headers.cs(220,0): at System.Net.Http.Functional.Tests.HttpClientHandlerTest_Headers.GetAsync_LargeHeader_Success(String headerName, Int32 headerValueLength)
          --- End of stack trace from previous location ---
          ----- Inner Stack Trace -----
          C:\Users\knatalia\dev\git\runtime\src\libraries\System.Net.Quic\src\System\Net\Quic\Implementations\MsQuic\MsQuicStream.cs(1386,0): at System.Net.Quic.Implementations.MsQuic.MsQuicStream.HandleEventConnectionClose(State state)
          C:\Users\knatalia\dev\git\runtime\src\libraries\System.Net.Quic\src\System\Net\Quic\Implementations\MsQuic\MsQuicStream.cs(965,0): at System.Net.Quic.Implementations.MsQuic.MsQuicStream.HandleEventShutdownComplete(State state, StreamEvent& evt)
          C:\Users\knatalia\dev\git\runtime\src\libraries\System.Net.Quic\src\System\Net\Quic\Implementations\MsQuic\MsQuicStream.cs(813,0): at System.Net.Quic.Implementations.MsQuic.MsQuicStream.HandleEvent(State state, StreamEvent& evt)
          C:\Users\knatalia\dev\git\runtime\src\libraries\System.Net.Quic\src\System\Net\Quic\Implementations\MsQuic\MsQuicStream.cs(772,0): at System.Net.Quic.Implementations.MsQuic.MsQuicStream.NativeCallbackHandler(IntPtr stream, IntPtr context, StreamEvent& streamEvent)
          --- End of stack trace from previous location ---
          C:\Users\knatalia\dev\git\runtime\src\libraries\System.Net.Quic\src\System\Net\Quic\Implementations\MsQuic\Internal\ResettableCompletionSource.cs(57,0): at System.Net.Quic.Implementations.MsQuic.Internal.ResettableCompletionSource`1.GetResult(Int16 token)
          C:\Users\knatalia\dev\git\runtime\src\libraries\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\Http3RequestStream.cs(796,0): at System.Net.Http.Http3RequestStream.ReadHeadersAsync(Int64 headersLength, CancellationToken cancellationToken)
          C:\Users\knatalia\dev\git\runtime\src\libraries\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\Http3RequestStream.cs(322,0): at System.Net.Http.Http3RequestStream.ReadResponseAsync(CancellationToken cancellationToken)
          C:\Users\knatalia\dev\git\runtime\src\libraries\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\Http3RequestStream.cs(201,0): at System.Net.Http.Http3RequestStream.SendAsync(CancellationToken cancellationToken)

Callstack shows MsQuicStream.HandleEventConnectionClose which means connection was closed while stream was in flight reading the response.

This brings me to thing we've discussed today:

Important thing I've remembered: it is not enough to wait until SHUTDOWN_COMPLETE event on stream to be sure it is safe to close the connection: it means shutdown is acked on transport level, but on app level peer might still be in the middle of reading. Long time ago we had a huge email thread on this topic with Nick and aligned that such synchronization should be done on app level. That's why in RunClientServer test method there was additional synchronization done via semaphores.

This exact issue manifests itself in our HTTP/3 tests, so we need to add some kind of app-level synchro there (or think of another solution).
@dotnet-issue-labeler dotnet-issue-labeler bot added area-System.Net.Http untriaged New issue has not been triaged by the area owner labels Jul 12, 2021
@ghost
Copy link

ghost commented Jul 12, 2021

Tagging subscribers to this area: @dotnet/ncl
See info in area-owners.md if you want to be subscribed.

Issue Details

I've seen it (rarely) happening on GetAsync_LargeHeader_Success, but it is most probably a generic HTTP/3 test issue (I suspect LoopbackServer)

System.Net.Http.Functional.Tests.SocketsHttpHandlerTest_HttpClientHandlerTest_Headers_Http3_MsQuic.GetAsync_LargeHeader_Success(headerName: "Content-Security-Policy", headerValueLength: 4618) [FAIL]
        System.Net.Http.HttpRequestException : An error occurred while sending the request.
        ---- System.Net.Quic.QuicOperationAbortedException : Operation aborted.
        Stack Trace:
          C:\Users\knatalia\dev\git\runtime\src\libraries\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\Http3RequestStream.cs(290,0): at System.Net.Http.Http3RequestStream.SendAsync(CancellationToken cancellationToken)
          C:\Users\knatalia\dev\git\runtime\src\libraries\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\Http3RequestStream.cs(298,0): at System.Net.Http.Http3RequestStream.SendAsync(CancellationToken cancellationToken)
          C:\Users\knatalia\dev\git\runtime\src\libraries\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\Http3Connection.cs(214,0): at System.Net.Http.Http3Connection.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
          C:\Users\knatalia\dev\git\runtime\src\libraries\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\HttpConnectionPool.cs(795,0): at System.Net.Http.HttpConnectionPool.TrySendUsingHttp3Async(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)      
          C:\Users\knatalia\dev\git\runtime\src\libraries\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\HttpConnectionPool.cs(871,0): at System.Net.Http.HttpConnectionPool.DetermineVersionAndSendAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
          C:\Users\knatalia\dev\git\runtime\src\libraries\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\HttpConnectionPool.cs(901,0): at System.Net.Http.HttpConnectionPool.SendAndProcessAltSvcAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)   
          C:\Users\knatalia\dev\git\runtime\src\libraries\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\HttpConnectionPool.cs(920,0): at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
          C:\Users\knatalia\dev\git\runtime\src\libraries\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\RedirectHandler.cs(30,0): at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
          C:\Users\knatalia\dev\git\runtime\src\libraries\System.Net.Http\src\System\Net\Http\HttpClient.cs(527,0): at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
          C:\Users\knatalia\dev\git\runtime\src\libraries\System.Net.Http\tests\FunctionalTests\HttpClientHandlerTest.Headers.cs(223,0): at System.Net.Http.Functional.Tests.HttpClientHandlerTest_Headers.<>c__DisplayClass8_0.<<GetAsync_LargeHeader_Success>b__1>d.MoveNext()
          --- End of stack trace from previous location ---
          C:\Users\knatalia\dev\git\runtime\src\libraries\Common\tests\System\Threading\Tasks\TaskTimeoutExtensions.cs(63,0): at System.Threading.Tasks.TaskTimeoutExtensions.WhenAllOrAnyFailed(Task[] tasks)
          C:\Users\knatalia\dev\git\runtime\src\libraries\Common\tests\System\Threading\Tasks\TaskTimeoutExtensions.cs(82,0): at System.Threading.Tasks.TaskTimeoutExtensions.WhenAllOrAnyFailed(Task[] tasks)
          C:\Users\knatalia\dev\git\runtime\src\libraries\Common\tests\System\Net\Http\GenericLoopbackServer.cs(38,0): at System.Net.Test.Common.LoopbackServerFactory.<>c__DisplayClass5_0.<<CreateClientAndServerAsync>b__0>d.MoveNext()
          --- End of stack trace from previous location ---
          C:\Users\knatalia\dev\git\runtime\src\libraries\Common\tests\System\Net\Http\Http3LoopbackServer.cs(100,0): at System.Net.Test.Common.Http3LoopbackServerFactory.CreateServerAsync(Func`3 funcAsync, Int32 millisecondsTimeout, GenericLoopbackOptions options)
          C:\Users\knatalia\dev\git\runtime\src\libraries\System.Net.Http\tests\FunctionalTests\HttpClientHandlerTest.Headers.cs(220,0): at System.Net.Http.Functional.Tests.HttpClientHandlerTest_Headers.GetAsync_LargeHeader_Success(String headerName, Int32 headerValueLength)
          --- End of stack trace from previous location ---
          ----- Inner Stack Trace -----
          C:\Users\knatalia\dev\git\runtime\src\libraries\System.Net.Quic\src\System\Net\Quic\Implementations\MsQuic\MsQuicStream.cs(1386,0): at System.Net.Quic.Implementations.MsQuic.MsQuicStream.HandleEventConnectionClose(State state)
          C:\Users\knatalia\dev\git\runtime\src\libraries\System.Net.Quic\src\System\Net\Quic\Implementations\MsQuic\MsQuicStream.cs(965,0): at System.Net.Quic.Implementations.MsQuic.MsQuicStream.HandleEventShutdownComplete(State state, StreamEvent& evt)
          C:\Users\knatalia\dev\git\runtime\src\libraries\System.Net.Quic\src\System\Net\Quic\Implementations\MsQuic\MsQuicStream.cs(813,0): at System.Net.Quic.Implementations.MsQuic.MsQuicStream.HandleEvent(State state, StreamEvent& evt)
          C:\Users\knatalia\dev\git\runtime\src\libraries\System.Net.Quic\src\System\Net\Quic\Implementations\MsQuic\MsQuicStream.cs(772,0): at System.Net.Quic.Implementations.MsQuic.MsQuicStream.NativeCallbackHandler(IntPtr stream, IntPtr context, StreamEvent& streamEvent)
          --- End of stack trace from previous location ---
          C:\Users\knatalia\dev\git\runtime\src\libraries\System.Net.Quic\src\System\Net\Quic\Implementations\MsQuic\Internal\ResettableCompletionSource.cs(57,0): at System.Net.Quic.Implementations.MsQuic.Internal.ResettableCompletionSource`1.GetResult(Int16 token)
          C:\Users\knatalia\dev\git\runtime\src\libraries\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\Http3RequestStream.cs(796,0): at System.Net.Http.Http3RequestStream.ReadHeadersAsync(Int64 headersLength, CancellationToken cancellationToken)
          C:\Users\knatalia\dev\git\runtime\src\libraries\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\Http3RequestStream.cs(322,0): at System.Net.Http.Http3RequestStream.ReadResponseAsync(CancellationToken cancellationToken)
          C:\Users\knatalia\dev\git\runtime\src\libraries\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\Http3RequestStream.cs(201,0): at System.Net.Http.Http3RequestStream.SendAsync(CancellationToken cancellationToken)

Callstack shows MsQuicStream.HandleEventConnectionClose which means connection was closed while stream was in flight reading the response.

This brings me to thing we've discussed today:

Important thing I've remembered: it is not enough to wait until SHUTDOWN_COMPLETE event on stream to be sure it is safe to close the connection: it means shutdown is acked on transport level, but on app level peer might still be in the middle of reading. Long time ago we had a huge email thread on this topic with Nick and aligned that such synchronization should be done on app level. That's why in RunClientServer test method there was additional synchronization done via semaphores.

This exact issue manifests itself in our HTTP/3 tests, so we need to add some kind of app-level synchro there (or think of another solution).

Author: CarnaViire
Assignees: -
Labels:

area-System.Net.Http, untriaged
Milestone: -

@ghost
Copy link

ghost commented Jul 12, 2021

Tagging subscribers to this area: @dotnet/ncl
See info in area-owners.md if you want to be subscribed.

Issue Details

I've seen it (rarely) happening on GetAsync_LargeHeader_Success, but it is most probably a generic HTTP/3 test issue (I suspect LoopbackServer)

System.Net.Http.Functional.Tests.SocketsHttpHandlerTest_HttpClientHandlerTest_Headers_Http3_MsQuic.GetAsync_LargeHeader_Success(headerName: "Content-Security-Policy", headerValueLength: 4618) [FAIL]
        System.Net.Http.HttpRequestException : An error occurred while sending the request.
        ---- System.Net.Quic.QuicOperationAbortedException : Operation aborted.
        Stack Trace:
          C:\Users\knatalia\dev\git\runtime\src\libraries\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\Http3RequestStream.cs(290,0): at System.Net.Http.Http3RequestStream.SendAsync(CancellationToken cancellationToken)
          C:\Users\knatalia\dev\git\runtime\src\libraries\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\Http3RequestStream.cs(298,0): at System.Net.Http.Http3RequestStream.SendAsync(CancellationToken cancellationToken)
          C:\Users\knatalia\dev\git\runtime\src\libraries\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\Http3Connection.cs(214,0): at System.Net.Http.Http3Connection.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
          C:\Users\knatalia\dev\git\runtime\src\libraries\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\HttpConnectionPool.cs(795,0): at System.Net.Http.HttpConnectionPool.TrySendUsingHttp3Async(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)      
          C:\Users\knatalia\dev\git\runtime\src\libraries\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\HttpConnectionPool.cs(871,0): at System.Net.Http.HttpConnectionPool.DetermineVersionAndSendAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
          C:\Users\knatalia\dev\git\runtime\src\libraries\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\HttpConnectionPool.cs(901,0): at System.Net.Http.HttpConnectionPool.SendAndProcessAltSvcAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)   
          C:\Users\knatalia\dev\git\runtime\src\libraries\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\HttpConnectionPool.cs(920,0): at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
          C:\Users\knatalia\dev\git\runtime\src\libraries\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\RedirectHandler.cs(30,0): at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
          C:\Users\knatalia\dev\git\runtime\src\libraries\System.Net.Http\src\System\Net\Http\HttpClient.cs(527,0): at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
          C:\Users\knatalia\dev\git\runtime\src\libraries\System.Net.Http\tests\FunctionalTests\HttpClientHandlerTest.Headers.cs(223,0): at System.Net.Http.Functional.Tests.HttpClientHandlerTest_Headers.<>c__DisplayClass8_0.<<GetAsync_LargeHeader_Success>b__1>d.MoveNext()
          --- End of stack trace from previous location ---
          C:\Users\knatalia\dev\git\runtime\src\libraries\Common\tests\System\Threading\Tasks\TaskTimeoutExtensions.cs(63,0): at System.Threading.Tasks.TaskTimeoutExtensions.WhenAllOrAnyFailed(Task[] tasks)
          C:\Users\knatalia\dev\git\runtime\src\libraries\Common\tests\System\Threading\Tasks\TaskTimeoutExtensions.cs(82,0): at System.Threading.Tasks.TaskTimeoutExtensions.WhenAllOrAnyFailed(Task[] tasks)
          C:\Users\knatalia\dev\git\runtime\src\libraries\Common\tests\System\Net\Http\GenericLoopbackServer.cs(38,0): at System.Net.Test.Common.LoopbackServerFactory.<>c__DisplayClass5_0.<<CreateClientAndServerAsync>b__0>d.MoveNext()
          --- End of stack trace from previous location ---
          C:\Users\knatalia\dev\git\runtime\src\libraries\Common\tests\System\Net\Http\Http3LoopbackServer.cs(100,0): at System.Net.Test.Common.Http3LoopbackServerFactory.CreateServerAsync(Func`3 funcAsync, Int32 millisecondsTimeout, GenericLoopbackOptions options)
          C:\Users\knatalia\dev\git\runtime\src\libraries\System.Net.Http\tests\FunctionalTests\HttpClientHandlerTest.Headers.cs(220,0): at System.Net.Http.Functional.Tests.HttpClientHandlerTest_Headers.GetAsync_LargeHeader_Success(String headerName, Int32 headerValueLength)
          --- End of stack trace from previous location ---
          ----- Inner Stack Trace -----
          C:\Users\knatalia\dev\git\runtime\src\libraries\System.Net.Quic\src\System\Net\Quic\Implementations\MsQuic\MsQuicStream.cs(1386,0): at System.Net.Quic.Implementations.MsQuic.MsQuicStream.HandleEventConnectionClose(State state)
          C:\Users\knatalia\dev\git\runtime\src\libraries\System.Net.Quic\src\System\Net\Quic\Implementations\MsQuic\MsQuicStream.cs(965,0): at System.Net.Quic.Implementations.MsQuic.MsQuicStream.HandleEventShutdownComplete(State state, StreamEvent& evt)
          C:\Users\knatalia\dev\git\runtime\src\libraries\System.Net.Quic\src\System\Net\Quic\Implementations\MsQuic\MsQuicStream.cs(813,0): at System.Net.Quic.Implementations.MsQuic.MsQuicStream.HandleEvent(State state, StreamEvent& evt)
          C:\Users\knatalia\dev\git\runtime\src\libraries\System.Net.Quic\src\System\Net\Quic\Implementations\MsQuic\MsQuicStream.cs(772,0): at System.Net.Quic.Implementations.MsQuic.MsQuicStream.NativeCallbackHandler(IntPtr stream, IntPtr context, StreamEvent& streamEvent)
          --- End of stack trace from previous location ---
          C:\Users\knatalia\dev\git\runtime\src\libraries\System.Net.Quic\src\System\Net\Quic\Implementations\MsQuic\Internal\ResettableCompletionSource.cs(57,0): at System.Net.Quic.Implementations.MsQuic.Internal.ResettableCompletionSource`1.GetResult(Int16 token)
          C:\Users\knatalia\dev\git\runtime\src\libraries\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\Http3RequestStream.cs(796,0): at System.Net.Http.Http3RequestStream.ReadHeadersAsync(Int64 headersLength, CancellationToken cancellationToken)
          C:\Users\knatalia\dev\git\runtime\src\libraries\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\Http3RequestStream.cs(322,0): at System.Net.Http.Http3RequestStream.ReadResponseAsync(CancellationToken cancellationToken)
          C:\Users\knatalia\dev\git\runtime\src\libraries\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\Http3RequestStream.cs(201,0): at System.Net.Http.Http3RequestStream.SendAsync(CancellationToken cancellationToken)

Callstack shows MsQuicStream.HandleEventConnectionClose which means connection was closed while stream was in flight reading the response.

This brings me to thing we've discussed today:

Important thing I've remembered: it is not enough to wait until SHUTDOWN_COMPLETE event on stream to be sure it is safe to close the connection: it means shutdown is acked on transport level, but on app level peer might still be in the middle of reading. Long time ago we had a huge email thread on this topic with Nick and aligned that such synchronization should be done on app level. That's why in RunClientServer test method there was additional synchronization done via semaphores.

This exact issue manifests itself in our HTTP/3 tests, so we need to add some kind of app-level synchro there (or think of another solution).

Author: CarnaViire
Assignees: -
Labels:

area-System.Net.Http, area-System.Net.Quic, untriaged
Milestone: -

@CarnaViire CarnaViire mentioned this issue Jul 15, 2021
Merged
@ManickaP ManickaP removed the untriaged New issue has not been triaged by the area owner label Jul 15, 2021
@ManickaP ManickaP added this to the 6.0.0 milestone Jul 15, 2021
@karelz karelz added the test-run-core Test failures in .NET Core test runs label Jul 20, 2021
@karelz
Copy link
Member

karelz commented Jul 20, 2021

Failures 5/20-7/20 (incl. PRs):

Day Failures
7/12 3x PR
7/13 6x PR
7/14 3x PR
7/15 1x PR
7/20 1x PR

@geoffkizer
Copy link
Contributor

I suspect this is related to #55025

The loopback server doesn't ever close the connection today. But it could be getting finalized, which seems like it could manifest as above.

If the loopback server waited for graceful shutdown, it would solve that.

@CarnaViire CarnaViire added the disabled-test The test is disabled in source code against the issue label Jul 21, 2021
@geoffkizer
Copy link
Contributor

I tested with my loopback fix and it seems to fix this issue.

@wfurt
Copy link
Member

wfurt commented Jul 24, 2021

Let's get it in.

@geoffkizer geoffkizer mentioned this issue Jul 24, 2021
Merged
@ghost ghost added the in-pr There is an active PR which will close this issue when it is merged label Jul 24, 2021
@ghost ghost removed the in-pr There is an active PR which will close this issue when it is merged label Jul 24, 2021
@geoffkizer geoffkizer reopened this Jul 24, 2021
@geoffkizer
Copy link
Contributor

Reopening because we still need to re-enable the GetAsync_LargeHeader_Success test. I believe the underlying issue is fixed now, but we should verify that we don't see failures anymore here before re-enabling the test.

@karelz karelz mentioned this issue Jul 26, 2021
Closed
@ManickaP
Copy link
Member

Triage: probably fixed, re-enable, check if it reproduces or not. Either close or punt as a result.

@ManickaP ManickaP modified the milestones: 6.0.0, Future Aug 12, 2021
@CarnaViire CarnaViire self-assigned this Aug 20, 2021
@CarnaViire CarnaViire mentioned this issue Aug 23, 2021
Merged
@ghost ghost added the in-pr There is an active PR which will close this issue when it is merged label Aug 23, 2021
CarnaViire added a commit that referenced this issue Aug 23, 2021
@CarnaViire
Re-enable GetAsync_LargeHeader_Success for H/3 (#57955)
ed32e0d 
Fixes #55508
@ghost ghost removed the in-pr There is an active PR which will close this issue when it is merged label Aug 23, 2021
@karelz karelz modified the milestones: Future, 7.0.0 Aug 26, 2021
ManickaP pushed a commit that referenced this issue Sep 2, 2021
@CarnaViire @ManickaP
Re-enable GetAsync_LargeHeader_Success for H/3 (#57955)
5b8bf67 
Fixes #55508
@ghost ghost locked as resolved and limited conversation to collaborators Sep 25, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@CarnaViire CarnaViire

Labels
area-System.Net.Quic disabled-test The test is disabled in source code against the issue test-run-core Test failures in .NET Core test runs
Projects
None yet
Milestone
7.0.0
5 participants
@CarnaViire @karelz @ManickaP @geoffkizer @wfurt