Logs filled with critical exceptions from Kestrel/Quic #45105
Comments
|
@mrgleba Do you see this on any app or is there anything interesting about your code? |
|
@karelz Is there any guidance on which libmsquic versions are compatible with .NET 7 on Linux? We have a note in the docs for .NET 6 here: https://learn.microsoft.com/en-us/aspnet/core/fundamentals/servers/kestrel/http3?view=aspnetcore-7.0#linux |
|
@adityamandaleeka Nothing I can point out. Just upgraded to 7.0 and enabled http/3. |
|
I can confirm it's the same with 2.1.1 |
|
Thanks @mrgleba |
|
Same error using Windows 2022 |
|
I can find the very same message on Windows. It appears sometimes after the client has completed the request and the client process has completed. |
|
I might be missing something, but this looks very similar to dotnet/runtime#73688. |
|
@rzikm @CarnaViire Am I mistaken in thinking this should have been fixed by dotnet/runtime#75179? Edit: I see that we have a lot of our own locking code in this area, so there may be an additional race. |
|
yes, this looks similar to dotnet/runtime#73688. We should probably collect System.Net.Quic logs and possibly rotaries from MsQuic as well https://github.com/microsoft/msquic/blob/main/docs/Diagnostics.md Note that the fix was specifically around ODE. As far as I know MsQuic could return the Or somebody should come up with simple repro we can investigate. |
|
I have this test webapplication I employ in a repo: https://github.com/ladeak/Http3Repl/tree/master/tests/TestWebApplication There is really nothing special about it with the 3 endpoints. Invoke any with H3 request, wait a minute or two, and the error will show. I typically encounter the issue simply just using the MapGet for "/". @amcasey I will increase the log level to further investigate. |
|
How do you drive it? From HttpClient e.g C# & MsQuic? I'm wondering if the peer aborts the stream and then it is not open at the time Kestrel tries to do abort.... |
|
Yes, I use HttpClient (which I understand uses MsQuic). |
|
I wrote a trivial HttpClient driver and have yet to see a repro (though it seems like a race, so that's hardly definitive). Did you find anything interesting in the logs @ladeak? |
|
Nothing so far, but also something apparent on .net8 preview 2. |
|
I've tried upgrading libmsquic to 2.2.1 but the problem persists |
|
To investigate this, we need BTW, do you kill the client process immediately after sending the request? Do you dispose the client? If not, this could be related to dotnet/runtime#71927 (TL;DR: MsQuic is a user space library so when you close the client app and the process disappears from the memory, anything that was buffered but not yet send by MsQuic gets lost. In the linked issues, it was the last ACK that got lost and server waiting on it timed out). To confirm if this is it, you should dispose the |
|
Just to clarify. The problem occurs on a prodction server (the logs are from customers' connections). I'll see if I can set up the logs, but it won't be easy in production :( |
|
|
|
This is my client console app (I am using .NET 8 Preview 4 at this time): using var client = new HttpClient();
var request = new HttpRequestMessage(HttpMethod.Get, "https://localhost:5001/");
request.Version = HttpVersion.Version30;
request.VersionPolicy = HttpVersionPolicy.RequestVersionExact;
var response = await client.SendAsync(request);
response.EnsureSuccessStatusCode();
Console.WriteLine(await response.Content.ReadAsStringAsync());
await Task.Delay(10000);(Not clear if you prefer the timeout after - or before the dispose). I see this in the server logs, I was able to reproduce, but the errors typically come ~1 min after the requests: The server: var builder = WebApplication.CreateBuilder(args);
builder.WebHost.UseKestrel(kestrel =>
{
kestrel.ListenAnyIP(5001, options =>
{
options.UseHttps();
options.Protocols = HttpProtocols.Http1AndHttp2AndHttp3;
});
});
var app = builder. Build();
app.MapGet("/", async context =>
{
await context.Response.WriteAsync("Hello World " + context.Request.Protocol.ToString());
});I have collected the logs as linked using the netsh command. The file is 6.5MB and seems binary, how can I share it @ManickaP ?
|
|
I don't know the limits but you can attach it here, or somewhere else and share a link. Or you can send it to my email mapichov at microsoft dot com. The etl file can be opened with PerfView: https://github.com/microsoft/perfview |
|
I have sent it in an email. If it does not work, I'll share it through OneDrive. |
|
So I found 2 occurrences of races with connection closure and in some cases the connection is faster and calls abort: aspnetcore/src/Servers/Kestrel/Core/src/Internal/Http3/Http3Connection.cs Lines 499 to 506 in b441d4e @JamesNK Could you look into what I found out? If I'm right, this is not a problem of S.N.Quic. Details from msquic logs: |
amcasey
added
area-networking
|
This still occurs with net8 & libmsquic.2.2.4 😢 |
dotnet-policy-service
bot
added
the
pending-ci-rerun
wtgodbe
removed
the
pending-ci-rerun
dotnet-policy-service
bot
added
the
pending-ci-rerun
wtgodbe
removed
the
pending-ci-rerun
|
Hello everyone. Any updates on this issue? I have something similar as well. 2024-09-12 20:52:56.705 -07:00 [FTL] Unexpected exception in HttpConnection.ProcessRequestsAsync. |
|
Here's another place the stream could (race to?) be aborted It looks like it might have the wrong error though, so probably a red herring. |
|
Toy repro: launch server, launch client, wait ~20 seconds Server using Microsoft.AspNetCore.Server.Kestrel.Core;
var builder = WebApplication.CreateBuilder(args);
builder.WebHost.ConfigureKestrel(options =>
{
options.ListenLocalhost(5001, listenOptions =>
{
listenOptions.Protocols = HttpProtocols.Http3;
listenOptions.UseHttps();
});
options.Limits.KeepAliveTimeout = TimeSpan.FromSeconds(5);
});
var app = builder.Build();
app.MapGet("/", () => "Hello, world!");
app.Run();Client using var client = new HttpClient()
{
BaseAddress = new Uri("https://localhost:5001"),
DefaultRequestVersion = new Version(3, 0),
DefaultVersionPolicy = HttpVersionPolicy.RequestVersionExact,
};
try
{
var response = await client.GetAsync("/");
response.EnsureSuccessStatusCode();
var data = await response.Content.ReadAsStringAsync();
Console.WriteLine(data);
}
catch (HttpRequestException e)
{
Console.WriteLine($"Request error: {e.Message}");
} |
|
Oops, my repro is for a slightly different (but probably related?) error: This validation was added in #55282. Edit: filed #57933 for the new issue. |
Is there an existing issue for this?
Describe the bug
I run an app in a linux docker container (tag 7.0) with http/3 enabled.
The libmsquic version in 2.1.4 (from ms repo).
The server logs are being filled with the following message:
This doesn't influence the operations as far as we see. The server responds normally despite the messages.
Expected Behavior
No messages in logs
Steps To Reproduce
No response
Exceptions (if any)
No response
.NET Version
7.0 runtime (no sdk)
Anything else?
No response
The text was updated successfully, but these errors were encountered: