Deadlock when creating typed HttpClient with DI and HttpClientFactory #35986
Comments
|
Triage: This may not be HttpClientFactory specific, it might be a DI deadlock that HttpClientFactory is manifesting. (cc @davidfowl for any spicy thoughts). |
Dotnet-GitSync-Bot
added
area-System.Net.Http
untriaged
|
Tagging subscribers to this area: @dotnet/ncl |
analogrelay
added
area-Extensions-DependencyInjection
and removed
area-System.Net.Http
labels
|
OK I was able to reproduce the issue in isolation. This is an issue in the http client factory implementation and it's use of lazy. using System;
using System.Threading;
using System.Threading.Tasks;
using Microsoft.Extensions.DependencyInjection;
namespace ConsoleApp38
{
class Program
{
[ThreadStatic]
public static int ThreadId;
static async Task Main(string[] args)
{
// Thread 1: Thing1 (transient) -> Thing0 (singleton)
// Thread 2: Thing2 (singleton) -> Thing1 (transient) -> Thing0 (singleton)
// This reproduces the dead lock between the Lazy<T> and the IServiceProvider. Whenever singleton or scoped
// services are resolved a lock is taken (either globally for singletons or per scope for scoped).
// 1. Thread 1 resolves the Thing1 which is a transient service
// 2. In parallel, Thread 2 resolves Thing2 which is a singleton
// 3. Thread 1 enters the factory callback for Thing1 and takes the lazy lock
// 4. Thread 2 takes the singleton lock for the container when it resolves Thing2
// 5. Thread 2 enters the factory callback for Thing1 and waits on the lazy lock
// 6. Thread 1 calls GetRequiredService<Thing0> on the service provider, this waits for the singleton lock that Thread1 has
// Thread 1 has the lazy lock and is waiting on the singleton lock
// Thread 2 has the singleton lock an is waiting on the lazy
var services = new ServiceCollection();
IServiceProvider sp = null;
var lazy = new Lazy<Thing1>(() =>
{
// Tries to take the singleton lock (global)
var thing0 = sp.GetRequiredService<Thing0>();
return new Thing1(thing0);
});
services.AddSingleton<Thing0>();
services.AddTransient(sp =>
{
if (ThreadId == 1)
{
Thread.Sleep(1000);
}
else
{
// Let Thread 1 over take Thread 2
Thread.Sleep(3000);
}
return lazy.Value;
});
services.AddSingleton<Thing2>();
sp = services.BuildServiceProvider();
var t1 = Task.Run(() =>
{
ThreadId = 1;
using var scope1 = sp.CreateScope();
scope1.ServiceProvider.GetRequiredService<Thing1>();
});
var t2 = Task.Run(() =>
{
ThreadId = 2;
using var scope2 = sp.CreateScope();
scope2.ServiceProvider.GetRequiredService<Thing2>();
});
await t1;
await t2;
}
public class Thing2
{
public Thing2(Thing1 thing1)
{
}
}
public class Thing1
{
public Thing1(Thing0 thing2)
{
}
}
public class Thing0
{
public Thing0()
{
}
}
}
} |
|
@meareindel are you still getting the repro on your machine? could you then perhaps check if adding app.ApplicationServices.GetRequiredService<HttpMessageHandlerBuilder>();to the beginning of Configure call in Startup class perhaps allows for avoiding this deadlock? |
|
Moving the fix to 6.0, this is not a regression and I am assuming the above workaround in #35986 (comment) should enable us to avoid the deadlock explained in this issue. |
|
Note for our future selves: @davidfowl and I talked a bit about this offline, and we were thinking a per-service lock instead of a global (technically per-scope) lock is likely the solution to this problem. |
I've just moved from Singleton to Scoped lifetime to avoid, considering that there was no need in singleton lifetime at all. I could try the workaround above on original test example for clarity, but could you please elaborate how would it work? As far as I see, the DefaultHttpMessageHandlerBuilder got registered as transient in AddHttpClient extension of IHttpClientFactory and has no explicit singleton dependencies which could then be resolved and cached. |
ghost
added
the
in-pr
ghost
removed
the
in-pr
ghost
locked as resolved and limited conversation to collaborators
Describe the bug
We have a .net core web service with, for simplicity, two controllers, each with one route. One of the controllers, let's say 'scoped' controller, has typed http client injected directly. Other one, - 'singleton' controller, - has some singleton service injected, which, in its turn, has the same typed http client injected. The typed http client itself is registered with headers propagation - but it could be any additional .AddHttpMessageHandler(Func<IServiceProvider, DelegatingHandler>) registered which makes calls similar to IServiceProvider.GetRequiredService for any singleton dependency.
At first time after web service start when there is two (or more) concurrent requests on routes in both controllers, and route in 'scoped' controller starts processing just a bit earlier, than route in 'singleton' controller, there is a deadlock in DI.
To Reproduce
Take the solution from https://github.com/meareindel/DI-deadlock/tree/master and run the single test in it. It could be needed to modify the value of seconds to wait between requesting tasks depending on a machine configuration. The value also needs to be other for debugging (a little bit more, around 10ms on my machine, for example).
Expected behavior
No deadlock occurs, http client created successfully --OR-- validation error when injecting transient service into singleton.
Screenshots
If applicable, add screenshots to help explain your problem.
Additional context
Add any other context about the problem here.
Include the output of
dotnet --infostacks of route threads
'singleton' controller route:
'scoped' controller route:
dotnet --info:
Пакет SDK для .NET Core (отражающий любой global.json):
Version: 3.1.101
Commit: b377529961
Среда выполнения:
OS Name: Windows
OS Version: 10.0.18363
OS Platform: Windows
RID: win10-x64
Base Path: C:\Program Files\dotnet\sdk\3.1.101\
Host (useful for support):
Version: 3.1.1
Commit: a1388f194c
.NET Core SDKs installed:
2.1.600 [C:\Program Files\dotnet\sdk]
2.1.601 [C:\Program Files\dotnet\sdk]
2.1.602 [C:\Program Files\dotnet\sdk]
2.1.604 [C:\Program Files\dotnet\sdk]
2.1.700 [C:\Program Files\dotnet\sdk]
2.1.701 [C:\Program Files\dotnet\sdk]
2.1.801 [C:\Program Files\dotnet\sdk]
2.1.802 [C:\Program Files\dotnet\sdk]
2.2.105 [C:\Program Files\dotnet\sdk]
3.1.101 [C:\Program Files\dotnet\sdk]
.NET Core runtimes installed:
Microsoft.AspNetCore.All 2.1.8 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
Microsoft.AspNetCore.All 2.1.9 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
Microsoft.AspNetCore.All 2.1.11 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
Microsoft.AspNetCore.All 2.1.12 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
Microsoft.AspNetCore.All 2.1.13 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
Microsoft.AspNetCore.All 2.2.3 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
Microsoft.AspNetCore.All 2.2.8 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
Microsoft.AspNetCore.App 2.1.8 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 2.1.9 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 2.1.11 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 2.1.12 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 2.1.13 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 2.2.3 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 2.2.8 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 3.1.1 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.NETCore.App 2.1.8 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 2.1.9 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 2.1.11 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 2.1.12 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 2.1.13 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 2.2.3 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 2.2.8 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 3.1.1 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.WindowsDesktop.App 3.1.1 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
To install additional .NET Core runtimes or SDKs:
https://aka.ms/dotnet-download
The workaround is to register singleton service as scoped, of course. But I mainly expects some guidelines/documentation notes about such case. Is it a real bug to fix or is it unsupported servces scopes configuration - then the DI runtime validation could be expanded to such cases?
The text was updated successfully, but these errors were encountered: