Microsoft.Extensions.Http.Resilience Breaks Application Insights Integration

[phil000]

Description

This may seem like an obscure one, but it's easily reproducible.

2 months ago we upgraded 2 solutions to Microsoft.Extensions.Http.Resilience 8.9.0. We didn't notice at the time but we have now noticed both projects have had all App Insights integration suppressed - all Traces, Requests and Dependencies stopped.

We have upgrade to Microsoft.Extensions.Http.Resilience 8.9.1 and the issue is reproducible in the local development environment.

NuGetSolver tool didn't really add much info.

Reproduction Steps

Add Microsoft.Extensions.Http.Resilience 8.9.1 to solution. App insights integration is disabled.
Remove Microsoft.Extensions.Http.Resilience 8.9.0 from solution App insights integration is working again.

Expected behavior

App insights integration is working

Actual behavior

App insights integration is disabled

Regression?

No

Known Workarounds

Remove Microsoft.Extensions.Http.Resilience

Configuration

.NET Core
Version: 8.0.8
Architecture: x64

Other information

No response

[RussKie]

2 months ago we upgraded 2 solutions to Microsoft.Extensions.Http.Resilience 8.9.0.

😲 8.9.0 was only released a week ago...

[RussKie]

Could you please provide a small stand-alone repro?

[phil000]

Sorry the original upgrade was to "Microsoft.Extensions.Http.Resilience" Version="8.6.0"
That had a bug where synchronous methods of HttpClient were not incorporated into retries so we waited for that fix (#5236).
We are now on version 8.9.1 and the issue of breaking App Insights that started for us with 8.6.0 still persists.
I will see if I can provide a small web project that exhibits the issue.

[phil000]

Can you provide an email address where I can send the repo?
It isn't something we can post publicly.

[RussKie]

You can submit your repro sample via the VS Feedback (an icon on the top RHS in VS or via via https://developercommunity.visualstudio.com/dotnet directly). There you'll be able to attach a zip archive and all the necessary repro instructions. Once submitted, please link the issue here so we can route it to our team.

Please note, that we're not able to debug customer apps (unless you have a service agreement, in which case you need to speak with your CSR), that's why we ask for small standalone repros.

[phil000]

The issue comes down to adding or removing this line, in a very minimal solution with only 3 NuGet package references.

With this AddResilienceHandler() all App Insights integration stops. Without it, the logging works as expected.

//TODO: Add or remove the 'AddResilienceHandler' call and it prevents App Insights integration from working.
bool wantAppInsightsWorking = false;
if (!wantAppInsightsWorking)
{
	builder.AddResilienceHandler(typeof(TClass).Name, builder =>
	{
		builder.AddRetry(new HttpRetryStrategyOptions
		{
			MaxRetryAttempts = 6,
			UseJitter = true,
			ShouldRetryAfterHeader = true,
			Delay = TimeSpan.FromMilliseconds(100),
			MaxDelay = TimeSpan.FromSeconds(10),
			BackoffType = DelayBackoffType.Exponential
		});
	});
}

<PackageReference Include="Microsoft.ApplicationInsights.AspNetCore" Version="2.22.0" />
<PackageReference Include="Microsoft.Extensions.Http.Resilience" Version="8.9.1" />
<PackageReference Include="Newtonsoft.Json" Version="13.0.3" />

[phil000]

Here's the minimalist repo.

Steps to Reproduce:

1. Add or remove the call to AddResilienceHandler found in WebsiteStartupBuilder.cs
2. Run the web app
3. Observe that App Insights logging is \ is not in the VS Output window. You'll see something like "Application Insights Telemetry (unconfigured):" when it's logging successfully, but Output window is completely empty when it is not working.

Repo.Web.zip

[RussKie]

It looks like this call somehow affects the AI registrations:

extensions/src/Libraries/Microsoft.Extensions.Http.Resilience/Resilience/ResilienceHttpClientBuilderExtensions.Resilience.cs

Line 118 in 39f5d27

_ = builder.Services.AddResiliencePipeline<HttpKey, HttpResponseMessage>(key, (builder, context) => configure(builder, new ResilienceHandlerContext(context)));

@martintmk @iliar-turdushev any thoughts?

[iliar-turdushev]

@phil000 Thank you for reporting this. We'll investigate and get back to you.

[iliar-turdushev]

The ticket is a duplicate of the following one #5222. The issue is caused by AppInsights. Here is detailed explanation of the issue #5222 (comment).

To fix the issue you have to reorder registration of AppInsights and Http.Resilience, in particular, first register AppInsights, and then Http.Resilience. For example, in the sample you provided you need to use the following order:

public void ConfigureServices(IServiceCollection services, IConfiguration configuration, bool isDevelopment)
{
    AddControllers(services);

    // First - AppInsights.
    AddTelemetry(services, isDevelopment);

    // Then - Http.Resilience and any other services that use Keyed Services feature of DI
    // https://learn.microsoft.com/en-us/aspnet/core/fundamentals/dependency-injection?view=aspnetcore-8.0#keyed-services.
    AddHttpClients(services);
}

@phil000 Please, confirm that the workaround works for you. If it does then we'll close the issue. Thank you.

[phil000]

Confirming that reordering the dependency registration solves the issue for us.
We have added a unit test to check for the presence of TelemetryClient in the collection after the service provider is built.
Please close the case.

[RussKie]

The AI bug is here: microsoft/ApplicationInsights-dotnet#2879

[RussKie]

Duplicate of #5222

[RussKie]

@iliar-turdushev could you please update the package's readme with this known issue?

[iliar-turdushev]

Sure, I will. Two customers already faced the issue.