[BUG] ManagedIdentityCredential throws OperationCanceledException when using SqlClient

[Rookian]

Library name and version

Azure.Identity 1.6.0, Microsoft.Data.SqlClient 4.1.0

Describe the bug

We receive almost everyday SqlExceptions with a inner OperationCanceledException.

{"level":0,"method":"Azure.Core.CancellationHelper.ThrowOperationCanceledException","line":0,"assembly":"Azure.Core, Version=1.25.0.0, Culture=neutral, PublicKeyToken=92742159e12e44c8"}
{"level":1,"method":"Azure.Core.CancellationHelper.ThrowIfCancellationRequested","line":0,"assembly":"Azure.Core, Version=1.25.0.0, Culture=neutral, PublicKeyToken=92742159e12e44c8"}
{"level":2,"method":"Azure.Core.Pipeline.ResponseBodyPolicy.ThrowIfCancellationRequestedOrTimeout","line":0,"assembly":"Azure.Core, Version=1.25.0.0, Culture=neutral, PublicKeyToken=92742159e12e44c8"}
{"level":3,"method":"Azure.Core.Pipeline.ResponseBodyPolicy+<ProcessAsync>d__5.MoveNext","line":0,"assembly":"Azure.Core, Version=1.25.0.0, Culture=neutral, PublicKeyToken=92742159e12e44c8"}
{"level":4,"method":"System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw","line":0,"assembly":"System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e"}
{"level":5,"method":"System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess","line":0,"assembly":"System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e"}
{"level":6,"method":"System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification","line":0,"assembly":"System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e"}
{"level":7,"method":"Azure.Core.Pipeline.RedirectPolicy+<ProcessAsync>d__5.MoveNext","line":0,"assembly":"Azure.Core, Version=1.25.0.0, Culture=neutral, PublicKeyToken=92742159e12e44c8"}
{"level":8,"method":"System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw","line":0,"assembly":"System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e"}
{"level":9,"method":"System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess","line":0,"assembly":"System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e"}
{"level":10,"method":"System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification","line":0,"assembly":"System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e"}
{"level":11,"method":"Azure.Core.Pipeline.RetryPolicy+<ProcessAsync>d__11.MoveNext","line":0,"assembly":"Azure.Core, Version=1.25.0.0, Culture=neutral, PublicKeyToken=92742159e12e44c8"}
{"level":12,"method":"System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw","line":0,"assembly":"System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e"}
{"level":13,"method":"Azure.Core.Pipeline.RetryPolicy+<ProcessAsync>d__11.MoveNext","line":0,"assembly":"Azure.Core, Version=1.25.0.0, Culture=neutral, PublicKeyToken=92742159e12e44c8"}
{"level":14,"method":"System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw","line":0,"assembly":"System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e"}
{"level":15,"method":"System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess","line":0,"assembly":"System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e"}
{"level":16,"method":"System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification","line":0,"assembly":"System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e"}
{"level":17,"method":"Azure.Identity.ManagedIdentitySource+<AuthenticateAsync>d__10.MoveNext","line":0,"assembly":"Azure.Identity, Version=1.6.0.0, Culture=neutral, PublicKeyToken=92742159e12e44c8"}
{"level":18,"method":"System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw","line":0,"assembly":"System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e"}
{"level":19,"method":"System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess","line":0,"assembly":"System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e"}
{"level":20,"method":"System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification","line":0,"assembly":"System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e"}
{"level":21,"method":"Azure.Identity.ManagedIdentityClient+<AuthenticateAsync>d__12.MoveNext","line":0,"assembly":"Azure.Identity, Version=1.6.0.0, Culture=neutral, PublicKeyToken=92742159e12e44c8"}
{"level":22,"method":"System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw","line":0,"assembly":"System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e"}
{"level":23,"method":"System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess","line":0,"assembly":"System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e"}
{"level":24,"method":"System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification","line":0,"assembly":"System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e"}
{"level":25,"method":"Azure.Identity.ManagedIdentityCredential+<GetTokenImplAsync>d__14.MoveNext","line":0,"assembly":"Azure.Identity, Version=1.6.0.0, Culture=neutral, PublicKeyToken=92742159e12e44c8"}
{"level":26,"method":"System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw","line":0,"assembly":"System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e"}
{"level":27,"method":"Azure.Identity.CredentialDiagnosticScope.FailWrapAndThrow","line":0,"assembly":"Azure.Identity, Version=1.6.0.0, Culture=neutral, PublicKeyToken=92742159e12e44c8"}
{"level":28,"method":"Azure.Identity.ManagedIdentityCredential+<GetTokenImplAsync>d__14.MoveNext","line":0,"assembly":"Azure.Identity, Version=1.6.0.0, Culture=neutral, PublicKeyToken=92742159e12e44c8"}
{"level":29,"method":"System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw","line":0,"assembly":"System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e"}
{"level":30,"method":"System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess","line":0,"assembly":"System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e"}
{"level":31,"method":"System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification","line":0,"assembly":"System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e"}
{"level":32,"method":"Azure.Identity.ManagedIdentityCredential+<GetTokenAsync>d__12.MoveNext","line":0,"assembly":"Azure.Identity, Version=1.6.0.0, Culture=neutral, PublicKeyToken=92742159e12e44c8"}
{"level":33,"method":"System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw","line":0,"assembly":"System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e"}
{"level":34,"method":"System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess","line":0,"assembly":"System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e"}
{"level":35,"method":"Azure.Identity.DefaultAzureCredential+<GetTokenFromSourcesAsync>d__15.MoveNext","line":0,"assembly":"Azure.Identity, Version=1.6.0.0, Culture=neutral, PublicKeyToken=92742159e12e44c8"}
{"level":36,"method":"System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw","line":0,"assembly":"System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e"}
{"level":37,"method":"System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess","line":0,"assembly":"System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e"}
{"level":38,"method":"System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification","line":0,"assembly":"System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e"}
{"level":39,"method":"Azure.Identity.DefaultAzureCredential+<GetTokenImplAsync>d__13.MoveNext","line":0,"assembly":"Azure.Identity, Version=1.6.0.0, Culture=neutral, PublicKeyToken=92742159e12e44c8"}
{"level":40,"method":"System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw","line":0,"assembly":"System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e"}
{"level":41,"method":"Azure.Identity.CredentialDiagnosticScope.FailWrapAndThrow","line":0,"assembly":"Azure.Identity, Version=1.6.0.0, Culture=neutral, PublicKeyToken=92742159e12e44c8"}
{"level":42,"method":"Azure.Identity.DefaultAzureCredential+<GetTokenImplAsync>d__13.MoveNext","line":0,"assembly":"Azure.Identity, Version=1.6.0.0, Culture=neutral, PublicKeyToken=92742159e12e44c8"}
{"level":43,"method":"System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw","line":0,"assembly":"System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e"}
{"level":44,"method":"System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess","line":0,"assembly":"System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e"}
{"level":45,"method":"System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification","line":0,"assembly":"System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e"}
{"level":46,"method":"Azure.Identity.DefaultAzureCredential+<GetTokenAsync>d__12.MoveNext","line":0,"assembly":"Azure.Identity, Version=1.6.0.0, Culture=neutral, PublicKeyToken=92742159e12e44c8"}
{"level":47,"method":"System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw","line":0,"assembly":"System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e"}
{"level":48,"method":"System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess","line":0,"assembly":"System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e"}
{"level":49,"method":"System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification","line":0,"assembly":"System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e"}
{"level":50,"method":"Microsoft.Data.SqlClient.ActiveDirectoryAuthenticationProvider+<AcquireTokenAsync>d__17.MoveNext","line":0,"assembly":"Microsoft.Data.SqlClient, Version=4.1.0.0, Culture=neutral, PublicKeyToken=23ec7fc2d6eaa4a5"}
{"level":51,"method":"System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw","line":0,"assembly":"System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e"}
{"level":52,"method":"System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess","line":0,"assembly":"System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e"}
{"level":53,"method":"System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification","line":0,"assembly":"System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e"}
{"level":54,"method":"Microsoft.Data.SqlClient.SqlInternalConnectionTds+<>c__DisplayClass146_1+<<GetFedAuthToken>b__1>d.MoveNext","line":0,"assembly":"Microsoft.Data.SqlClient, Version=4.1.0.0, Culture=neutral, PublicKeyToken=23ec7fc2d6eaa4a5"}
{"level":55,"method":"System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw","line":0,"assembly":"System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e"}
{"level":56,"method":"System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess","line":0,"assembly":"System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e"}
{"level":57,"method":"System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification","line":0,"assembly":"System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e"}
{"level":58,"method":"Microsoft.Data.SqlClient.SqlInternalConnectionTds.GetFedAuthToken","line":0,"assembly":"Microsoft.Data.SqlClient, Version=4.1.0.0, Culture=neutral, PublicKeyToken=23ec7fc2d6eaa4a5"}

Expected behavior

Clients might need to be able to increase the timeout or Microsoft should fix the endpoint that is responsible for providing an authentication token for ManagedIdentityCredential.

Actual behavior

It seems like SqlClient tries to get a token from ManagedIdentityCredential, but fails due to a timeout.

Anything we can do here?

Reproduction Steps

Environment

Azure AppService (Linux)
.NET 6.0

[jsquire]

//cc: @christothes

[jsquire]

Thank you for your feedback. Tagging and routing to the team member best able to assist.

[christothes]

Hi @Rookian - Could you clarify whether or not this is occurring when you are not intending to use the ManagedIdentityCredential? The timeout that occurs when Azure.Identity attempts to determine if an endpoint is available is expected and a necessary part of the discovery process. These should be first chance exceptions, however, so I'm curious why you are seeing them in logging.

I don't believe the SQLClient exposes a way to configure the options for the DefaultAzureCredential, which would allow you to set the ExcludeManagedIdentityCredential option.

[Rookian]

@christothes we use Authentication=Active Directory Default; in our SQL connection string in an AppService that has a Managed Identity assigned.

[christothes]

Unfortunately, your options are limited for diagnosing this since SQLClient wraps Azure.Identity in a way that does not expose the options to enable logging or configure timeouts. You may want to raise this issue there to get their take on it.

[ghost]

Hi, we're sending this friendly reminder because we haven't heard back from you in 7 days. We need more information about this issue to help address it. Please be sure to give us your input. If we don't hear back from you within 14 days of this comment the issue will be automatically closed. Thank you!

[Rookian]

I opened a new issue dotnet/SqlClient#1946