Managed Identity Access Token is refreshed very often

[kevinharing]

Describe the bug

I recently started using managed identity authentication functionality from SqlClient after using our own access token caching implementation, but I'm seeing a lot of random token refreshes and I'm wondering if this is expected. Below is a screenshot from app insights showing the refresh calls over a period of 24 hours of one app instance.

Before:

After:

I would expect the token to be cached around 24 hours before a refresh is triggered if the token lifetime is 24 hours. What I am experiencing though is that it refreshes at a seemingly random interval.

This behavior is slowing down a bunch of our requests as you can probably imagine.

One other thing I noticed is calls to the below endpoints when the token is refreshed. We did not see these pop up before the change.

• GET crl3.digicert.com/Omniroot2025.crl
• GET mscrl.microsoft.com/pki/mscorp/crl/Microsoft%20RSA%20TLS%20CA%2001.crl

Currently I'm considering moving back to our own implementation, because this is just plain dumb behavior.

To reproduce

Just run a application with using the below segment in the connection string.

Authentication=Active Directory Managed Identity

Expected behavior

Refresh the token just before the token expires (5 minutes before or something).

Further technical details

Microsoft.Data.SqlClient version: 4.0.1
.NET target: 6.0.1
SQL Server version: Azure SQL Server
Operating system: aspnet:6.0-bullseye-slim

[JRahnama]

@kevinharing, a token should be refreshed in about 10 hours (5-10 minutes before expiration time) and it would be cached to make it easier to use. Is that happening sooner than that time for you?

Even if a token is acquired manually when it goes to the pool when it is re-used (after 10 hours) server checks credentials and it may throw token the expired exception.

[kevinharing]

@kevinharing, a token should be refreshed in about 10 hours (5-10 minutes before expiration time) and it would be cached to make it easier to use. Is that happening sooner than that time for you?

A token should be refreshed 5-10 minutes before expiration time indeed (not 10 hours specifically as the token lifetime is configurable in AAD). As can be seen on the screenshots, this is happening way more often than once every 10 hours.

Even if a token is acquired manually when it goes to the pool when it is re-used (after 10 hours) server checks credentials and it may throw token the expired exception.

What do you mean with "server checks credentials"? A token does not contain credentials.

[JRahnama]

What do you mean with "server checks credentials"? A token does not contain credentials.

A better term would be revalidation. We have a very good topic on this matter. It could be seen at #767

[siewers]

@kevinharing
We just did an investigation about a similar problem.
We are using the aspnet:6.0-alpine base image, but I believe the behavior is comparable.

From our testing, the issue seems to be caused by the fact our local user didn't have a home directory (created with adduser --disabled-password --no-create-home --system ...).
When the revocation lists are downloaded, they will be cached in the user home directory. If this directory is unavailable, it will just be ignored and the CRLs are re-download each time a certificate needs to be validated.
This is also clear from the comment here: https://github.com/dotnet/runtime/blob/main/src/libraries/System.Security.Cryptography/src/System/Security/Cryptography/X509Certificates/OpenSslCrlCache.cs#L229-L231

What we did was to simply remove the --no-create-home after which we only saw three requests to CRLs:

http://mscrl.microsoft.com/pki/mscorp/crl/Microsoft%20RSA%20TLS%20CA%2001.crl
http://crl3.digicert.com/Omniroot2025.crl
http://mscrl.microsoft.com/pki/mscorp/crl/Microsoft%20RSA%20TLS%20CA%2002.crl

Previously we saw thousands of such requests, which was slowing down a lot of our requests.

[scottaddie]

With the upgrade to Azure.Identity v1.8.0, this problem will be solved. When Authentication=Active Directory Managed Identity is used, the Azure Identity library's ManagedIdentityCredential type is used. As of v1.8.0, that credential type caches and refreshes tokens by default. See https://github.com/Azure/azure-sdk-for-net/blob/b9950ad3a3287c1a121f7c901072a8faddcd6a15/sdk/identity/Azure.Identity/src/Credentials/ManagedIdentityCredential.cs#L87-L89 for more details.

[ErikEJ]

@scottaddie Are you recommending an explicit Update today with Managed Identity and the current version of MDS?

[scottaddie]

@ErikEJ My recommendation is to use the v5.2.0 release of MDS when that becomes available.

[ErikEJ]

@kevinharing Have you tested with 5.2 preview 4 ?

[kevinharing]

@kevinharing Have you tested with 5.2 preview 4 ?

Nope, and to be frank, I currently don’t have time to test this out thoroughly.

[zheweiwangMicrosoft]

Any updates? I think it is still happening with MDS 5.2.0-preview5.24024.3.
#2343

[kf-gonzalez2]

Hi @zheweiwangMicrosoft this will be targeted to be addressed by PR #2380, and you can give it a try with the pipeline's artifacts.

[PaulVrugt]

@David-Engel @DavoudEshtehari did this fix make any release?

[ErikEJ]

@PaulVrugt It was added to 5.1.6 and 5.2.2 #2380