Limit OpenSSL TLS Session cache for client SSL_CTX instances #110154

rzikm · 2024-11-25T15:32:51Z

This pull request limits the OpenSSL TLS Session cache for client SSL_CTX instances to 500, which should be enough for most use cases and should save some memory resources for clients.

Exceptional use cases (web scrapers?) can increase the cache size via AppCtx switch or environmental variable.

We should still investigate issues like #109600 to make sure we are not needlessly overpopulating the sessions cache.

rzikm · 2024-11-25T15:34:39Z

@wfurt we've talked about this recently, so I put it up lest I forget, but this change might not be as useful once we figure out the bug causing the "leak" in the linked issue.

wfurt · 2024-11-25T16:49:23Z

src/libraries/Common/src/Interop/Unix/System.Security.Cryptography.Native/Interop.OpenSsl.cs

        private const string TlsCacheSizeCtxName = "System.Net.Security.TlsCacheSize";
        private const string TlsCacheSizeEnvironmentVariable = "DOTNET_SYSTEM_NET_SECURITY_TLSCACHESIZE";
+        private const int DefaultTlsCacheSizeClient = 500; // since we keep only one TLS Session per hostname, 500 should be enough to cover most scenarios
+        private const int DefaultTlsCacheSizeServer = -1; // use implementation default


if we have two different values I'm wondering if we should also split the variables...? The 500 for client seems like good default to me.

I thought about that as well, but we either would have to

rename the existing configuration knob (which could be argued to be a breaking change)

introduce two more knobs to kontrol client/server separately

introduce one new with a server/client suffix and reuse the existing one for the other role

and neither of the above is a clear winner so I wanted to not complicate things needlessly.

I think I'm fine with keeping it as it is. In most cases the workloads would be separated anyway and the once where they don't may simply share the same value as they do today.

wfurt

LGTM

rzikm · 2025-01-17T14:08:04Z

Test failure possibly related

Process terminated. Assertion failed.
name != IntPtr.Zero
   at Interop.OpenSsl.RemoveSessionCallback(IntPtr ctx, IntPtr session) in /_/src/libraries/Common/src/Interop/Unix/System.Security.Cryptography.Native/Interop.OpenSsl.cs:line 829
   at Interop.Ssl.<SslRead>g____PInvoke|22_0(IntPtr __ssl_native, Byte* __buf_native, Int32 __num_native, SslErrorCode* __error_native)
   at Interop.Ssl.<SslRead>g____PInvoke|22_0(IntPtr __ssl_native, Byte* __buf_native, Int32 __num_native, SslErrorCode* __error_native)
   at Interop.Ssl.SslRead(SafeSslHandle ssl, Byte& buf, Int32 num, SslErrorCode& error) in /_/artifacts/obj/System.Net.Security/Debug/net10.0-linux/Microsoft.Interop.LibraryImportGenerator/Microsoft.Interop.LibraryImportGenerator/LibraryImports.g.cs:line 2713
   at Interop.OpenSsl.Decrypt(SafeSslHandle context, Span`1 buffer, SslErrorCode& errorCode) in /_/src/libraries/Common/src/Interop/Unix/System.Security.Cryptography.Native/Interop.OpenSsl.cs:line 644
   at System.Net.Security.SslStreamPal.DecryptMessage(SafeDeleteSslContext securityContext, Span`1 buffer, Int32& offset, Int32& count) in /_/src/libraries/System.Net.Security/src/System/Net/Security/SslStreamPal.Unix.cs:line 87
   at System.Net.Security.SslStream.Decrypt(Span`1 buffer, Int32& outputOffset, Int32& outputCount) in /_/src/libraries/System.Net.Security/src/System/Net/Security/SslStream.Protocol.cs:line 1021
   at System.Net.Security.SslStream.DecryptData(Int32 frameSize) in /_/src/libraries/System.Net.Security/src/System/Net/Security/SslStream.IO.cs:line 788
   at System.Net.Security.SslStream.ReadAsyncInternal[TIOAdapter](Memory`1 buffer, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Security/src/System/Net/Security/SslStream.IO.cs:line 870
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine) in /_/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncMethodBuilderCore.cs:line 38
   at System.Net.Security.SslStream.ReadAsyncInternal[TIOAdapter](Memory`1 buffer, CancellationToken cancellationToken)
   at System.Net.Security.SslStream.ReadAsync(Memory`1 buffer, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Security/src/System/Net/Security/SslStream.cs:line 891
   at System.Net.Http.HttpConnection.InitialFillAsync(Boolean async) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnection.cs:line 1628
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine) in /_/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncMethodBuilderCore.cs:line 38
   at System.Net.Http.HttpConnection.InitialFillAsync(Boolean async)
   at System.Net.Http.HttpConnection.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnection.cs:line 645
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine) in /_/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncMethodBuilderCore.cs:line 38
   at System.Net.Http.HttpConnection.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithNtProxyAuthAsync(HttpConnection connection, HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/ConnectionPool/HttpConnectionPool.cs:line 400
   at System.Net.Http.HttpConnectionPool.SendWithNtConnectionAuthAsync(HttpConnection connection, HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/ConnectionPool/HttpConnectionPool.cs:line 390
   at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/ConnectionPool/HttpConnectionPool.cs:line 497
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/ExecutionContext.cs:line 179
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread) in /_/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncTaskMethodBuilderT.cs:line 356
   at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/TaskContinuation.cs:line 795
   at System.Threading.Tasks.Task.RunContinuations(Object continuationObject) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/Task.cs:line 3478
   at System.Runtime.CompilerServices.AsyncValueTaskMethodBuilder`1.SetResult(TResult result) in /_/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncValueTaskMethodBuilderT.cs:line 47
   at System.Threading.Tasks.TaskCompletionSourceWithCancellation`1.WaitWithCancellationAsync(CancellationToken cancellationToken) in /_/src/libraries/Common/src/System/Threading/Tasks/TaskCompletionSourceWithCancellation.cs:line 23
   at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/ExecutionContext.cs:line 264
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread) in /_/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncTaskMethodBuilderT.cs:line 356
   at System.Threading.ThreadPoolWorkQueue.Dispatch() in /_/src/libraries/System.Private.CoreLib/src/System/Threading/ThreadPoolWorkQueue.cs:line 1118
   at System.Threading.PortableThreadPool.WorkerThread.WorkerThreadStart() in /_/src/libraries/System.Private.CoreLib/src/System/Threading/PortableThreadPool.WorkerThread.cs:line 127
   at System.Threading.Thread.StartCallback() in /_/src/coreclr/System.Private.CoreLib/src/System/Threading/Thread.CoreCLR.cs:line 104

rzikm · 2025-01-20T14:29:57Z

Only partially related. in TLS 1.3, OpenSSL may keep the session in the cache even if the callback rejects it.

https://github.com/openssl/openssl/blob/2478d3b7f5c4c2da9828e05308b34a4b078035f8/ssl/ssl_lib.c#L4521-L4555

Since we reject sessions without target hostname, then when the cached item is evicted from the internal cache, we hit the assert from previous comment. (and we hit it in this PR because we lowered the limit, so the eviction happens earlier). This does not cause any problems in the production because we still check against null later in the code.

The scenario which triggers this is when IP address is used instead of the target host. Then we would (wrongly) enable session tracking, but (correctly) still would not resume sessions, because we look for only non-ip target host names in the cache. I fixed this by not enabling session tracking in this uncommon scenario.

rzikm · 2025-01-20T15:45:26Z

/azp run runtime-libraries-coreclr outerloop

azure-pipelines · 2025-01-20T15:45:42Z

Azure Pipelines successfully started running 1 pipeline(s).

Limit OpenSSL TLS Session cache for client SSL_CTX instances.

deac098

dotnet-issue-labeler bot added the area-System.Net.Security label Nov 25, 2024

rzikm requested a review from wfurt November 25, 2024 15:33

dotnet-policy-service bot assigned rzikm Nov 25, 2024

wfurt reviewed Nov 25, 2024

View reviewed changes

build-analysis bot mentioned this pull request Nov 25, 2024

[wasm] Out of memory exception in System.Xml.XmlWriterApiTests.TCFullEndElement tests #110090

Open

karelz added this to the 10.0.0 milestone Dec 17, 2024

rzikm marked this pull request as draft December 17, 2024 12:51

dotnet-policy-service bot removed this from the 10.0.0 milestone Jan 16, 2025

dotnet-policy-service bot closed this Jan 16, 2025

rzikm reopened this Jan 16, 2025

wfurt approved these changes Jan 16, 2025

View reviewed changes

rzikm marked this pull request as ready for review January 17, 2025 11:42

Merge branch 'main' into constrain-client-sslcache

3c65c81

Don't enable TLS cache callbacks when using IP address as target host.

0fe34ea

rzikm merged commit ab75795 into dotnet:main Jan 21, 2025
79 of 90 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Limit OpenSSL TLS Session cache for client SSL_CTX instances #110154

Limit OpenSSL TLS Session cache for client SSL_CTX instances #110154

rzikm commented Nov 25, 2024

rzikm commented Nov 25, 2024

wfurt Nov 25, 2024

rzikm Nov 26, 2024

wfurt Jan 16, 2025

wfurt left a comment

rzikm commented Jan 17, 2025 •

edited

Loading

rzikm commented Jan 20, 2025

rzikm commented Jan 20, 2025

azure-pipelines bot commented Jan 20, 2025

Limit OpenSSL TLS Session cache for client SSL_CTX instances #110154

Limit OpenSSL TLS Session cache for client SSL_CTX instances #110154

Conversation

rzikm commented Nov 25, 2024

rzikm commented Nov 25, 2024

wfurt Nov 25, 2024

Choose a reason for hiding this comment

rzikm Nov 26, 2024

Choose a reason for hiding this comment

wfurt Jan 16, 2025

Choose a reason for hiding this comment

wfurt left a comment

Choose a reason for hiding this comment

rzikm commented Jan 17, 2025 • edited Loading

rzikm commented Jan 20, 2025

rzikm commented Jan 20, 2025

azure-pipelines bot commented Jan 20, 2025

rzikm commented Jan 17, 2025 •

edited

Loading