Router - Document DotNet High CPU Usage

[huntharo]

Motivations

• This high CPU usage observed with the default thread config in DotNet is likely a problem others are encountering and may not be aware of
• There likely needs to be an improvement made to the DotNet work stealing algorithm to reduce stealing when there is not enough work to go around to all the threads and/or there needs to be a tuning of how many threads to create

Video Review of Performance Improvement - YouTube

Click to view on YouTube

Test Setup

• Rust Extension with Current Thread Runtime (default, but specified here)
  • Command line: LAMBDA_DISPATCH_RUNTIME=current_thread LAMBDA_DISPATCH_FORCE_DEADLINE=60 AWS_LAMBDA_FUNCTION_VERSION=\$LATEST AWS_LAMBDA_FUNCTION_MEMORY_SIZE=512 AWS_LAMBDA_FUNCTION_NAME=dogs AWS_LAMBDA_RUNTIME_API=localhost:5051 AWS_REGION=us-east-2 AWS_ACCESS_KEY_ID=test-access-key-id AWS_SECRET_ACCESS_KEY=test-secret-access-key AWS_SESSION_TOKEN=test-session-token cargo run --release --bin extension 2>&1 | tee extension.log
• Router setup with 10 concurrent requests
  • Rough command line: DOTNET_ThreadPool_UnfairSemaphoreSpinLimit=5 LAMBDA_DISPATCH_ChannelCount=10 LAMBDA_DISPATCH_MaxConcurrentCount=10 LAMBDA_DISPATCH_AllowInsecureControlChannel=true LAMBDA_DISPATCH_PreferredControlChannelScheme=http LAMBDA_DISPATCH_FunctionName=dogs AWS_LAMBDA_SERVICE_URL=http://localhost:5051 AWS_REGION=us-east-2 AWS_ACCESS_KEY_ID=test-access-key-id AWS_SECRET_ACCESS_KEY=test-secret-access-key AWS_SESSION_TOKEN=test-session-token src/PwrDrvr.LambdaDispatch.Router/bin/Release/net8.0/PwrDrvr.LambdaDispatch.Router 2>&1 | tee router.log
• 10 concurrent connections from oha
  • oha -c 10 -z 60s http://127.0.0.1:5001/ping

Test Cases

Router with Default ThreadPool Threads, DOTNET_ThreadPool_UnfairSemaphoreSpinLimit not Specified

• 17,000 RPS
• ❌ ❌ 700% Router CPU
• 94% Extension CPU
• Router command line: LAMBDA_DISPATCH_ChannelCount=10 LAMBDA_DISPATCH_MaxConcurrentCount=10 LAMBDA_DISPATCH_AllowInsecureControlChannel=true LAMBDA_DISPATCH_PreferredControlChannelScheme=http LAMBDA_DISPATCH_FunctionName=dogs AWS_LAMBDA_SERVICE_URL=http://localhost:5051 AWS_REGION=us-east-2 AWS_ACCESS_KEY_ID=test-access-key-id AWS_SECRET_ACCESS_KEY=test-secret-access-key AWS_SESSION_TOKEN=test-session-token src/PwrDrvr.LambdaDispatch.Router/bin/Release/net8.0/PwrDrvr.LambdaDispatch.Router 2>&1 | tee router.log

Router with Default ThreadPool Threads, DOTNET_ThreadPool_UnfairSemaphoreSpinLimit=5

• 18,000 RPS
• ❌ 320% Router CPU
• 92% Extension CPU
• Router command line: DOTNET_ThreadPool_UnfairSemaphoreSpinLimit=5 LAMBDA_DISPATCH_ChannelCount=10 LAMBDA_DISPATCH_MaxConcurrentCount=10 LAMBDA_DISPATCH_AllowInsecureControlChannel=true LAMBDA_DISPATCH_PreferredControlChannelScheme=http LAMBDA_DISPATCH_FunctionName=dogs AWS_LAMBDA_SERVICE_URL=http://localhost:5051 AWS_REGION=us-east-2 AWS_ACCESS_KEY_ID=test-access-key-id AWS_SECRET_ACCESS_KEY=test-secret-access-key AWS_SESSION_TOKEN=test-session-token src/PwrDrvr.LambdaDispatch.Router/bin/Release/net8.0/PwrDrvr.LambdaDispatch.Router 2>&1 | tee router.log

Router with Max 1 Worker Thread, Max 1,000 (default) Completion Port Threads, DOTNET_ThreadPool_UnfairSemaphoreSpinLimit Unspecified

ThreadPool.SetMinThreads(1, 1);
ThreadPool.GetMaxThreads(out var workerThreads, out var completionPortThreads);
Console.WriteLine($"ThreadPool.GetMaxThreads returned {workerThreads} worker threads and {completionPortThreads} completion port threads");
var result = ThreadPool.SetMaxThreads(1, completionPortThreads);
Console.WriteLine($"ThreadPool.SetMaxThreads returned {result}");
ThreadPool.GetMaxThreads(out workerThreads, out completionPortThreads);
Console.WriteLine($"ThreadPool.GetMaxThreads returned {workerThreads} worker threads and {completionPortThreads} completion port threads");

• 17,000 RPS
• ✅ 124% Router CPU
• 90% Extension CPU
• Router command line: LAMBDA_DISPATCH_ChannelCount=10 LAMBDA_DISPATCH_MaxConcurrentCount=10 LAMBDA_DISPATCH_AllowInsecureControlChannel=true LAMBDA_DISPATCH_PreferredControlChannelScheme=http LAMBDA_DISPATCH_FunctionName=dogs AWS_LAMBDA_SERVICE_URL=http://localhost:5051 AWS_REGION=us-east-2 AWS_ACCESS_KEY_ID=test-access-key-id AWS_SECRET_ACCESS_KEY=test-secret-access-key AWS_SESSION_TOKEN=test-session-token src/PwrDrvr.LambdaDispatch.Router/bin/Release/net8.0/PwrDrvr.LambdaDispatch.Router 2>&1 | tee router.log

Router with Max 32,767 (default) Worker Threads, Max 1 Completion Port Thread, DOTNET_ThreadPool_UnfairSemaphoreSpinLimit Unspecified

ThreadPool.SetMinThreads(1, 1);
ThreadPool.GetMaxThreads(out var workerThreads, out var completionPortThreads);
Console.WriteLine($"ThreadPool.GetMaxThreads returned {workerThreads} worker threads and {completionPortThreads} completion port threads");
var result = ThreadPool.SetMaxThreads(workerThreads, 1);
Console.WriteLine($"ThreadPool.SetMaxThreads returned {result}");
ThreadPool.GetMaxThreads(out workerThreads, out completionPortThreads);
Console.WriteLine($"ThreadPool.GetMaxThreads returned {workerThreads} worker threads and {completionPortThreads} completion port threads");

• 17,000 RPS
• ❌ ❌ 720% Router CPU
• 90% Extension CPU
• Router command line: LAMBDA_DISPATCH_ChannelCount=10 LAMBDA_DISPATCH_MaxConcurrentCount=10 LAMBDA_DISPATCH_AllowInsecureControlChannel=true LAMBDA_DISPATCH_PreferredControlChannelScheme=http LAMBDA_DISPATCH_FunctionName=dogs AWS_LAMBDA_SERVICE_URL=http://localhost:5051 AWS_REGION=us-east-2 AWS_ACCESS_KEY_ID=test-access-key-id AWS_SECRET_ACCESS_KEY=test-secret-access-key AWS_SESSION_TOKEN=test-session-token src/PwrDrvr.LambdaDispatch.Router/bin/Release/net8.0/PwrDrvr.LambdaDispatch.Router 2>&1 | tee router.log

Router with Max 12 Worker Threads, Max 1 Completion Port Thread, DOTNET_ThreadPool_UnfairSemaphoreSpinLimit Unspecified

ThreadPool.SetMinThreads(1, 1);
var result = ThreadPool.SetMaxThreads(12, 1);
Console.WriteLine($"ThreadPool.SetMaxThreads returned {result}");

Note: Limiting worker threads to 25 drops CPU from 700% to 500-600%, then limiting worker threads to 20 drops CPU further from 500-600% to 300%, limiting to 2 drops CPU to 175%, and finally limiting to 1 drops CPU to 125%.

• 17,000 RPS
• ❌ 290% Router CPU
• 90% Extension CPU
• Router command line: LAMBDA_DISPATCH_ChannelCount=10 LAMBDA_DISPATCH_MaxConcurrentCount=10 LAMBDA_DISPATCH_AllowInsecureControlChannel=true LAMBDA_DISPATCH_PreferredControlChannelScheme=http LAMBDA_DISPATCH_FunctionName=dogs AWS_LAMBDA_SERVICE_URL=http://localhost:5051 AWS_REGION=us-east-2 AWS_ACCESS_KEY_ID=test-access-key-id AWS_SECRET_ACCESS_KEY=test-secret-access-key AWS_SESSION_TOKEN=test-session-token src/PwrDrvr.LambdaDispatch.Router/bin/Release/net8.0/PwrDrvr.LambdaDispatch.Router 2>&1 | tee router.log

Router with Max 1 Worker Thread, Max 1 Completion Port Thread, DOTNET_ThreadPool_UnfairSemaphoreSpinLimit Unspecified

ThreadPool.SetMinThreads(1, 1);
var result = ThreadPool.SetMaxThreads(1, 1);
Console.WriteLine($"ThreadPool.SetMaxThreads returned {result}");

• 17,000 RPS
• ✅ 125% Router CPU
• 90% Extension CPU
• Router command line: LAMBDA_DISPATCH_ChannelCount=10 LAMBDA_DISPATCH_MaxConcurrentCount=10 LAMBDA_DISPATCH_AllowInsecureControlChannel=true LAMBDA_DISPATCH_PreferredControlChannelScheme=http LAMBDA_DISPATCH_FunctionName=dogs AWS_LAMBDA_SERVICE_URL=http://localhost:5051 AWS_REGION=us-east-2 AWS_ACCESS_KEY_ID=test-access-key-id AWS_SECRET_ACCESS_KEY=test-secret-access-key AWS_SESSION_TOKEN=test-session-token src/PwrDrvr.LambdaDispatch.Router/bin/Release/net8.0/PwrDrvr.LambdaDispatch.Router 2>&1 | tee router.log