Replies: 7 comments 1 reply
-
|
@Simonl9l I tried reproducing this and my simple cases of having an Minimal API return a 303 had no issues. This is my simple repo code From the logs it looks like the Lambda code got stuck returning back from the ASP.NET Core part back to |
Beta Was this translation helpful? Give feedback.
-
|
@normj - Thanks for jumping in on this. Ha - I see you edited to remove the I've rigged the Lambda code to do exactly as you have in code as above - plugged into the rest of the setup for routes etc so as not to have to mess with the client side code - but am still getting the same timeout issue. To confirm that you deployed the above in an AWS Lambda? Of note, we're building via Docker to a Single Executable that we Zip and deploy via CDK - and are using the al2.v14 runtime Here are the detailed logs: I have retested locally and the 303 works as expected. So this is narrowing down the variables... Should we have any concern with the use of the custom runtime? We are running .Net 7 here - hence custom runtime! Here is the dockerfile - were not AoT or Trimming with the latest finding: FROM public.ecr.aws/amazonlinux/amazonlinux:2 AS base
WORKDIR /dotnet
RUN yum update -y && yum install -y clang llvm krb5-devel openssl-devel zip tar gzip compat-openssl10 libicu
RUN curl -sSL https://dot.net/v1/dotnet-install.sh | bash /dev/stdin -Channel 7.0 -InstallDir /usr/share/dotnet
RUN ln -s /usr/share/dotnet/dotnet /usr/bin/dotnet
FROM base AS build-env
WORKDIR /build
COPY . .
FROM build-env AS build-stage
ARG TARGET_PATH
ARG TARGET_RID
ARG AOT
ARG TRIM
ARG CONTAINED
ARG SINGLE
WORKDIR /build/$TARGET_PATH
RUN dotnet publish -r $TARGET_RID -c Release -o /build/publish --self-contained $CONTAINED -p:PublishSingleFile=$SINGLE -p:PublishAoT=$AOT -p:PublishTrimmed=$TRIM -p:StripSymbols=true
WORKDIR /build
RUN cp /build/publish/bootstrap bootstrap
RUN zip Lambda.zip bootstrap
FROM scratch AS export-stage
COPY --from=build-stage /build/Lambda.zip / |
Beta Was this translation helpful? Give feedback.
-
|
@normj did some digging with the team, and turned on The behavior we are seeing is that cold runs succeed, but immediate subsequent runs (Warm) time out. Tracking per the source code here Per other notes will contact you offline. Successful Cold Start: Subsequent call look like this: Notes on a logging issues: It seems that we suspect these events should actually come out up top: |
Beta Was this translation helpful? Give feedback.
-
|
@normj - Just following up as we continue to research this. We did find this possibly related issue: dotnet/aspnetcore#45373 and modified the handler as below (both adding a parameter and using the ar builder = WebApplication.CreateBuilder(args);
// Add AWS Lambda support. When application is run in Lambda Kestrel is swapped out as the web server with Amazon.Lambda.AspNetCoreServer. This
// package will act as the webserver translating request and responses between the Lambda event source and ASP.NET Core.
builder.Services.AddAWSLambdaHosting(LambdaEventSource.RestApi);
var app = builder.Build();
app.UseHttpsRedirection();
app.MapGet("/", () => "Welcome to running ASP.NET Core Minimal API on AWS Lambda");
async Task<IResult> Handler(HttpContext context, CancellationToken ctx)
{
Console.WriteLine("In Function");
context.Response.Headers.Add("Location", "https://aws.amazon.com/");
return Results.StatusCode(303);
}
Delegate delegated = Handler;
app.MapPost("/api/stripe/create-checkout-session", delegated);
app.Run();Our observation are that we get mostly alternate succeed (with 303)/fail (with timeout) invocations, but do occasionally get a couple of succeed's in a row. It essentially works every time after a Of note if we switch the csproj back to However given our build chain and use of the Dockerfile above (comment) the RUN dotnet publish -r linux-x64 \
--configuration Release \
--output /build/publish \
--self-contained true \
-p:PublishSingleFile=true \
-p:PublishAoT=false \
-p:PublishTrimmed=false \
-p:StripSymbols=true \
-p:PublishReadyToRun=true and we are using Additionally our assumption is to install .Net 7 into the We also then emit a zip of the built lambda to the container host and deploy as a zip then in a container. Again do you see any issue with this approach - on basis that we configure the lambda as Finally I've tweaked the discussion subject as this is all .Net 7 runtime related All help appreciated! Whilst short term we can probably workaround .Net 6 limitation vs .Net new features, we'd sure like to get a working/stable environment for .Net 7 and not have to wait for .net LTS support. |
Beta Was this translation helpful? Give feedback.
-
|
@normj just following up. We have for the time being dropped back to the .net6 runtime for our Minimal API/LambdaHosting lambdas, until there is more support for .Net 7. We do have observations that it would be good to share and have communicated directly as we don't fully understand the runtime code and don't want to commend based on that here, as noted other issues. We're somewhat surprised others have not run into this issue with Minimal API with .Net 7 (with Custom runtime). |
Beta Was this translation helpful? Give feedback.
-
|
Catching up on the thread here, let me know if I missed any questions. I have taken your code here and published using .NET 7 as a single file to I find the description of how you are using containers to build and deploy a bit confusing. Any chance you could create me a sample repo I can clone and walk through? Out of curiosity what are the Lambda memory settings you are using? Also what code size does Lambda report for the function. For me it reports 41MB.
|
Beta Was this translation helpful? Give feedback.
-
|
@normj Apologies, things have been a tad crazy. We had temporally reverted back to using .Net 6, as that had solved the issue, and temporarily has perhaps extended longer then planned; but now need to leverage some .Net 7 C#11 features... Generally the timeout issue appears unrelated to the 303 use case. We have updated to the latest Here is a link to a repo, this is based 100% from the standard
<PropertyGroup>
<OutputType>Exe</OutputType>
<TargetFramework>net7.0</TargetFramework>
<ImplicitUsings>enable</ImplicitUsings>
<Nullable>enable</Nullable>
<GenerateRuntimeConfigurationFiles>true</GenerateRuntimeConfigurationFiles>
<AWSProjectType>Lambda</AWSProjectType>
<AssemblyName>bootstrap</AssemblyName>
<!-- This property makes the build directory similar to a publish directory and helps the AWS .NET Lambda Mock Test Tool find project dependencies. -->
<CopyLocalLockFileAssemblies>true</CopyLocalLockFileAssemblies>
<!-- Generate Native AOT image during publishing to improve cold start time. -->
<PublishAot>false</PublishAot>
<!-- StripSymbols tells the compiler to strip debugging symbols from the final executable if we're on Linux and put them into their own file.
This will greatly reduce the final executable's size.-->
<StripSymbols>true</StripSymbols>
<!-- TrimMode partial will only trim assemblies marked as trimmable. To reduce package size make all assemblies trimmable and set TrimMode to full.
If there are trim warnings during build, you can hit errors at runtime.-->
<TrimMode>partial</TrimMode>
<!-- Generate ready to run images during publishing to improvement cold starts. -->
<PublishReadyToRun>true</PublishReadyToRun>
<RestorePackagesWithLockFile>true</RestorePackagesWithLockFile>
</PropertyGroup>We also fixed the The build process uses the:
The repo has a GitHub Workflow that can be configured to build/deploy it as above, given the steps in the As such the repo can be cloned to reproduce the same outcomes. Per the repo we initially added some addition Services for the .Net Code Generated Serialization, and a few other logging or AWS based services. This was on the basis it was related to additional code we were implementing than a base behavior. We noted that all calls timed out! From this, we then removed (commented out) all the additional code that differs from the Our observations are that:
The Cloud Watch logging indicates the timedout requests never gets out of the middleware as the since we have the logging level set quite low: {
"Logging": {
"LogLevel": {
"Default": "Debug",
"Microsoft.AspNetCore": "Debug"
}
}
}the logging is quite verbose. Our initial conclusion is that the default route works since there is NO DI or other request mapping; but in other scenarios the middleware is likely invoked. Also we note that if we gap request by ~30 seconds it seem fine (for the same request). On rapid fire it chokes. Our guess is that there is some kind of difference with how the We do hope this is enough to prompt more investigation from you and the the AWS .Net team, as we're at a critical juncture to need to switch to .net 7. Please advise! |
Beta Was this translation helpful? Give feedback.
-
|
related to logged Bug - #1543 |
Beta Was this translation helpful? Give feedback.
-
As part of a Stripe/Checkout integration with a request with a
application/x-www-form-urlencodedpayload, it requires that a 303 status is returned with the response header being the Stripe checkout url provided but he service:This is run though both cloud front and an RestApi Gateway
here is the Lambda log (I extended the timeout to 30 secs):
Here are the Gateway logs:
I need the
303to get back to the browser so to will the use the Response Location and perform aGETThis all work cleanly locally with Kestrel
I have no idea what is cause the timeout - it seems to be happening the Handler/Wrapper - thoughts?
Beta Was this translation helpful? Give feedback.
All reactions