AL2023 base image support

[Sparticuz]

I just wanted to create an issue to track the AL2 -> AL2023 transition. I compiled chromium to work on AL2023, but it will not run properly in Lambda because of AL2 not supporting some glibc versions.

@fromtheexchange Thank you for the suggestions. We are aware of the AL2 -> AL2022 transition and are working on plans for how to roll this out in Lambda, although at this stage we're not in a position to give a schedule. It's a little complicated because the minimum Linux kernel version is different between AL2 and AL2022.

Originally posted by @jtuliani in #47 (comment)

EDIT: cough AL2023 instead of AL2022

[Simonl9l]

are there any workarounds here such that we can deploy .Net 7.0 Arm64 lambdas per aws/aws-lambda-dotnet#1310

[jtuliani]

Thank you for raising this issue. This should be addressed in AL2022, but unfortunately you can't use AL2022 in Lambda yet because it requires a kernel upgrade. We're working on that, but I don't have a schedule to share at this time, and to set expectations it's likely to be well into next year. Unfortunately, I don't have a workaround I can offer in the meantime.

[Simonl9l]

@jtuliani are there any other forums (Microsoft) that we can comment on to encourage movement, of is this entirely under Amazon/AWS/Lambda team control? Do we need to reach out to our AWS Account Management?

We are under the impression that there is an underlying EOL issue with the prior/current version, that expires early next year.

[tmitchel2]

@jtuliani Is there the possibility of getting a pre-release AL2022 for lambda earlier?

[Simonl9l]

@tmitchel2 as I understand it (so very possibly incorrect/incomplete) having a pre-release image is not sufficient, as the AWS Lambda back end would also need to support said base image per the “Kernel upgrade” as @jtuliani mentions!

We’ve been pondering reaching out to our AWS account management representatives to see if they can help apply any influence.

Our best efforts at a work around is that . in our case, it’s currently impossible to build a .Net 7 AoT (native binary) from a M1 Apple Silicon environment for x64 using QEMU emulation as the Docker build crashes - not really surprising given combination of Rosetta x64 emulation too. Whilst we can trigger a GitHub workflow to do this (via an Ubuntu runner) and place the built Lambda package zip in S3 and have CDK deployments pick it up from there. This is very inefficient from a developer experience/productivity perspective.

Now that .Net 7 SDK is released, one hopes that there is more incentive to get this resolved, as this is a blocking dependency given .Net7 uses the newer glibc libraries. However it seems AWS predominantly only generally support LTS runtimes, so we’d have to wait another 12 months…and continue to be dependent on custom runtime images.

As we scale up we will really need to switch to ARM based Lambda’s to more effectively manage costs. AoT offer significant cold start performance improvements such that we may need to not need to pursue provisioned capacity - so to us getting this resolved is critical.

We gather the current version EOL is now 2024 so that incentive is now less pressing unfortunately!

[tmitchel2]

@Simonl9l Yup Im in exactly the same position. Im on arm64 with NET 6.0 and want to upgrade to NET 7.0 to leverage AOT compilation to get rid of the cold start time. Ive taken the x86 route for now so I can at least upgrade but obviously not desirable from a cost perspective.

I only have an x86 mac, but I don't understand your issue with having an M1. I just use cdk for setting up my environment builds, I had no issues in that area to build on arm or x86 variants. The only issue was the glibc one on arm + NET 7.0 at runtime. Cdk has made the process seemless and easy in my scenario for remote builds and then locally I just build for osx or forget the platform specific stuff entirely.

[FMKerckhof]

@jtuliani would you have any update on the expected timing? How far ahead of us in "well into next year" for 2023?

We're working on that, but I don't have a schedule to share at this time, and to set expectations it's likely to be well into next year. Unfortunately, I don't have a workaround I can offer in the meantime.

[tmitchel2]

@FMKerckhof sounds like we just need to wait for .NET 7.0.4 which will work with the existing AL2 lambda host.

[FMKerckhof]

thanks for weighing in @tmitchel2 - I need specific c++ libraries but for certain R packages (not for .NET) which need to be deployed in a custom runtime container, sadly for those there is no update trajectory foreseen that will resolve the issue hence I'd like an outlook on the availability of a base image.

[cforce]

Graal native images build done with oracle native image buildpack (maven, gradle) don‘t run as the used GLIBc version are not available at runtime at the outdated lambda base image. Please deliver a recent image with recent GLIBC support.
It’s half a year ago since this request was raised and since then it was not possible for aws to update the base image?
We develop and build on „modern“ images like 2021 jammy (glibc 2.34) but not like its provided on 2018 bionic(2.7) 😄 or focal (2.32). I expect aws to deliver and use lambda base images and together glibc support in sync with major distro but not years later 🙈
Already supported on Azure functions since quite a while, isn‘it? https://learn.microsoft.com/en-us/azure/azure-functions/functions-run-local?tabs=v4%2Clinux%2Cjava%2Cportal%2Cbash#publish

[cforce]

aws/aws-lambda-go#340
actions/runner-images#6647
quarkusio/quarkus#25897
awslabs/aws-lambda-rust-runtime#455

[Sparticuz]

For anyone that missed this

from @jtuliani #92 (comment)

Today, we have published a preview AWS Lambda base container image for AL2023, named ‘provided.al2023’. You can use this image to build and deploy your Lambda functions using AL2023-based containers.

This image is tagged ‘preview’ and should not be used for production workloads. We invite you to provide your feedback on this preview image below. We are working on GA support for AL2023 in Lambda, as both a base container image and a managed runtime. This is scheduled to ship in the coming weeks.

The provided.al2023 is based on the AL2023-minimal image, which is available in public Amazon Linux ECR. It is a version of AL2023 containing the only essential components necessary to install other packages. In particular, it uses microdnf as the package manager (AL2023 replaces yum with dnf; AL2023-minimal uses a special version of dnf called microdnf, and dnf is a symlink for microdnf). Also, curl and gnupg2 are included as their minimal versions, curl-minimal and gnupg2-minimal.

The AL2023 image includes glibc version 2.34 (previously, version 2.26 was included in provided.al2).

Going forward, the provided.al2023 image will be used as the base for future runtime releases, including Node.js 20, Python 3.12, Java 21 and .NET 8.

[runlevel5]

As of 22 Feb 2023, the AL2023 has reached GA I believe (because the preview tag is no longer around):

The public.ecr.aws/lambda/provided:al2023 could be used if you prefer to adopt it now

[jtuliani]

Correct, AL2023 support was launched on 2023-11-10: https://aws.amazon.com/blogs/compute/introducing-the-amazon-linux-2023-runtime-for-aws-lambda/

Closing this issue.