.NET 8 Publish with AOT on Amazon Linux still using internal docker image

[dguisinger]

Describe the bug

My understanding from Norm's comments in #1611 is that "dotnet lambda publish" will use a docker image if its not running on Amazon Linux.

I have tried converting my project over to AOT compiling and am using a CodeBuild image with Amazon Linux 2023 (AMAZON_LINUX_2_5). However it is still using the docker image to build.

My project needs access to CodeArtifact to download private Nuget packages.

Expected Behavior

I would expect AOT compiling to not use Docker when running inside Amazon Linux as is described.

Current Behavior

.NET 8 AOT compiling using Amazon Linux 2023 uses an AWS provided Docker image to build, causing build failures due to no customization of the build environment.

From the logs:

[Container] 2024/04/01 03:22:39.580275 Running command dotnet lambda package -pl Xxxxxxxxxxx -o output/Xxxxxxxxx.zip

217 | Amazon Lambda Tools for .NET Core applications (5.10.3)
218 | Project Home: https://github.com/aws/aws-extensions-for-dotnet-cli, https://github.com/aws/aws-lambda-dotnet
219 |  
220 | Found /etc/os-release
221 | Architecture not provided, defaulting to x86_64 for container build image.
222 | Executing publish command
223 | Starting container for native AOT build using build image: public.ecr.aws/sam/build-dotnet8:latest-x86_64.
224 | ... invoking 'docker run --name tempLambdaBuildContainer-6e74898f-a244-4dad-b855-17e6f1cad8f1 --rm --volume "/codebuild/output/src1827729237/src":/tmp/source/ -i public.ecr.aws/sam/build-dotnet8:latest-x86_64 dotnet publish "/tmp/source/Xxxxxxxxxx" --output "/tmp/source/Xxxxxxxxxxx/bin/Release/net8.0/publish" --configuration "Release" --framework "net8.0" /p:GenerateRuntimeConfigurationFiles=true --runtime linux-x64 --self-contained True /p:StripSymbols=true' from directory /codebuild/output/src1827729237/src
225 | ... docker run: Unable to find image 'public.ecr.aws/sam/build-dotnet8:latest-x86_64' locally
226 | ... docker run: latest-x86_64: Pulling from sam/build-dotnet8

Reproduction Steps

I don't believe this is project code related

Possible Solution

At a minimum, documentation of this behavior seems to be lacking to completely understand how it works. Or it could be a bug that is using the Docker image to build even when its not necessary.

Additional Information/Context

I created a Q&A issue about this 5 days ago with no response, so I am escalating to a github issue.
#1712

AWS .NET SDK and/or Package version used

Amazon.Lambda.Tools 5.10.3

Targeted .NET Platform

.NET 8

Operating System and version

AmazonLinux

[ashishdhingra]

Needs review with the team.

[ashishdhingra]

@dguisinger Good afternoon. Somehow, I'm unable to reproduce the issue.

• Used the current AMI image al2023-ami-2023.4.20240429.0-kernel-6.1-x86_64 to create Amazon Linux 2023 EC2 instance.
• SSH into EC2 instance.
• Installed .NET 8

  sudo rpm --import https://packages.microsoft.com/keys/microsoft.asc
  sudo wget -O /etc/yum.repos.d/microsoft-prod.repo https://packages.microsoft.com/config/fedora/37/prod.repo
  sudo dnf install -y dotnet-sdk-8.0
  dotnet --version > /tmp/dotnet-version
  

• To build linux native AOT binaries also installed (refer https://github.com/awslabs/dotnet-nativeaot-labs/blob/main/README.md#prerequisites and https://learn.microsoft.com/en-us/dotnet/core/deploying/native-aot/?tabs=net7%2Clinux-ubuntu#prerequisites):

  dnf install -y clang zlib-devel 
  

• Installed Docker using command sudo yum install -y docker.
• Installed latest version of Amazon.Lambda.Tools using command dotnet tool update -g Amazon.Lambda.Tools and current templates using command dotnet new install Amazon.Lambda.Templates.
• Restarted the EC2 instance.
• SSH into EC2 instance.
• Created .NET Native AOT Lambda project:
  • Created a directory named dotnetnaticeaottest.
  • Changed current directory to dotnetnaticeaottest.
  • Used command dotnet new lambda.NativeAOT.
  • Changed directory to src/dotnetnaticeaottest.
  • Updated Amazon.Lambda.RuntimeSupport and Amazon.Lambda.Annotations NuGet packages using dotnet add package command.
  • Executed dotnet lambda deploy-function (kindly note that this assumes that you have AWS profile with credentials and region configured).

Below is the output of dotnet lambda deploy-function command:

Amazon Lambda Tools for .NET Core applications (5.10.5)
Project Home: https://github.com/aws/aws-extensions-for-dotnet-cli, https://github.com/aws/aws-lambda-dotnet
	
Found /etc/os-release
Linux distribution is Amazon Linux 2023, NativeAOT container build is optional
Executing publish command
... invoking 'dotnet publish', working folder '/home/ec2-user/dotnetnaticeaottest/src/dotnetnaticeaottest/bin/Release/net8.0/publish'
... dotnet publish "/home/ec2-user/dotnetnaticeaottest/src/dotnetnaticeaottest" --output "/home/ec2-user/dotnetnaticeaottest/src/dotnetnaticeaottest/bin/Release/net8.0/publish" --configuration "Release" --framework "net8.0" --self-contained true /p:GenerateRuntimeConfigurationFiles=true --runtime linux-x64
... publish: MSBuild version 17.9.8+b34f75857 for .NET
... publish:   Determining projects to restore...
... publish:   Restored /home/ec2-user/dotnetnaticeaottest/src/dotnetnaticeaottest/dotnetnaticeaottest.csproj (in 2.5 sec).
... publish:   dotnetnaticeaottest -> /home/ec2-user/dotnetnaticeaottest/src/dotnetnaticeaottest/bin/Release/net8.0/linux-x64/dotnetnaticeaottest.dll
... publish:   Generating native code

Notice the log line Linux distribution is Amazon Linux 2023, NativeAOT container build is optional.

Could you please use the latest version of lambda tools and check if the issue is still reproducible?

Thanks,
Ashish

[normj]

While we figure out what is going wrong and should be able to force non-container builds by adding the --use-container-for-build false switch to the command you are calling with Amazon.Lambda.Tools.

[dguisinger]

@ashishdhingra ,
Is there a difference between CodeBuild's Amazon Linux 2023 image and the EC2 image that would cause it to fail detection?

Regarding if I have updated to the latest and tried again, I shelved my entire project a month ago because I just didn't have time to deal with it at the moment - My entire 4-month experience with .NET 8 on Lambda been extremely frustrating... The only reason I was even pushing to do AoT was someone mentioned it fixed all of the random timeout issues he was having on .NET 7 on Lambda, similar to the issues I've been having with .NET 8 on Lambda, which you guys have also been unable to reproduce....

I suppose its also possible there is something wrong in my hand-converted project files as they started as .NET 6, moved to .NET 8, and then added AoT. I don't know if there is anything in the project file that is used when detecting what container settings to use.

The CodeBuild buildspec.yml I've been using looks like this:

phases:
  install:
    commands:
      - npm install aws-cdk -g
      - curl -sSL https://dot.net/v1/dotnet-install.sh | bash /dev/stdin --channel LTS
      - dotnet tool install -g AWS.CodeArtifact.NuGet.CredentialProvider
      - dotnet tool install -g Amazon.Lambda.Tools
  build:
    commands:
      - dotnet nuget add source "https://xxxxxxxxx.d.codeartifact.us-east-1.amazonaws.com/nuget/xxxxxxxx/v3/index.json" -n "xxxxxxx/xxxxxxx"
      - aws codeartifact login --tool dotnet --domain xxxxxxx --domain-owner xxxxxxxxx --repository xxxxxxxx
      - dotnet restore
      - dotnet build
      - dotnet lambda package -pl xxxxxxx -o output/xxxxxxx.zip
artifacts:
  files:
    - '**/*'

The configuration section of my project file looks like:

<PropertyGroup>
      <OutputType>Exe</OutputType>
    <TargetFramework>net8.0</TargetFramework>
    <Nullable>enable</Nullable>
    <ImplicitUsings>enable</ImplicitUsings>
    <AWSProjectType>Lambda</AWSProjectType>
    <CopyLocalLockFileAssemblies>true</CopyLocalLockFileAssemblies>
    <PublishAot>true</PublishAot>
    <StripSymbols>true</StripSymbols>
    <UserSecretsId>xxxxxxx</UserSecretsId>
    <PublishTrimmed>true</PublishTrimmed>
    <JsonSerializerIsReflectionEnabledByDefault>true</JsonSerializerIsReflectionEnabledByDefault>
  </PropertyGroup>

@normj thanks for that suggestion, I will try it. Just to make sure, you are saying to add --use-container-for-build false to "dotnet lambda package"? Is that flag documented? Like I mentioned in the original issue, its hard to find any real documentation around this process, it just spins up a container where I don't seem to have any control over the build setup. I wouldn't mind it building in a container if I had a clue as to how to connect to it to CodeArtifact using the IAM role that CodeBuild container is running under.... the CodeArtifact login and Nuget source registration requires executing my own pre-build commands.

[Beau-Gosse-dev]

It might be documented somewhere else, but you can see slightly more details in the help command. dotnet lambda package --help

We look at the contents of the file /etc/os-release on the build machine to see what OS it is. Would you be able to print those contents and share it with us? Or if you let me know how you setup the build or exactly which image you're running the build on, I can look into that file.

[Beau-Gosse-dev]

Is this the configuration you're using? If so, it seems like the image should have the correct text in the os-release file to understand it doesn't need to use a container.

[normj]

@dguisinger That switch shows up when you run dotnet lambda package --help to see the list switches.

> dotnet lambda package --help
Amazon Lambda Tools for .NET Core applications (5.10.5)
Project Home: https://github.com/aws/aws-extensions-for-dotnet-cli, https://github.com/aws/aws-lambda-dotnet

package:
   Command to package a Lambda project either into a zip file or docker image if --package-type is set to "image". The output can later be deployed to Lambda with either deploy-function command or with another tool.

   dotnet lambda package [arguments] [options]
   Arguments:
      <ZIP-FILE> The name of the zip file to package the project into
   Options:
      --disable-interactive                   When set to true missing required parameters will not be prompted for.
      --region                                The region to connect to AWS services, if not set region will be detected from the environment.
      --profile                               Profile to use to look up AWS credentials, if not set environment credentials will be used.
      --profile-location                      Optional override to the search location for Profiles, points at a shared credentials file.
      --aws-access-key-id                     The AWS access key id. Used when setting credentials explicitly instead of using --profile.
      --aws-secret-key                        The AWS secret key. Used when setting credentials explicitly instead of using --profile.
      --aws-session-token                     The AWS session token. Used when setting credentials explicitly instead of using --profile.
      -pl    | --project-location             The location of the project, if not set the current directory will be assumed.
      -cfg   | --config-file                  Configuration file storing default values for command line arguments.
      -pcfg  | --persist-config-file          If true the arguments used for a successful deployment are persisted to a config file.
      -c     | --configuration                Configuration to build with, for example Release or Debug.
      -f     | --framework                    Target framework to compile, for example netcoreapp3.1.
      -farch | --function-architecture        The architecture of the Lambda function. Valid values: x86_64 or arm64. Default is x86_64
      --msbuild-parameters                    Additional msbuild parameters passed to the 'dotnet publish' command. Add quotes around the value if the value contains spaces.
      -fl    | --function-layers              Comma delimited list of Lambda layer version arns
      -o     | --output-package               The zip file that will be created with compiled and packaged Lambda function.
      -dvc   | --disable-version-check        Disable the .NET Core version check. Only for advanced usage.
      -pt    | --package-type                 The deployment package type for Lambda function. Valid values: image, zip
      -it    | --image-tag                    Docker image name and tag in the 'name:tag' format.
      -df    | --dockerfile                   The docker file used to build the image. Default value is "Dockerfile".
      -dbo   | --docker-build-options         Additional options passed to the "docker build" command.
      -dbwd  | --docker-build-working-dir     The directory to execute the "docker build" command from.
      --docker-host-build-output-dir          If set a "dotnet publish" command is executed on the host machine before executing "docker build". The output can be copied into image being built.
      -ucfb  | --use-container-for-build      Use a local container to build the Lambda binary. A default image will be provided if none is supplied.
      -cifb  | --container-image-for-build    The container image tag (with version) to be used for building the Lambda binary.
      -cmd   | --code-mount-directory         Path to the directory to mount to the build container. Otherwise, look upward for a solution folder.

[dguisinger]

@Beau-Gosse-dev ... I am feeling incredibly stupid at this moment. I am so sorry for wasting everyone's time on this. I went to double check what CodeBuild had for settings after you posted that image and it was still on a different code build image. My CDK script had been updated to Amazon Linux 2 2023, but apparently it missed getting ran and I was too focused on trying to figure out how to get my Nuget source into your containerized build process, that I completely missed the CDK pipeline project hadn't been re-deployed. I redeployed the pipeline and it works

[normj]

@dguisinger No worries, it happens to all of us. Does that mean we can close the issue?

[dguisinger]

Yes sir, I will go ahead and close it. Thanks Norm

[github-actions]

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.