aws_lambda: cdk synth/deploy not working with lambda packaging on linux/arm64

[mjvirt]

Describe the bug

Attempting to run cdk synth/cdk deploy on Fedora Linux 36 on ARM64 with a Lambda function and with Docker Engine (not Desktop) installed, results in the error:

cdk synth
Bundling asset xxx...
docker: Error response from daemon: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: exec: "bash": executable file not found in $PATH: unknown.
jsii.errors.JavaScriptError: 
  Error: Failed to bundle asset I..., bundle output is located at /.../cdk.out/asset.8a2c1d48adc998a901bb4dd1a769638c9610807ff4c9b40fbde5b08670d8cb6c-error: Error: docker exited with status 127

Attempting to run docker with the image used (gleaned from docker --debug is fine:

docker run -ti public.ecr.aws/sam/build-python3.9 "bash" "-c" "uname -a"

Linux 394000d0e14b 5.19.6-200.fc36.aarch64 #1 SMP PREEMPT_DYNAMIC Wed Aug 31 16:40:17 UTC 2022 aarch64 aarch64 aarch64 GNU/Linux

But I have narrowed the issue down to the fact that specifying a uid/gid (which the cdk does) reproduces the error:

docker run -u 1000:1000 -ti public.ecr.aws/sam/build-python3.9 "bash" "-c" "uname -a"
docker: Error response from daemon: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: exec: "bash": executable file not found in $PATH: unknown.

Expected Behavior

cdk synth and cdk deploy should work just fine on linux/arm64.

Current Behavior

The cdk specifies the uid/gid of the user as part of synth and deploy to execute the docker container. However, this fails with:

cdk synth
Bundling asset xxx...
docker: Error response from daemon: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: exec: "bash": executable file not found in $PATH: unknown.
jsii.errors.JavaScriptError: 
  Error: Failed to bundle asset I..., bundle output is located at /.../cdk.out/asset.8a2c1d48adc998a901bb4dd1a769638c9610807ff4c9b40fbde5b08670d8cb6c-error: Error: docker exited with status 127

Reproduction Steps

With the CDK develop an app stack that includes a Lambda definition like below on Fedora Linux 36 with Docker Enginer (not Desktop, which is not available on Linux/ARM64) installed.

        notification_function = lambda_.Function(self, "SomeFunction",
            code=lambda_.Code.from_asset(path.join("lambda"),
                bundling=BundlingOptions(
                    image=lambda_.Runtime.PYTHON_3_9.bundling_image,
                    command=["bash", "-c", "pip3 install -r requirements.txt -t /asset-output && cp -au . /asset-output"]
                )
            ),
            runtime=lambda_.Runtime.PYTHON_3_9,
            handler="main.handler",
            timeout=Duration.seconds(900),
        )

Possible Solution

Unsure. As we know that running the docker container without specifying uid/gid (so, root) I guess doing so as part of the cdk would be one (perhaps insecure) workaround.

Additional Information/Context

This is a problem that I have reproduced at least from cdk version >= 2.38.0 and up.

CDK CLI Version

2.41.0 (build 6ad48a3)

Framework Version

No response

Node.js Version

v16.16.0

OS

Linux fedora 5.19.6-200.fc36.aarch64 #1 SMP PREEMPT_DYNAMIC Wed Aug 31 16:40:17 UTC 2022 aarch64 aarch64 aarch64 GNU/Linux

(Parallels VM on MacOS 12.5.1)

Language

Python

Language Version

Python 3.10.6

Other information

No response

[corymhall]

@mjvirt I think it will only run with -u 1000:1000 if it can't get info on the current user (os.userInfo). Is there something with your system configuration that would be causing that?

You can also try providing the user to use https://docs.aws.amazon.com/cdk/api/v2/docs/aws-cdk-lib.BundlingOptions.html#user

[mjvirt]

Apologies for the delayed response. I tried BundlingOptions user with 0 (root) and it works.
Then setting it to 1000 (or any other number) causes the issue. Just to be clear my local uid:gid is 1000:1000 so it appears that bit is picked up correctly (I presume it does that so that uid:gid is matched for the /asset-output volume for permissions?).
I can't think of anything in my set-up that would cause this. I presume this is supposed to work on Fedora (36) for arm64?
I am using the public image for aws_lambda build - and as I said, it works fine with 0/root.

[mjvirt]

I can confirm that I have just reproduced this on Fedora Linux 36 on x86_64 platform. Same issue:

Bundling asset [...]..
docker: Error response from daemon: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: exec: "bash": stat bash: permission denied: unknown.

The workaround of setting user="0" still applies.

[christophgysin]

Duplicate of #12610?

See #12610 (comment) to reproduce the root issue.