NodejsFunction: support buildx and docker-container driver

[lmouhib]

Describe the bug

NodeJsFunction fail the build during cdk synth when using Python. I am not providing my own Dockerfile. The implementation is in this repo. Below is the log for the failur.

#14 [10/10] RUN /sbin/useradd -u 1000 user && chmod 711 /
#14 CACHED
WARNING: No output specified with docker-container driver. Build result will only remain in the build cache. To push result image into registry use --push or to load image into docker use --load

View build details: docker-desktop://dashboard/build/sad_heisenberg/sad_heisenberg0/x0isk1zvr1xmvyt4hs4a2ssit

What's Next?
  View a summary of image vulnerabilities and recommendations → docker scout quickview
Bundling asset Stack/InteractiveSessionProvider/OnEventHandlerFunction/Code/Stage...
esbuild cannot run locally. Switching to Docker bundling.
Unable to find image 'cdk-9ee7a11fc0243092baf39e8f9d149e65e683a14e5695909d84aa461dc52a6dab:latest' locally
docker: Error response from daemon: pull access denied for cdk-9ee7a11fc0243092baf39e8f9d149e65e683a14e5695909d84aa461dc52a6dab, repository does not exist or may require 'docker login': denied: requested access to the resource is denied.
See 'docker run --help'.
jsii.errors.JavaScriptError: 
  @jsii/kernel.RuntimeError: Error: Failed to bundle asset Stack/InteractiveSessionProvider/OnEventHandlerFunction/Code/Stage, bundle output is located at /Users/yyyy/xxxxx/cdk.out/bundling-temp-e33d0e767876b374cc4c34eda8c2c07e1cc83da5543b15d1b7bae574e0fb9f6e-error: Error: docker exited with status 125
  --> Command: docker run --rm -u "504:20" -v "/var/folders/ty/hzqp2fl51r9c1146jdmgl0mc0000gr/T/jsii-kernel-srRVBP/node_modules/@cdklabs/aws-data-solutions-framework/lib/processing/lib/spark-runtime/emr-containers/resources/lambdas/managed-endpoint:/asset-input:delegated" -v "/Users/yyyy/xxxxx/emr-cost-tracking-bug-fix/cdk.out/bundling-temp-e33d0e767876b374cc4c34eda8c2c07e1cc83da5543b15d1b7bae574e0fb9f6e:/asset-output:delegated" -w "/" cdk-9ee7a11fc0243092baf39e8f9d149e65e683a14e5695909d84aa461dc52a6dab bash -c "esbuild --bundle \"/asset-input/index.mjs\" --target=node20 --platform=node --outfile=\"/asset-output/index.js\" --external:@aws-sdk/*"
      at Kernel._Kernel_ensureSync (/private/var/folders/ty/hzqp2fl51r9c1146jdmgl0mc0000gr/T/tmpva2camqw/lib/program.js:10491:23)
      at Kernel.sinvoke (/private/var/folders/ty/hzqp2fl51r9c1146jdmgl0mc0000gr/T/tmpva2camqw/lib/program.js:9876:102)
      at KernelHost.processRequest (/private/var/folders/ty/hzqp2fl51r9c1146jdmgl0mc0000gr/T/tmpva2camqw/lib/program.js:11696:36)
      at KernelHost.run (/private/var/folders/ty/hzqp2fl51r9c1146jdmgl0mc0000gr/T/tmpva2camqw/lib/program.js:11656:22)
      at Immediate._onImmediate (/private/var/folders/ty/hzqp2fl51r9c1146jdmgl0mc0000gr/T/tmpva2camqw/lib/program.js:11657:46)
      at process.processImmediate (node:internal/timers:478:21)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/yyyy/xxxxx/app.py", line 10, in <module>
    EmrCostTrackingBugFixStack(app, "Stack",
  File "/Users/yyyy/xxxxx/.venv/lib/python3.9/site-packages/jsii/_runtime.py", line 118, in __call__
    inst = super(JSIIMeta, cast(JSIIMeta, cls)).__call__(*args, **kwargs)
  File "/Users/yyyy/xxxxx/emr_cost_tracking_bug_fix/emr_cost_tracking_bug_fix_stack.py", line 17, in __init__
    emr_eks_cluster = dsf.processing.SparkEmrContainersRuntime.get_or_create(self,
  File "/Users/yyyy/xxxxx/.venv/lib/python3.9/site-packages/cdklabs/aws_data_solutions_framework/processing/__init__.py", line 1551, in get_or_create
    return typing.cast("SparkEmrContainersRuntime", jsii.sinvoke(cls, "getOrCreate", [scope, props]))
  File "/Users/yyyy/xxxxx/.venv/lib/python3.9/site-packages/jsii/_kernel/__init__.py", line 149, in wrapped
    return _recursize_dereference(kernel, fn(kernel, *args, **kwargs))
  File "/Users/yyyy/xxxxx/.venv/lib/python3.9/site-packages/jsii/_kernel/__init__.py", line 418, in sinvoke
    response = self.provider.sinvoke(
  File "/Users/yyyy/xxxxx/.venv/lib/python3.9/site-packages/jsii/_kernel/providers/process.py", line 383, in sinvoke
    return self._process.send(request, InvokeResponse)
  File "/Users/yyyy/xxxxx/.venv/lib/python3.9/site-packages/jsii/_kernel/providers/process.py", line 342, in send
    raise RuntimeError(resp.error) from JavaScriptError(resp.stack)
RuntimeError: Error: Failed to bundle asset Stack/InteractiveSessionProvider/OnEventHandlerFunction/Code/Stage, bundle output is located at /Users/yyyy/xxxxx/cdk.out/bundling-temp-e33d0e767876b374cc4c34eda8c2c07e1cc83da5543b15d1b7bae574e0fb9f6e-error: Error: docker exited with status 125

Expected Behavior

The cdk synth should succeed

Current Behavior

CDK synth fail

Reproduction Steps

I can link

Possible Solution

No response

Additional Information/Context

No response

CDK CLI Version

2.128.0

Framework Version

No response

Node.js Version

v20.11.1

OS

Mac

Language

Python

Language Version

3.9.6

Other information

No response

[pahud]

We can't debug your code from an external repo but looking at your provided error messages:

Unable to find image 'cdk-9ee7a11fc0243092baf39e8f9d149e65e683a14e5695909d84aa461dc52a6dab:latest' locally
docker: Error response from daemon: pull access denied for cdk-9ee7a11fc0243092baf39e8f9d149e65e683a14e5695909d84aa461dc52a6dab, repository does not exist or may require 'docker login': denied: requested access to the resource is denied.

Looks like

1. cdk-9ee7a11fc0243092baf39e8f9d149e65e683a14e5695909d84aa461dc52a6dab:latest does not exist at local
2. docker daemon was trying to pull that from your staging ecr repo and failed due to permission denied.

I am guessing this might be the permission issue.

Instead of looking at your linked code from the external repo, are you able to provide a minimal code snippet that reproduces this issue here? Simplifying the code would help us address the root cause easier.

[lmouhib]

We can't debug your code from an external repo but looking at your provided error messages:

The repo is hosted under an AWS github organisation. Also I am not providing any Dockerfile, its using the Dockerfile provided with cdk-lib to build the nodejs function.

docker daemon was trying to pull that from your staging ecr repo and failed due to permission denied
On the same code base if I used finchI successfully build and deploy, but with docker it fails with permission denied.

[pahud]

If that error only happens from https://github.com/awslabs/data-solutions-framework-on-aws please submit a ticket to that repo for their visibility.

If you believes it still happens in aws-cdk-lib please provide a minimal code snippet here so we can reproduce it.

[github-actions]

This issue has not received a response in a while. If you want to keep this issue open, please leave a comment below and auto-close will be canceled.

[lmouhib]

@pahud i am one the maintainer the repo I mention. The issues is not with the repo, but happens on how CDK pull the docker image, which I have no control on, because I do not specify that image.

[bwg]

I had this same error when to bundling a NodejsFunction via docker in GitHub Actions. The issue was that Docker Buildx defaults to using the docker-container driver for new builders, and that driver does not automatically load images into the internal image store (unlike the docker driver). So when the CDK attempts to run the esbuild container it just built, Docker doesn't have any reference for it.
https://docs.docker.com/build/drivers/#loading-to-local-image-store

Fortunately, Docker has provided a way to configure the builder to load the images by default:
https://docs.docker.com/build/drivers/#load-by-default

The workaround is to configure the builder with the default-load=true driver opt flag.

@lmouhib check your Docker builder configuration to see what driver its using.

A solution that could be implemented by the CDK team would be to call docker build with the --load arg, to explicitly tell the builder to load the image to the local image store. This would remove the need for any special Docker configurations.

[pahud]

WARNING: No output specified with docker-container driver. Build result will only remain in the build cache. To push result image into registry use --push or to load image into docker use --load

You generally don't have to specify --push or --load. Looks like you have opted in the docker-container driver, which is not the default driver. The docker-container driver is an alternative driver provided by the docker buildx command, which is a CLI plugin that extends the docker command with the ability to build multi-platform images and perform other advanced build operations.

CDK at this moment does not fully support docker buildx command. Are you able to switch back to the default docker driver and try again?

[bwg]

CDK at this moment does not fully support docker buildx command. Are you able to switch back to the default docker driver and try again?

Unsure who you were replying to, but we use GitHub actions for deployment, and have a re-usable action that configures docker with buildx. Using the standard docker driver does not work in our case to support multi-arch builds. Fortunately we are able to work around by configuring buildx to load by default, but it was a pretty painful thing to debug.

So while the legacy-builder / docker driver don't require an explicit --load argument, multi-arch builds are a legitimate use case. Having the CDK explicitly provide a --load argument is a pretty easy fix to cover that use-case and prevent an issue that can be tricky to diagnose.

[pahud]

Yeah I think NodejsFunction at this moment does not allow users to specify additional build arguments like --load to the bundling process under the hood. Before we have a PR to address that, I guess we'll need to use the default driver instead.

We welcome any pull requests for that. Making this a p2 feature request and please help us prioritize with 👍 .

[lmouhib]

I have not tested the work around proposed. However, the reason for using buildx on my end is because I am using mac silicon (m1), and most of the time I build locally fox x86. I would mention, the same code run on an ec2 x86 I connect to remotely to synth CDK.