Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

SyntaxError: Unexpected token '?' on Amazon Linux 2 CodeBuild images (Node 14 required starting at 2.28.0) #20739

Closed
wr-cdargis opened this issue Jun 14, 2022 · 49 comments
Labels
bug This issue is a bug. package/tools Related to AWS CDK Tools or CLI

Comments

@wr-cdargis
Copy link

wr-cdargis commented Jun 14, 2022

Describe the bug

Running command cdk synth with latest aws-cdk from npm produces an error.

Expected Behavior

I made no changes to source. The install step installs the most recent aws-cdk and it seems this error came with the newest version.

Current Behavior

[Container] 2022/06/14 18:50:24 Running command cdk synth
--
663 | /usr/local/lib/node_modules/aws-cdk/lib/index.js:12322
664 | return process.env.CDK_HOME ? path.resolve(process.env.CDK_HOME) : path.join((os.userInfo().homedir ?? os.homedir()).trim() \|\| "/", ".cdk");

SyntaxError: Unexpected token '?'
--
668 | at wrapSafe (internal/modules/cjs/loader.js:915:16)
669 | at Module._compile (internal/modules/cjs/loader.js:963:27)
670 | at Object.Module._extensions..js (internal/modules/cjs/loader.js:1027:10)
671 | at Module.load (internal/modules/cjs/loader.js:863:32)
672 | at Function.Module._load (internal/modules/cjs/loader.js:708:14)
673 | at Module.require (internal/modules/cjs/loader.js:887:19)
674 | at require (internal/modules/cjs/helpers.js:74:18)
675 | at Object.<anonymous> (/usr/local/lib/node_modules/aws-cdk/bin/cdk.js:3:15)
676 | at Module._compile (internal/modules/cjs/loader.js:999:30)
677 | at Object.Module._extensions..js (internal/modules/cjs/loader.js:1027:10)

Reproduction Steps

npm install -g aws-cdk
cdk synth

Possible Solution

No response

Additional Information/Context

I was able to explicitly install version 2.27.0 to confirm this is an issue with 2.28.0 (but I still have this problem later in the self mutate phase where I can't control the version). This was all done on a .NET CDK app. My apologies for the lack of info I know you need more I am just trying to get some attention on this as soon as I can.

CDK CLI Version

aws-cdk@2.28.0

Framework Version

No response

Node.js Version

12.22.12

OS

Linux

Language

.NET

Language Version

No response

Other information

No response

@wr-cdargis wr-cdargis added bug This issue is a bug. needs-triage This issue or PR still needs to be triaged. labels Jun 14, 2022
@github-actions github-actions bot added the package/tools Related to AWS CDK Tools or CLI label Jun 14, 2022
@tmitchel2
Copy link

tmitchel2 commented Jun 14, 2022

Im getting this and its super annoying, it's because the new cdk version 2.28.0 requires nodejs v14+. Unfortunately, Im using AMAZON_LINUX_2_ARM_2 and that build image doesn't even have the option to use a more recent version.

First attempt at workaround

Im attempting to use the 'n' command as below:
installCommands:[n 16.15.1']
However, this still only gets you up to the update-pipeline phase and that still errors out as you can't control the node version in that phase

Second attempt at workaround

Is to pin the cdk version to the previous version rather than use ^2.0.0... Im trying that now. Nope doesn't work as per beton's comment

PLEASE AWS can you upgrade all your build images to support cdk, namely nodejs 14+

@benton
Copy link

benton commented Jun 14, 2022

I think this is because of the NodeJS version 12, as I can reproduce it with Node 12 but not with Node 16.

When using the CDK with Node 12, I get the warning:

npm WARN notsup Unsupported engine for aws-cdk@2.28.0: wanted: {"node":">= 14.15.0"} (current: {"node":"12.22.2","npm":"6.14.13"})
npm WARN notsup Not compatible with your version of node/npm: aws-cdk@2.28.0

The problem is that when deploying to production, the standard CDK Pipeline image (for ARM anyway) is outdated! The latest version of Node in the AmazonLinux2 package repo for ARM is still 12!

Plus, even if you try to pin the version of the CDK in your build, the built-in SelfMutate / UpdatePipeline step still tries to run

npm install -g aws-cdk@2

which then prints the above warning, and proceeds to fail with the error cited in the original post for this issue.

[Container] 2022/06/14 20:05:12  BUILD: 1 commands
[Container] 2022/06/14 20:05:12 Phase complete: DOWNLOAD_SOURCE State: SUCCEEDED
[Container] 2022/06/14 20:05:12 Phase context status code:  Message: 
[Container] 2022/06/14 20:05:12 Entering phase INSTALL
[Container] 2022/06/14 20:05:12 Running command npm install -g aws-cdk@2
/usr/local/bin/cdk -> /usr/local/lib/node_modules/aws-cdk/bin/cdk
npm WARN notsup Unsupported engine for aws-cdk@2.28.0: wanted: 
{
    "node": ">= 14.15.0"
}
 (current: 
{
    "node": "12.22.2",
    "npm": "6.14.13"
}
)    
npm WARN notsup Not compatible with your version of node/npm: aws-cdk@2.28.0
npm WARN optional SKIPPING OPTIONAL DEPENDENCY: fsevents@2.3.2 (node_modules/aws-cdk/node_modules    /fsevents):
npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for fsevents@2.3.2: wanted 
{
    "os": "darwin",
    "arch": "any"
}
 (current: 
{
    "os": "linux",
    "arch": "arm64"
}
)

+ aws-cdk@2.28.0
added 1 package from 1 contributor in 2.327s

[Container] 2022/06/14 20:05:22 Phase complete: INSTALL State: SUCCEEDED
[Container] 2022/06/14 20:05:22 Phase context status code:  Message: 
[Container] 2022/06/14 20:05:22 Entering phase PRE_BUILD
[Container] 2022/06/14 20:05:22 Phase complete: PRE_BUILD State: SUCCEEDED
[Container] 2022/06/14 20:05:22 Phase context status code:  Message: 
[Container] 2022/06/14 20:05:23 Entering phase BUILD
[Container] 2022/06/14 20:05:23 Running command cdk -a . deploy Pipeline --require-approval=never --verbose
/usr/local/lib/node_modules/aws-cdk/lib/index.js:12322
  return process.env.CDK_HOME ? path.resolve(process.env.CDK_HOME) : path.join((os.userInfo().homedir ??         os.homedir()).trim() || "/", ".cdk");
                                                                                                   ^

SyntaxError: Unexpected token '?'
    at wrapSafe (internal/modules/cjs/loader.js:915:16)
    at Module._compile (internal/modules/cjs/loader.js:963:27)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:1027:10)
    at Module.load (internal/modules/cjs/loader.js:863:32)
    at Function.Module._load (internal/modules/cjs/loader.js:708:14)
    at Module.require (internal/modules/cjs/loader.js:887:19)
    at require (internal/modules/cjs/helpers.js:74:18)
    at Object.<anonymous> (/usr/local/lib/node_modules/aws-cdk/bin/cdk.js:3:15)
    at Module._compile (internal/modules/cjs/loader.js:999:30)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:1027:10)

[Container] 2022/06/14 20:05:23 Command did not exit successfully cdk -a . deploy Pipeline --require-    approval=never --verbose exit status 1
[Container] 2022/06/14 20:05:23 Phase complete: BUILD State: FAILED

Please update the aws_cdk.aws_codebuild.LinuxBuildImage.AMAZON_LINUX_2_ARM_2 image ASAP! Production pipelines for ARM are broken!

@tmitchel2
Copy link

Nope my prod CI CD is now toast. As per bentons comment the self mutate phase seems to default to installing latest cdk only... perhaps there is a way to override that behavour but im not aware of it....

@wr-cdargis
Copy link
Author

@tmitchel2 not sure if there is anything we can do if its installing the cdk on its own? I did notice that the CodePipelineProps object has a SelfMutationCodeBuildDefaults with PartialBuildSpec. I haven't given that a shot yet. But the word Partial doesn't have me hopeful.

@tmitchel2
Copy link

tmitchel2 commented Jun 14, 2022

Next attempt from me at workaround is

const pipeline = new CodePipeline(this, 'AppPipeline', {
  pipelineName: 'AppPipeline',
  // selfMutation: false,
  selfMutationCodeBuildDefaults: {
      buildEnvironment: {
          buildImage: LinuxBuildImage.STANDARD_5_0,
      },
  },
  codeBuildDefaults: {
      buildEnvironment: {
          buildImage: LinuxArmBuildImage.AMAZON_LINUX_2_STANDARD_2_0
      },

Ok no that didn't work so Im hoping that turning off selfMutation works...last temporary option i think.

@wr-cdargis
Copy link
Author

wr-cdargis commented Jun 14, 2022

@tmitchel2 my thought was to use the partial build spec to uninstall 2.28.0 and then install 2.27.0 not sure if that is possible. It's not clear to me if the partial runs before or after the command that ends up installing 2.28.0:

Running command npm install -g aws-cdk@2

@tmitchel2
Copy link

@wr-cdargis yup im seeing about that also.... i think i've been mucking up my testing though as im not deploying from my local machine to update the pipeline as this is one of those chicken and egg cdk issues.

@djendrju87
Copy link

What helped me is installing hardcoded version of cdk in buildspec file:
npm install -g aws-cdk@1.160.0
Of course you can use any other version (just not the latest one).

@benton
Copy link

benton commented Jun 14, 2022

Disabling the self-mutation feature of the Pipeline, and then deploying the Pipeline manually, successfully gets past the (non-existent) mutation step.

But apparently the cdk-assets library is also incompatible with Node 12, because the standard Assets Pipeline step then fails.

[Container] 2022/06/14 22:00:04 Entering phase BUILD
--
[Container] 2022/06/14 22:00:04 Running command cdk-assets --path "assembly-Pipeline-Prod/PipelineProdCompute0DF2D855.assets.json" --verbose publish "854cac76b0b0e907cc29a9d7f4fad799f29c830c30488e844e5e00329f2ddfc6:109193998902-us-east-2"
/usr/local/lib/node_modules/cdk-assets/node_modules/@aws-cdk/cloud-assembly-schema/lib/manifest.js:160
if (semver.gt(actual, maxSupported) && !options?.skipVersionCheck) {
^
 
SyntaxError: Unexpected token '.'
at wrapSafe (internal/modules/cjs/loader.js:915:16)
at Module._compile (internal/modules/cjs/loader.js:963:27)
at Object.Module._extensions..js (internal/modules/cjs/loader.js:1027:10)
at Module.load (internal/modules/cjs/loader.js:863:32)
at Function.Module._load (internal/modules/cjs/loader.js:708:14)
at Module.require (internal/modules/cjs/loader.js:887:19)
at require (internal/modules/cjs/helpers.js:74:18)
at Object.<anonymous> (/usr/local/lib/node_modules/cdk-assets/node_modules/@aws-cdk/cloud-assembly-schema/lib/index.js:15:14)
at Module._compile (internal/modules/cjs/loader.js:999:30)
at Object.Module._extensions..js (internal/modules/cjs/loader.js:1027:10)
        
[Container] 2022/06/14 22:00:04 Command did not exit successfully cdk-assets --path "assembly-Pipeline-Prod/PipelineProdCompute0DF2D855.assets.json" --verbose publish "854cac76b0b0e907cc29a9d7f4fad799f29c830c30488e844e5e00329f2ddfc6:109193998902-us-east-2" exit status 1

I see that the x86_64 images are also limited to Node 12.
Does this mean that all standard CDK Pipelines are currently broken?

@tmitchel2
Copy link

I think all but Ubuntu standard:5.0 would be broken? noice.

@tmitchel2
Copy link

This works! :). needs a manual push...

        const pipeline = new CodePipeline(this, 'AppPipeline', {
            pipelineName: 'AppPipeline',
            selfMutationCodeBuildDefaults: {
                partialBuildSpec: BuildSpec.fromObject({
                    phases: {
                       install: {
                        commands: [
                            "n 16.15.1"
                        ]
                       }
                    }
                })
            },
            codeBuildDefaults: {
                buildEnvironment: {
                    buildImage: LinuxArmBuildImage.AMAZON_LINUX_2_STANDARD_2_0
                }
            },

@benton
Copy link

benton commented Jun 14, 2022

Nice - I also see asset_publishing_code_build_defaults, so looks like we may be able to override the image for each step.

Or better yet, maybe just pin the version with the partialBuildSpec

@tmitchel2
Copy link

Yes i just saw that,,, im going to try moving it into codeBuildDefaults as i think that applies to all....

@wr-cdargis
Copy link
Author

Pinning with CodeBuildDefaults worked for me!

@tmitchel2
Copy link

How do you pin?

@wr-cdargis
Copy link
Author

wr-cdargis commented Jun 14, 2022

I moved your partial build spec to CodeBuildDefaults. .NET:

CodeBuildDefaults = new CodeBuildOptions
{
    BuildEnvironment = new BuildEnvironment
    {
        BuildImage = LinuxBuildImage.AMAZON_LINUX_2_3,
    },
    PartialBuildSpec = BuildSpec.FromObject(new Dictionary<string, object>
    {
        {
            "version", "0.2"
        },
        {
            "phases", new Dictionary<string, object>
            {
                {
                    "build", new Dictionary<string, object>
                    {
                        {
                            "commands", new [] { "n 16.15.1" }
                        }
                    }
                }
            }
        }
    }),
},

@tmitchel2
Copy link

tmitchel2 commented Jun 14, 2022

Ah ok i thought you managed to pin the version of cdk... yup looks like that works :)

And the final for others using typescript

        const pipeline = new CodePipeline(this, 'AppPipeline', {
            pipelineName: 'AppPipeline',
            codeBuildDefaults: { 
                buildEnvironment: {
                    buildImage: LinuxArmBuildImage.AMAZON_LINUX_2_STANDARD_2_0
                },
                partialBuildSpec: BuildSpec.fromObject({
                    phases: {
                       install: {
                        commands: [
                            "n 16.15.1"
                        ]
                       }
                    }
                })
            },

@rowilken
Copy link

rowilken commented Jun 16, 2022

@tmitchel2
I tried to adapt your workaround since we are using CdkPipeline instead of CodePipeline in out stack but it seems I can only set the buildImage there but not the partialBuildSpec:

const pipeline = new pipelines.CdkPipeline(this, 'Pipeline', {
  ...
  synthAction: pipelines.SimpleSynthAction.standardNpmSynth({
    ...
    environment: {
      buildImage: codebuild.LinuxBuildImage.STANDARD_5_0,
      computeType: codebuild.ComputeType.LARGE,
    },
  }),
});

Any ideas how to get the workaround you proposed working with CdkPipeline as well?

@tmitchel2
Copy link

I guess the 'workaround' is to upgrade to latest cdk?

@rowilken
Copy link

I don't get your idea - the latest version is 2.28.1 and I am still experiencing the issues described here for 2.28.0, so I think I still need a workaround using another build image and partial build spec as you described before ...

@tmitchel2
Copy link

I presumed you were using an old version because CdkPipeline is old and deprecated now. You just need to change you code to start using CodePipeline instead.

@rowilken
Copy link

rowilken commented Jun 16, 2022

OK, got your idea:

  1. Migrate to the new API
  2. Applying the fix you proposed for CodePipeline
  3. Hoping for AWS to supply a fixed image in the near future and remove the workaround after that

@jwoehrle
Copy link
Contributor

jwoehrle commented Jun 20, 2022

what worked for me is pinning the CDK version from the Pipeline construct. Afterwards do a manual npx cdk deploy MyPipelineStack from my machine (node 16.15.0).


const pipeline = new CodePipeline(this, 'Pipeline', {
            ...
            cliVersion: '2.27.0',
            ...

ganeshnj pushed a commit to aws/aws-lambda-dotnet that referenced this issue Jun 27, 2022
## Motivation
aws/aws-cdk#20739

## Change
Pin CDK version to 1.159.0. Once CodeBuild starts supporting NodeJS 14 LTS, this can be removed.

## Result
Self mutation succeeds.
ganeshnj pushed a commit to aws/aws-lambda-dotnet that referenced this issue Jun 27, 2022
## Motivation
aws/aws-cdk#20739

## Change
Pin CDK version to 1.159.0. Once CodeBuild starts supporting NodeJS 14 LTS, this can be removed.

## Result
Self mutation succeeds.
@peterwoodworth peterwoodworth removed the needs-triage This issue or PR still needs to be triaged. label Jul 1, 2022
@nuhasha
Copy link

nuhasha commented Jul 5, 2022

the issue solved for me, when I upgraded to the latest version of AWS CDK

npm install
npm update -g aws-cdk

after upgrading I got this error:
npm WARN notsup Unsupported engine for aws-cdk@1.162.0: wanted: {"node":">= 14.15.0"} (current: {"node":"12.22.12","npm":"6.14.16"}) npm WARN notsup Not compatible with your version of node/npm: aws-cdk@1.162.0
So, I installed the latest version of Node.js [node-v14.19.3]

@ashishchandr70
Copy link

ashishchandr70 commented Jul 5, 2022

For those using CdkPipeline, you can set the cdkCliVersion on the pipeline props, which also applies it to the UpdatePipeline step. If you're using a 1.x version of the cdk with CdkPipelines api on codebuild AL2 images, you can use this to use an older version of the cdk cli that supports Node 12 (<=1.159.0 or earlier matching the framework version you're using).

For v2 users on AL2, add a step to your buildspec to install nodejs 14 or later using nvm as detailed above for synth steps and self-mutation steps as needed.

We are currently releasing a fix to the ConfirmPermissionsBroadening step to correctly use ubuntu standard 5 so keep an eye out for that. 47b5ca0

I tried it with this code snippet but it fails in the NPM Build stage on the second pass (after NPM_Build runs and succeeds, UpdatePipeline runs and succeeds and the NPM_Build runs again and fails).



[Container] 2022/07/05 23:00:46 Running command npx cdk synth
--
159 | Unexpected token '?'
160 | Subprocess exited with error 1
161 |  
162 | [Container] 2022/07/05 23:00:51 Command did not exit successfully npx cdk synth exit status 1
163 | [Container] 2022/07/05 23:00:51 Phase complete: BUILD State: FAILED
164 | [Container] 2022/07/05 23:00:51 Phase context status code: COMMAND_EXECUTION_ERROR Message: Error while executing command: npx cdk synth. Reason: exit status 1

Our node version is 14.x

const pipeline = new CdkPipeline(this, 'CiCd', {
            pipelineName: `CiCd-${this.namespace}`,
            cloudAssemblyArtifact,
            sourceAction: new codepipelineActions.CodeStarConnectionsSourceAction({
                actionName: "GitHub",
                output: sourceArtifact,                
                connectionArn: "arn:aws:codestar-connections:us-east-xxxx",
                owner: "Owner-Name",
                repo: "Repo-Name",
                branch: this.branch
            }),
            synthAction: SimpleSynthAction.standardNpmSynth({
                actionName: `NPM_Build`,
                sourceArtifact,
                cloudAssemblyArtifact,
                installCommand: `npm run bootstrap`,
                buildCommand: `npx cross-env NODE_ENV=${this.namespace} npm run build`
            }),
            cdkCliVersion: "1.100.0"    // this is an workaround suggested by AWS CDK team per https://github.com/aws/aws-cdk/issues/20739#issuecomment-1166052414
        });

@MrArnoldPalmer
Copy link
Contributor

@ashishchandr70 sorry it's unclear at what phase your pipeline is failing? Also are you running on an AL2 image?

@rix0rrr rix0rrr changed the title aws-cdk:SyntaxError: Unexpected token '?' SyntaxError: Unexpected token '?' on Amazon Linux 2 CodeBuild images (Node 14 required starting at 2.28.0) Jul 7, 2022
@rix0rrr rix0rrr pinned this issue Jul 7, 2022
@rix0rrr rix0rrr removed their assignment Jul 7, 2022
@ashishchandr70
Copy link

@ashishchandr70 sorry it's unclear at what phase your pipeline is failing? Also are you running on an AL2 image?

Hi @MrArnoldPalmer - please see this image. On triggering the pipeline, the Build stage and UpdatePipeline stages work. The UpdatePipeline is set to self mutate and it retriggers the build stage for a second pass. That is when it fails again.

Our build image was set to version standard:4.0 but I had to manually update it to aws/codebuild/standard:5.0, which is an Ubuntu image.

image

@CodeWithOz
Copy link

@MrArnoldPalmer I tried setting the cdkCliVersion to the version you mentioned but it didn't make a difference. This is the config I have as my pipeline config:

    const sourceArtifact = new Artifact()
    const buildArtifact = new Artifact()
    const project = new PipelineProject(stack, `BuildImageProject`, {
      environment: {
        buildImage: LinuxBuildImage.STANDARD_5_0,
        privileged: true,
      },
      buildSpec: BuildSpec.fromObject({
        version: "0.2",
        phases: {
          pre_build: {
            commands: [
              "echo Logging in to Amazon ECR...",
              "aws ecr get-login-password --region $AWS_DEFAULT_REGION | docker login --username $USERNAME --password-stdin $REPOSITORY_URI",
              'TAG="$(echo $CODEBUILD_RESOLVED_SOURCE_VERSION | head -c 8)"',
              'IMAGE_URI="${REPOSITORY_URI}"',
              'IMAGE_URI_TAGGED="${REPOSITORY_URI}:${TAG}"',
              'IMAGE_URI_LATEST="${REPOSITORY_URI}:latest"',
              'CONTAINER_NAME="${APP_NAME}"',
            ],
          },
          build: {
            commands: ['docker build -t "$IMAGE_URI_TAGGED" -t "$IMAGE_URI_LATEST" .'],
          },
          post_build: {
            commands: [
              'docker push "$IMAGE_URI"',
              `printf '[{"name":"%s","imageUri":"%s"}]' "$CONTAINER_NAME" "$IMAGE_URI_LATEST" > ${imageFileName}`,
            ],
          },
        },
        artifacts: {
          files: [imageFileName],
        },
      }),
    })

    const buildActionConfig = new CodeBuildAction({
      actionName: `BuildImage`,
      project: project,
      input: sourceArtifact,
      checkSecretsInPlainTextEnvVariables: true,
      environmentVariables: {
        AWS_DEFAULT_REGION: {
          type: BuildEnvironmentVariableType.PLAINTEXT,
          value: stack.region,
        },
        REPOSITORY_URI: {
          type: BuildEnvironmentVariableType.PLAINTEXT,
          value: this.ecrRepository.repositoryUri,
        },
        APP_NAME: {
          type: BuildEnvironmentVariableType.PLAINTEXT,
          value: serviceName,
        },
      },
      outputs: [buildArtifact],
      type: CodeBuildActionType.BUILD,
    })

/* --- the pipeline config --- */
{
        pipelineName: `${serviceName}-Pipeline`,
        selfMutating: false,
        crossAccountKeys: false,
        cloudAssemblyArtifact: buildArtifact,
        sourceAction: new GitHubV2SourceAction({
          actionName: "GitHub",
          owner: owner,
          repo: repo,
          oauthToken: SecretValue.secretsManager(...),
          output: sourceArtifact,
          branch: branch,
          filters: filters ? filters : defaultFilters,
        }),
        synthAction: buildActionConfig,
        cdkCliVersion: '1.159.0',
}

Does that give an idea what could be wrong? Please let me know if there's more info I can provide.

@CodeWithOz
Copy link

Is it possible that my old config is getting cached somewhere and that cache is not being invalidated when I create a new build? I'm looking at the build details for the failed and I can see that the build spec and environment don't actually correspond to what's in my code. Any ideas how I can investigate this? Please bear with me I'm filling in for someone else and this is not my area of expertise.

@vdanniel
Copy link

This breaking a few related projects like the Simple Forecast Solution. For now we are fixing the version to the pervious version or cdk2.28.

@ashishchandr70
Copy link

@ashishchandr70 sorry it's unclear at what phase your pipeline is failing? Also are you running on an AL2 image?

Hi @MrArnoldPalmer - please see this image. On triggering the pipeline, the Build stage and UpdatePipeline stages work. The UpdatePipeline is set to self mutate and it retriggers the build stage for a second pass. That is when it fails again.

Our build image was set to version standard:4.0 but I had to manually update it to aws/codebuild/standard:5.0, which is an Ubuntu image.

image

Any insight on this from the AWS team?

@MrArnoldPalmer
Copy link
Contributor

@CodeWithOz I don't believe that is possible. Are you having similar failures to @ashishchandr70 where build step is failing after sulf-mutating?

@ashishchandr70 cdkCliVersion dictates the version inside of the self-mutating a synth actions. For your synth action config pass synthCommand to explicitly override the cdk version called by npx IE npx aws-cdk@version synth. However, if the project you are within takes a dependency on aws-cdk in it's package.json and that dependency points to the right version (1.159.0) then npx cdk should call that if it's installed correctly, so I believe it's possible that the package.json isn't pointing to the right cdk version (or has a ^) on the version.

@ashishchandr70
Copy link

@CodeWithOz I don't believe that is possible. Are you having similar failures to @ashishchandr70 where build step is failing after sulf-mutating?

@ashishchandr70 cdkCliVersion dictates the version inside of the self-mutating a synth actions. For your synth action config pass synthCommand to explicitly override the cdk version called by npx IE npx aws-cdk@version synth. However, if the project you are within takes a dependency on aws-cdk in it's package.json and that dependency points to the right version (1.159.0) then npx cdk should call that if it's installed correctly, so I believe it's possible that the package.json isn't pointing to the right cdk version (or has a ^) on the version.

Thank you @MrArnoldPalmer . Our package.json does depend on aws-cdk and it points to version "1.100.0" so that is a hard dependency.

Our pipeline, like I pointed out in my earlier response, does (now, after reading one of your replies on this issue) have a cdkCliVersion also set to "1.100.0" and it fails. See below:


const pipeline = new CdkPipeline(this, 'CiCd', {
            pipelineName: `CiCd-${this.namespace}`,
            cloudAssemblyArtifact,
            sourceAction: new codepipelineActions.CodeStarConnectionsSourceAction({
                actionName: "GitHub",
                output: sourceArtifact,                
                connectionArn: "arn:aws:codestar-connections:us-east-xxxx",
                owner: "Owner-Name",
                repo: "Repo-Name",
                branch: this.branch
            }),
            synthAction: SimpleSynthAction.standardNpmSynth({
                actionName: `NPM_Build`,
                sourceArtifact,
                cloudAssemblyArtifact,
                installCommand: `npm run bootstrap`,
                buildCommand: `npx cross-env NODE_ENV=${this.namespace} npm run build`
            }),
            cdkCliVersion: "1.100.0"    // this is an workaround suggested by AWS CDK team per https://github.com/aws/aws-cdk/issues/20739#issuecomment-1166052414
        });

I did notice that the UpdatePipeline (Self Mutate) and the Assets stages of our pipeline are using default buildspecs so the UpdatePipeline buildspec looks like this

{
  "version": "0.2",
  "phases": {
    "install": {
      "commands": "npm install -g aws-cdk"
    },
    "build": {
      "commands": [
        "cdk -a . deploy Pipeline-dev --require-approval=never --verbose"
      ]
    }
  }
}

As you can see, the install command will probably install the latest version of aws-cdk.

Similarly, the Assets stage also has a default buildspec:

{
  "version": "0.2",
  "phases": {
    "install": {
      "commands": "npm install -g cdk-assets"
    },
    "build": {
      "commands": [
        "cdk-assets --path \"assembly-Pipeline-dev-dev/PipelinedevAppStack176E047A.assets.json\" --verbose publish \"5b6d1a82a9dcedb3aab07fdaad79fecab765374fb1667d28ba3c1a02d760eceb:current_account-current_region\""
      ]
    }
  }
}

I think this may be my issue and I will need to not rely on the default buildspec but pass a custom buildspec via our code.

@MrArnoldPalmer
Copy link
Contributor

So here that the UpdatePipeline action buildspec is installing npm cdk with the installVersion suffix appended to it which is passed from the CdkPipeline props

When creating a minimal pipeline stack like so:
export class Issue20739Stack extends cdk.Stack {
  constructor(scope: cdk.Construct, id: string, props?: cdk.StackProps) {
    super(scope, id, props);

    const sourceArtifact = new codepipeline.Artifact();
    const cloudAssemblyArtifact = new codepipeline.Artifact();
    const sourceAction = new codepipelineActions.GitHubSourceAction({
      actionName: 'Source',
      output: sourceArtifact,
      owner: 'mrarnoldpalmer',
      repo: 'my-repo',
      oauthToken: cdk.SecretValue.secretsManager('mygithubtoken'),
    });

    new pipelines.CdkPipeline(this, 'Pipeline', {
      cdkCliVersion: '1.100.0',
      cloudAssemblyArtifact,
      sourceAction,
      synthAction: pipelines.SimpleSynthAction.standardNpmSynth({
        sourceArtifact,
        cloudAssemblyArtifact,
        environment: {
          privileged: true,
        },
      }),
    });
  }
}

The output template shows the buildspec output for the UpdatePipeline step to be

          "BuildSpec": "{\n  \"version\": \"0.2\",\n  \"phases\": {\n    \"install\": {\n      \"commands\": \"npm install -g aws-cdk@1.100.0\"\n    },\n    \"build\": {\n      \"commands\": [\n        \"cdk -a . deploy Issue20739Stack --require-approval=never --verbose\"\n      ]\n    }\n  }\n}",
          "Type": "CODEPIPELINE"
        },

And without the 1.100.0 suffix when removing the cdkCliVersion prop.

Did you manually deploy the pipeline stack from your machine (or another that has Node 14 +) after making the change?

@ashishchandr70
Copy link

ashishchandr70 commented Jul 14, 2022

@MrArnoldPalmer I am using the CodePipeline so the pipeline is only being deployed there, not from my local machine. It fails even at the Build stage.

The image we were using was aws/codebuild/standard:4.0. I had to manually change this to aws/codebuild/standard:5.0 for the pipeline to work, which means we have to manually intervene for each deployment.

I also wanted to re-state that we had upgraded our Dockerfiles to use Node.js 14 since the initial reports around this issue pertained to removal of support (from AWS) for Node 12 and below. So all of our images are building with Node.js 14.

This is my code

export class PipelineStack extends Stack {
    public readonly namespace: string;
    public readonly branch: string;
    public readonly isPreProd: boolean;

    constructor(scope: Construct, id: string, props?: PipelineStackProps) {
        super(scope, id, props);

        this.branch = props?.branch || 'develop';
        this.namespace = props?.namespace ?? 'dev';
        this.isPreProd = props?.isPreProd ?? this.namespace !== 'prod';

        const sourceArtifact = new codepipeline.Artifact();
        const cloudAssemblyArtifact = new codepipeline.Artifact();

        const pipeline = new CdkPipeline(this, 'CiCd', {
            pipelineName: `CiCd-${this.namespace}`,
            cloudAssemblyArtifact,
            sourceAction: new codepipelineActions.CodeStarConnectionsSourceAction({
                actionName: "GitHub",
                output: sourceArtifact,
                connectionArn: "<some arn>",
                owner: "me",
                repo: "my-repo",
                branch: this.branch
            }),
            synthAction: SimpleSynthAction.standardNpmSynth({
                actionName: `NPM_Build`,
                sourceArtifact,
                cloudAssemblyArtifact,
                installCommand: `npm run bootstrap`,
                buildCommand: `npx cross-env NODE_ENV=${this.namespace} npm run build`,
                environment:{
                    privileged: true
                }
            }),
            cdkCliVersion: "1.100.0"    // this is an workaround suggested by AWS CDK team per https://github.com/aws/aws-cdk/issues/20739#issuecomment-1166052414
        });

        //Define the BuildPipeline stage that will deploy the KronoApp
        const appStage = pipeline.addApplicationStage(
            new KronoPipelineStage(this, `${this.namespace}`, {
                namespace: this.namespace,
                isPreProd: this.isPreProd,
                region: this.region,
                account: this.account
            })
        );

    }
}

@MrArnoldPalmer
Copy link
Contributor

MrArnoldPalmer commented Jul 14, 2022

@ashishchandr70 all of the above posted fixes require a manual deploy as self-update can't run synth successfully to make the changes to the code to change the cdk cli version or update the containers to standard:5.0, after that deploy it will be able to self-update as usual again.

Closing this out as codebuild has made AL2:4.0 available which has NodeJS 16 as standard, so users wanting to continue to use AL2 with the latest CDK CLI should be able to. If you get stuck with this error in self-update, update your codebuild images and run a deploy of the pipeline stack to get the updates.

@github-actions
Copy link

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.

@tmitchel2
Copy link

@MrArnoldPalmer AL2_4 is only x86 and not arm so ideally shouldn't be fully closed out.

@MrArnoldPalmer
Copy link
Contributor

Since there is no information on when NodeJS 14 will be available on the provided ARM images, I think our best bet would be to write some logic to detect when a cdk pipeline is building/self-updating/etc on an arm image and automatically add the steps to install node14 to the buildspec. @tmitchel2 wanna open a new issue for that? You may also have a better alternative in mind.

@tmitchel2
Copy link

My workaround as per above works for me and is fine enough as a workaround, however, I just wanted to make it clear that the Arm images are getting left behind a bit and I guess still need to be addressed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug This issue is a bug. package/tools Related to AWS CDK Tools or CLI
Projects
None yet
Development

No branches or pull requests