cdk pipelines: Pipeline in infinite loop / always mutates and deploys

[miles-po]

Describe the bug

Seeing a serious issue where my CodePipeline always triggers a pipeline mutation. It continues building assets for each invocation but puts the Source and Builds steps in a loop. Something’s broken, but I don’t know where to look to unbreak it. I have deleted the stack and recreated, but it continues to happen. Something in the common CDK config? This started happening when I enabled caching in another branch, but I have since reverted the change. There is no caching on the pipeline anymore. There is no custom hashing or signature logic.

Expected Behavior

Once-through per push to the repository (perhaps two on an actual pipeline stack change when a mutation is warranted).

Current Behavior

Source ->
Build ->
UpdatePipeline -> Assets, etc.
Source ->
Build ->
UpdatePipeline -> Assets, etc.
Source ->
Build ->
UpdatePipeline -> Assets, etc.

Reproduction Steps

Don't know for sure how to reproduce. Perhaps overcache with custom and then turn off custom caching?

Possible Solution

No response

Additional Information/Context

The pipeline is not complex. GitHub source with token from SecretsManager. npm ci in the CDK directory and cdk synth during build. In a monorepo where the CDK code is in a cdk subdirectory. cdk/cdk.out is the primary output directory relative to repo root. Large ARM64 compute type for the build/synth step. Normal ARM64 instances for all other steps. One stage with an app stack. No direct filesystem calls. Nothing weird. Just CDK API.

All build logs appear to be identical. No clear reason why a stack mutation would take place.

CDK CLI Version

2.87.0

Framework Version

No response

Node.js Version

18.16.1

OS

CodePipeline/CodeBuild's Linux

Language

Typescript

Language Version

~4.9.5

Other information

No response

[miles-po]

Discovered the problem. Was tagging the stacks with a last modified timestamp. This obviously changed on every invocation. This was why it constantly cycled. Hopefully my error will help another dev solve their problem in the future.

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.

[Jurajzovinec]

@miles-po thanks for sharing your research and I guess it may relate to my problem. My code pipeline is in a constant self-update mode when I this chunk of code handling env variables:

    for (const name of APP_STAGE_ENV_VARIABLES) {
      const secretName: AppStageSecretName = `${stage}-${name}`;
      envVariables[name] = cdk.SecretValue.secretsManager(secretName).toString();
    }

Do you think it relates? Thank you in advance for your help

[miles-po]

@Jurajzovinec If I had to guess, I'd say one of your environment variables keeps changing. Look for one with either a timestamp or a unique id like a build job id.