(aws-cdk-lib/pipelines): CDKPipelines source / input change from v1 to v2 doesn't allow 5 sources

[natalie-white-aws]

Describe the bug

Multiple teams have reported that migrating from CDK v1 to v2 when using CDKPipelines results in no longer being able to use 5 sources for CodeBuild. The symptom is that at synth time, they get a "Build/CodeBuild cannot have more than 5 input artifacts" error from the core CodeBuild service, which is a hard limit.

The approach of using additional_inputs for multiple sources is suggested in this GitHub issue:
#14493
but it's not quite what was previously allowed with the Source add action functionality.

Expected Behavior

Minimal changes to the CDKPipeline in the v1 to v2 upgrade, or prescriptive guidance of how to do so for pipelines that have more than one input.

Current Behavior

After upgrading to CDK v2 and attempting to configure the same multi-source Source actions via the CodeBuild additional_inputs list, the symptom is that at synth time, they get a "Build/CodeBuild cannot have more than 5 input artifacts" error from the core CodeBuild service, which is a hard limit.

Reproduction Steps

Steps to reproduce:

1. Start with a CDK v1 app using CDKPipelines and 5 sources (via source.add_action against the CDKPipeline's Source stage).
2. Upgrade that app to CDK v2, which requires using the CodeBuild additional_inputs mechanism inherited from the ShellStep interface.
3. At synth time, you'll get the error "Build/CodeBuild cannot have more than 5 input artifacts" because the CodeBuild provisioning requires an input parameter, which is added to the 5 additional inputs that were originally in the v1 Source action.

Possible Solution

I think this may be due to the removal of the Source stage entirely in the move to v2, subsequent confusion between ShellSteps and CodeBuild steps, and the lack of documentation / examples for / validation that a CodeBuild step can't have more than 5 inputs (because the CodeBuild service itself has that hard limit.

My requested changes would be to

1. Allow a CodeBuild step to take more than one input natively, rather than relying on the additional_input structure inherited from ShellStep, or enforce only 4 additional inputs since the CodeBuild step requires one input in its required parameter list.
2. Update the CDKPipelines documentation to make this limit more clear.
3. Create examples that show how to use CDKPipelines when building a project that needs more than one input source.

What #3 looks like depends on whether If #1 is possible. If #1 is not possible, is our prescriptive guidance to configure one or more ShellSteps before the CodeBuild step to accumulate inputs first, then build?

Either way, we can still do #2 and #3 to improve the developer experience. Thanks!

Additional Information/Context

Need to ensure that whatever change is made whether the pipeline can be triggered by a change in any of the multiple inputs, rather than just the one specified in the required input parameter.

CDK CLI Version

2.93.0

Framework Version

No response

Node.js Version

16.20.0

OS

Linux

Language

Python

Language Version

Python version 3.9.16

Other information

No response

[hoegertn]

Hi, thanks for this feedback.

Can you please elaborate on which v1 version you were? I suspect that the change you are hitting is not related to CDk v1 vs v2 but the GA of CDKPipelines as this changed the interface compared to the alpha.

The class CDKPipeline does not exist anymore but is CodePipeline. This is a different API.

[natalie-white-aws]

Previous version was CDK 1.116.0

[indrora]

Indeed this appears to be a regression in an L2 construct: The underlying CloudFormation has no documented maximums: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-codepipeline-pipeline-stages.html

[hoegertn]

@natalie-white-aws Thanks for the info. That was exactly the version where CDK pipelines went GA. Did you use the class CodePipeline (https://github.com/aws/aws-cdk/blob/v1-main/packages/%40aws-cdk/pipelines/lib/codepipeline/codepipeline.ts#L282) or the legacy CdkPipeline class (https://github.com/aws/aws-cdk/blob/v1-main/packages/%40aws-cdk/pipelines/lib/legacy/pipeline.ts#L215).

The latter was never meant to be GA and was kept for compatibility but officially deprecated in 1.129.0

[hoegertn]

Indeed this appears to be a regression in an L2 construct: The underlying CloudFormation has no documented maximums: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-codepipeline-pipeline-stages.html

I cannot follow? CDK pipelines is not an L2 but an L3 construct, not to be confused with the CodePipeline L2 in the aws-codepipeline namespace.

[indrora]

@natalie-white-aws Could you please elaborate on a few things?

• What versions of what libraries are you using?
• We advise that teams migrating from a very old version of CDKv1 perform two migrations: First to the last supported version of CDKv1, then to the current CDKv2. Have you performed this double-migration yet?
• What exactly are you attempting to accomplish?

[natalie-white-aws]

What versions of what libraries are you using?
Can you be more specific? Happy to do a deep dive with the team working on this if you want to set it up (find me on Slack). :)

We advise that teams migrating from a very old version of CDKv1 perform two migrations: First to the last supported version of CDKv1, then to the current CDKv2. Have you performed this double-migration yet?
No, the team is doing a direct migration; but if the new CDK Pipelines L2/L3 doesn't handle more than 5 inputs, this seems like a moot point.

What exactly are you attempting to accomplish?
Upgrade from CDK v1 to CDKv2 with the same Pipeline structure with build 6 inputs.

[natalie-white-aws]

Indeed this appears to be a regression in an L2 construct: The underlying CloudFormation has no documented maximums: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-codepipeline-pipeline-stages.html

The CodeBuild additional inputs limit is documented in CodePipeline even though the limit for Secondary Sources in CodeBuild is 12. I think the primary reason the limit isn't enforce in CloudFormation is because the Action is abstracted to cover all action types, and each type has its own limits.

The issue isn't that CloudFormation isn't enforcing the service limit, it's that the way CDKPipelines handles inputs changed from a Source Stage with multiple Actions (which is only limited by the number of actions in a stage) to a Build action, which does have this limit.