AWS CodePipeline V2 Support

[angel3cu]

Describe the feature

The L2 construct Pipeline from namespace aws_codepipeline does not support the new V2 pipelines. This is crucial for running more advanced workflows that involve variables or get triggered on tags instead of pushing to a branch.

Use Case

We want to run the pipelines when a tag is created instead of when a commit is pushed to a branch. Pipeline V2 is the only option.

We could use the L1 construct that is already provided, but this would:

• Require a complete rewrite of our pipeline definition.
• Make our code much more complicated and difficult to maintain.

Proposed Solution

Add support for V2 pipelines in the existing Pipeline construct. Ensure that there is an option to trigger the pipeline on a tag, as supported in the AWS UI and CloudFormation.

Other Information

No response

Acknowledgements

• I may be able to implement this feature request
• This feature might incur a breaking change

CDK version used

2.114.1

Environment details (OS name and version, etc.)

macOS (the CDK Code runs on AWS CodePipeline)

[ymhiroki]

+1

The pull request #28538 implements this feature.

[pahud]

Awesome and thanks for the pull request.

[go-to-k]

Hi, The PR with the features has been merged.

[github-actions]

This issue has received a significant amount of attention so we are automatically upgrading its priority. A member of the community will see the re-prioritization and provide an update on the issue.

[andreprawira]

@go-to-k @pahud do you guys know if there is V2 support for Pipeline L3 Construct? Or any way to integrate V2 features while still using L3 Construct? The L3 Construct that i meant is this . Currently, we already have our pipelines defined

pipeline = pipelines.CodePipeline(
            self,
            "l3-pipeline",
            pipeline_name="l3-pipeline",
            code_build_defaults=pipelines.CodeBuildOptions(
                build_environment=codebuild.BuildEnvironment(
                    build_image=codebuild.LinuxBuildImage.STANDARD_6_0,
                    privileged=True,
                    compute_type=codebuild.ComputeType.LARGE,
                ),
                partial_build_spec=codebuild.BuildSpec.from_object(
                    {
                        "env": {
                            "git-credential-helper": "yes",
                        }
                    }
                ),
            ),
            synth=pipelines.CodeBuildStep(
                "Synth",
                input=pipelines.CodePipelineSource.connection(
                    repo_string="my-bitbucket-repo",
                    branch="master",
                    connection_arn="my-bitbucket-dummy-connection-arn",
                    code_build_clone_output=True,
                    trigger_on_push=True,
                ),
                partial_build_spec=codebuild.BuildSpec.from_object(
                    {
                        "phases": {
                            "pre_build": {
                                "commands": [
                                    "ls -la",
                                ],
                            },
                        }
                    }
                ),
                commands=[
                    "npm install -g aws-cdk",
                    "pip install -r requirements.txt",
                    "npx cdk synth -c customer=$customer",
                ],
                env={"customer": props.infra.name},
                role_policy_statements=[
                    iam.PolicyStatement(
                        actions=["sts:AssumeRole"],
                        resources=["*"],
                        conditions={
                            "StringEquals": {
                                "iam:ResourceTag/aws-cdk:bootstrap-role": "lookup"
                            }
                        },
                    )
                ],
            ),
            self_mutation=True,
            cross_account_keys=True,
            use_change_sets=False,
            docker_enabled_for_synth=True,
            docker_enabled_for_self_mutation=True,
        )

And what we want is to use V2 feature to filter based on branch

triggers=[codepipeline.TriggerProps(
                provider_type=codepipeline.ProviderType.CODE_STAR_SOURCE_CONNECTION,
                git_configuration=codepipeline.GitConfiguration(
                    source_action=source_action,
                    push_filter=[codepipeline.GitPushFilter(
                        tags_excludes=["exclude1", "exclude2"],
                        tags_includes=["include*"]
                    )]
                )
            )]

But we dont know how to integrate the 2 together, I'm assuming it is not yet exist, and that there needs to be a PR raised to include V2 features for L3 pipeline construct?

[go-to-k]

@andreprawira

At this time, the parameter to specify V2 type is not yet available in the pipelines module, but you can pass your existing pipeline with V2 type to props.
https://github.com/aws/aws-cdk/blob/main/packages/aws-cdk-lib/pipelines/README.md#using-an-existing-aws-codepipeline

It is good to refer to following parameters in order to match the original pipeline settings in the pipelines module.
https://github.com/aws/aws-cdk/blob/main/packages/aws-cdk-lib/pipelines/lib/codepipeline/codepipeline.ts#L468-L480

[andreprawira]

i was able to create a V2 pipeline with GitPushFilter and from the docs and it says "Git tags is the only supported event type" so do you guys know when is Branch going to be added as another supported event type? @go-to-k @pahud @kaizencc

we need this feature asap

[go-to-k]

Please wait for this PR. Reacting to the linked issue may increase the priority of your review.

#29127

[andreprawira]

I tried to use codepipeline.GitPushFilter on my code and i can cdk synth succesfully but when i do cdk deploy i get an error Triggers for connection must reference a CodeStarSourceConnection action. Note that when you specify a trigger, you cant specify more than one trigger per source action. (Service: AWSCodePipeline, Status Code 400) Below is my code

from aws_cdk import (
    Stack,
    Environment,
    aws_codebuild as codebuild,
    aws_iam as iam,
    pipelines as pipelines,
    aws_codepipeline as codepipeline,
    aws_codepipeline_actions as codepipeline_actions,
)

        source_output = codepipeline.Artifact()
        l2_pipeline = codepipeline.Pipeline(
            self,
            "l2-codepipeline",
            pipeline_type=codepipeline.PipelineType.V2,
            pipeline_name="l2-cdk-pipeline",
            cross_account_keys=True,
            triggers=[
                codepipeline.TriggerProps(
                    provider_type=codepipeline.ProviderType.CODE_STAR_SOURCE_CONNECTION,
                    git_configuration=codepipeline.GitConfiguration(
                        source_action=codepipeline_actions.CodeStarConnectionsSourceAction(
                            action_name="BitBucket_Source",
                            owner="Me",
                            repo="my-dummy-repo",
                            branch="main",
                            output=source_output,
                            connection_arn="my-dummy-codestar-connection-arn",
                        ),
                        push_filter=[
                            codepipeline.GitPushFilter(
                                tags_excludes=["v1.0", "version/**"],
                                tags_includes=["b/**"],
                            )
                        ],
                    ),
                )
            ],
        )

        # Define the pipeline
        pipeline = pipelines.CodePipeline(
            self,
            "l3-cdk-pipeline",
            code_pipeline=l2_pipeline,
            code_build_defaults=pipelines.CodeBuildOptions(
                build_environment=codebuild.BuildEnvironment(
                    build_image=codebuild.LinuxBuildImage.STANDARD_6_0,
                    privileged=True,
                    compute_type=codebuild.ComputeType.LARGE,
                ),
                partial_build_spec=codebuild.BuildSpec.from_object(
                    {
                        "env": {
                            "git-credential-helper": "yes",
                        }
                    }
                ),
            ),
            synth=pipelines.CodeBuildStep(
                "Synth",
                # input=pipelines.CodePipelineSource.connection(
                #     repo_string="my-dummy-repo",
                #     branch="main",
                #     connection_arn="my-dummy-codestar-connection-arn",
                #     code_build_clone_output=True,
                #     trigger_on_push=True,
                # ),
                additional_inputs={
                    "ui-repository": pipelines.CodePipelineSource.connection(
                        repo_string="my-dummy-ui-repo",
                        branch="main",
                        connection_arn="my-dummy-codestar-connection-arn",
                        code_build_clone_output=True,
                        trigger_on_push=True,
                    ),
                },
                partial_build_spec=codebuild.BuildSpec.from_object(
                    {
                        "phases": {
                            "pre_build": {
                                "commands": [
                                    "ls -la",
                                ],
                            },
                        }
                    }
                ),
                commands=[
                    "npm install -g aws-cdk",
                    "pip install -r requirements.txt",
                    "npx cdk synth -c customer=$customer",
                ],
                env={"customer": props.infra.name},
                role_policy_statements=[
                    iam.PolicyStatement(
                        actions=["sts:AssumeRole"],
                        resources=["*"],
                        conditions={
                            "StringEquals": {
                                "iam:ResourceTag/aws-cdk:bootstrap-role": "lookup"
                            }
                        },
                    )
                ],
            ),
            self_mutation=True,
            use_change_sets=False,
            docker_enabled_for_synth=True,
            docker_enabled_for_self_mutation=True,
        )

Has anyone gotten that error before and know what i'm missing? Thx

[go-to-k]

I see. The CodePipeline construct in pipelines module generates a source action internally. On the other hand, the Trigger feature requires a source action, but the action in the CodePipeline in the pipelines is not be exposed. So at this time, it is difficult to use triggers in pipelines unless you use complicated escape hatch.

If you want to solve this problem as a feature of the CDK, it might be better if you could create a separate issue with specific use cases.

[dguisinger]

It has been a couple weeks since the last update on this issue. Any word on when we can see this implemented?

[go-to-k]

This issue should be closed since it has already been merged and released.

#28538

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.

[CarlosPintoWindsor]

@andreprawira did you solve it ?

[moltar]

@CarlosPintoWindsor See #29124 (comment)