Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(pipelines): refactor source stage and artifact handling #11401

Closed
wants to merge 2 commits into from
Closed

feat(pipelines): refactor source stage and artifact handling #11401

wants to merge 2 commits into from

Conversation

hoegertn
Copy link
Contributor

refs #10872

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license

@gitpod-io
Copy link

gitpod-io bot commented Nov 10, 2020

@hoegertn hoegertn marked this pull request as draft November 10, 2020 22:34
@hoegertn
Copy link
Contributor Author

@rix0rrr as discussed the PR draft for the source action changes

What I did:

  • Created an ISource interface to provide sources to CDK Pipelines
  • Created implementations for all supported source providers
  • Added autogeneration for sourceArtifact and cloudAssemblyArtifact
  • Modified the SynthAction to accept the artifacts later in the process for auto-management

Feedback is much appreciated

@aws-cdk-automation
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: AutoBuildProject6AEA49D1-qxepHUsryhcu
  • Commit ID: 535509f
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@hoegertn
Copy link
Contributor Author

The moment this is supported in CFN/CDK I would set this as default in the CodeCommitSource for CDK Pipelines.
https://aws.amazon.com/de/about-aws/whats-new/2020/11/aws-codepipeline-source-action-aws-codecommit-supports-git-clone/

This si what developers expect when accessing the source artifact.


/**
* The CodePipeline action build and synthesis step of the CDK app
*
* @default - Required unless `codePipeline` or `sourceAction` is given
*/
readonly synthAction?: codepipeline.IAction;
readonly synthAction?: ISynthAction;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it would be okay to call this synth: ISynth as well.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure, will handle this when we discussed the rest.

@@ -21,19 +23,28 @@ export interface CdkPipelineProps {
*
* @default - Required unless `codePipeline` is given
*/
readonly sourceAction?: codepipeline.IAction;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm SUPER debating whether it's worth to hard-break backwards compatibility, or whether we should be doing our best to maintain it in order to give people a reasonable migration path off of the current API.

I think I'd prefer the deprecation/migration path, but on the other hand I'm not sure exactly what that should or would look like, and I think the implementation will become too complicated.

Maybe we stage a grander refactoring of CDK Pipelines on a separate, longer-lived branch, until we're ready to release it?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was thinking of a backward-compatible way but I think this will make the code unreadable. The reason for dev preview is to experiment and break APIs when needed imho.

*/
readonly cloudAssemblyArtifact: codepipeline.Artifact;
readonly sourceArtifact?: codepipeline.Artifact;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The goal is to auto-derive these artifacts. So we definitely shouldn't be adding them. I also think we don't necessarily want to give users the power to override them here.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We need to if we want users to be able to provide a custom CodePipeline with existing source stage.

//
}

public provideSourceAction(sourceArtifact: codepipeline.Artifact): codepipeline.IAction {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think instead of provideSourceAction I'd be in favor of a more general addToPipeline(pipelineSourceProps): void; and instead of returning a single Action the ISource is free to do WHATEVER to mutate the pipeline. It can add a single Action, it can add twenty.

Feels like it gives more flexibility.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Honestly, I am not sure if flexibility is what I want to achieve here. I want the user to provide a source not to do arbitrary stuff with the pipeline.

* @param sourceArtifact The source artifact of the CodePipeline
* @param cloudAssemblyArtifact The artifact where the CloudAssembly should be emitted
*/
configureArtifacts(sourceArtifact: codepipeline.Artifact, cloudAssemblyArtifact: codepipeline.Artifact): void;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Feels like this could also have been an addToPipeline() method of some sort.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am a big fan of concrete APIs that exactly tell the user what to do, but we can switch to a more generic approach.

/**
* The BitBucket repo slug. (e.g. "aws/aws-cdk")
*/
readonly slug: string;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is not really a user-friendly term. Why not just repo ?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am not attached to the name. I just used the working from BB/GH

* @example 'arn:aws:codestar-connections:us-east-1:123456789012:connection/12345678-abcd-12ab-34cdef5678gh'
* @see https://docs.aws.amazon.com/codepipeline/latest/userguide/connections-create.html
*/
readonly connectionArn: string;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wow, is this required? That looks pretty bad :/

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That is the way BB is currently configured and GH will move to this too with GithubSourcev2

/**
* The GitHub repo slug. (e.g. "aws/aws-cdk")
*/
readonly slug: string;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same here.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

see above

*
* @see https://docs.aws.amazon.com/codepipeline/latest/userguide/GitHub-create-personal-token-CLI.html
*
* @default - use a SecretsManager secret called GitHub
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ooo. On the one hand I like this level of defaulting, but I wonder if it's too much?

It'll be easy to forget and the error will be completely non-descriptive (I believe it'll be something like "Internal Error").

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe there is a way to check if it exists?

* CodePipeline action to synthesize CDK applications.
* The source artifact and the cloud assembly artifact are lazily initialized by the CDK pipeline.
*/
export interface ISynthAction extends codepipeline.IAction {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think this needs to extend an IAction anymore.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch. It definitely should not be an IAction

@rix0rrr
Copy link
Contributor

rix0rrr commented Nov 16, 2020

I think it's worth me investing some effort into making the underlying Action classes mutable where it makes sense, to make it easier to implement the classes here.

Will start on that right now.

@hoegertn
Copy link
Contributor Author

I think it's worth me investing some effort into making the underlying Action classes mutable where it makes sense, to make it easier to implement the classes here.

Will start on that right now.

What exactly do you want to make easier by this? Do you want to get rid of the factory style?

@rix0rrr
Copy link
Contributor

rix0rrr commented Jun 17, 2021

Superseded by #12326

@rix0rrr rix0rrr closed this Jun 17, 2021
@hoegertn hoegertn deleted the pipeline-sources branch July 17, 2021 16:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants