Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

TEP-0118: Enable pipeline to handle matrix include params #6248

Conversation

EmmaMunley
Copy link
Contributor

@EmmaMunley EmmaMunley commented Feb 28, 2023

[TEP-0090: Matrix] introduced Matrix to the PipelineTask specification such that the PipelineTask executes a list of TaskRuns or Runs in parallel with the specified list of inputs for a Parameter or with different combinations of the inputs for a set of Parameters.

To build on this TEP-0018 introduced Matrix.Include, which allows passing in a specific combinations of Parameters into the Matrix.

This PR enables pipeline to handle matrix include params.

Note: This feature is still in preview mode. Implementation logic will be added in subsequent commits.

/kind feature

Submitter Checklist

As the author of this PR, please check off the items in this checklist:

  • Has Docs included if any changes are user facing
  • Has Tests included if any functionality added or changed
  • Follows the commit message standard
  • Meets the Tekton contributor standards (including
    functionality, content, code)
  • Has a kind label. You can add one by adding a comment on this PR that contains /kind <type>. Valid types are bug, cleanup, design, documentation, feature, flake, misc, question, tep
  • Release notes block below has been updated with any user facing changes (API changes, bug fixes, changes requiring upgrade notices or deprecation warnings)
  • Release notes contains the string "action required" if the change requires additional action from users switching to the new release

Release Notes

NONE

@tekton-robot tekton-robot added kind/feature Categorizes issue or PR as related to a new feature. release-note-none Denotes a PR that doesnt merit a release note. labels Feb 28, 2023
@tekton-robot
Copy link
Collaborator

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
To complete the pull request process, please assign imjasonh after the PR has been reviewed.
You can assign the PR to them by writing /assign @imjasonh in a comment when ready.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@tekton-robot tekton-robot added the size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. label Feb 28, 2023
@tekton-robot
Copy link
Collaborator

The following is the coverage report on the affected files.
Say /test pull-tekton-pipeline-go-coverage to re-run this coverage report

File Old Coverage New Coverage Delta
pkg/apis/pipeline/v1/pipeline_validation.go 99.4% 99.5% 0.0
pkg/apis/pipeline/v1beta1/pipeline_conversion.go 94.8% 94.3% -0.5
pkg/apis/pipeline/v1beta1/pipeline_validation.go 99.6% 99.6% 0.0
pkg/reconciler/pipelinerun/resources/apply.go 97.5% 95.1% -2.3

@tekton-robot
Copy link
Collaborator

The following is the coverage report on the affected files.
Say /test pull-tekton-pipeline-go-coverage-df to re-run this coverage report

File Old Coverage New Coverage Delta
pkg/apis/pipeline/v1/pipeline_validation.go 99.4% 99.5% 0.0
pkg/apis/pipeline/v1beta1/pipeline_conversion.go 94.8% 94.3% -0.5
pkg/apis/pipeline/v1beta1/pipeline_validation.go 99.6% 99.6% 0.0
pkg/reconciler/pipelinerun/resources/apply.go 97.5% 95.1% -2.3

@jerop jerop added this to the Pipelines v0.46 milestone Feb 28, 2023
@tekton-robot tekton-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Mar 1, 2023
@EmmaMunley EmmaMunley force-pushed the pipeline-conversion-matrix-include branch 2 times, most recently from d72aa67 to 430f575 Compare March 1, 2023 18:05
@tekton-robot tekton-robot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Mar 1, 2023
@tekton-robot
Copy link
Collaborator

The following is the coverage report on the affected files.
Say /test pull-tekton-pipeline-go-coverage to re-run this coverage report

File Old Coverage New Coverage Delta
pkg/apis/pipeline/v1beta1/pipeline_conversion.go 94.8% 94.3% -0.5
pkg/apis/pipeline/v1beta1/resultref.go 100.0% 97.0% -3.0
pkg/reconciler/pipelinerun/resources/apply.go 97.5% 95.1% -2.3

@tekton-robot
Copy link
Collaborator

The following is the coverage report on the affected files.
Say /test pull-tekton-pipeline-go-coverage-df to re-run this coverage report

File Old Coverage New Coverage Delta
pkg/apis/pipeline/v1beta1/pipeline_conversion.go 94.8% 94.3% -0.5
pkg/apis/pipeline/v1beta1/resultref.go 100.0% 97.0% -3.0
pkg/reconciler/pipelinerun/resources/apply.go 97.5% 95.1% -2.3

@JeromeJu
Copy link
Member

JeromeJu commented Mar 1, 2023

/assign

@QuanZhang-William
Copy link
Member

/assign

Copy link
Member

@QuanZhang-William QuanZhang-William left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think in general we need to make sure the logic and tests are synced for both v1 and v1beta1 API versions at this time?

pkg/apis/pipeline/v1beta1/resultref.go Outdated Show resolved Hide resolved
pkg/apis/pipeline/v1/resultref_test.go Show resolved Hide resolved
pkg/reconciler/pipelinerun/resources/apply.go Outdated Show resolved Hide resolved
pkg/reconciler/pipelinerun/resources/apply.go Outdated Show resolved Hide resolved
pkg/apis/pipeline/v1/pipeline_validation_test.go Outdated Show resolved Hide resolved
@EmmaMunley EmmaMunley force-pushed the pipeline-conversion-matrix-include branch from 430f575 to 627d912 Compare March 2, 2023 03:51
@tekton-robot
Copy link
Collaborator

The following is the coverage report on the affected files.
Say /test pull-tekton-pipeline-go-coverage to re-run this coverage report

File Old Coverage New Coverage Delta
pkg/apis/pipeline/v1beta1/pipeline_conversion.go 94.8% 94.3% -0.5
pkg/reconciler/pipelinerun/resources/apply.go 97.5% 95.1% -2.3

@tekton-robot
Copy link
Collaborator

The following is the coverage report on the affected files.
Say /test pull-tekton-pipeline-go-coverage-df to re-run this coverage report

File Old Coverage New Coverage Delta
pkg/apis/pipeline/v1beta1/pipeline_conversion.go 94.8% 94.3% -0.5
pkg/reconciler/pipelinerun/resources/apply.go 97.5% 95.1% -2.3

@EmmaMunley
Copy link
Contributor Author

/hold Thanks Quan! Holding this for now. May need to include implementation FanOut() logic with this as well.

@tekton-robot tekton-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Mar 2, 2023
@EmmaMunley EmmaMunley force-pushed the pipeline-conversion-matrix-include branch 2 times, most recently from cb8a770 to 77c38a3 Compare March 9, 2023 15:58
@tekton-robot
Copy link
Collaborator

The following is the coverage report on the affected files.
Say /test pull-tekton-pipeline-go-coverage-df to re-run this coverage report

File Old Coverage New Coverage Delta
pkg/apis/pipeline/v1/pipeline_types.go 96.9% 97.1% 0.2
pkg/apis/pipeline/v1/pipeline_validation.go 99.0% 99.0% -0.1
pkg/apis/pipeline/v1beta1/pipeline_conversion.go 96.1% 95.4% -0.7
pkg/apis/pipeline/v1beta1/pipeline_types.go 96.3% 96.6% 0.3
pkg/apis/pipeline/v1beta1/pipeline_validation.go 99.0% 99.0% -0.1
pkg/reconciler/pipelinerun/resources/apply.go 97.5% 95.1% -2.3

@EmmaMunley EmmaMunley force-pushed the pipeline-conversion-matrix-include branch 2 times, most recently from 74532d8 to c5ff9b3 Compare March 9, 2023 21:30
@tekton-robot
Copy link
Collaborator

The following is the coverage report on the affected files.
Say /test pull-tekton-pipeline-go-coverage-df to re-run this coverage report

File Old Coverage New Coverage Delta
pkg/apis/pipeline/v1/pipeline_types.go 96.9% 97.1% 0.2
pkg/apis/pipeline/v1/pipeline_validation.go 99.0% 99.0% -0.1
pkg/apis/pipeline/v1beta1/pipeline_conversion.go 96.1% 95.4% -0.7
pkg/apis/pipeline/v1beta1/pipeline_types.go 96.3% 96.6% 0.3
pkg/apis/pipeline/v1beta1/pipeline_validation.go 99.0% 99.0% -0.1
pkg/reconciler/pipelinerun/resources/apply.go 97.5% 98.0% 0.6

@EmmaMunley EmmaMunley changed the title TEP-0118: Add validation for pipeline conversion, pipeline validation and pipeline run TEP-0118: Enable pipeline to handle matrix include params Mar 9, 2023
@tekton-robot
Copy link
Collaborator

The following is the coverage report on the affected files.
Say /test pull-tekton-pipeline-go-coverage-df to re-run this coverage report

File Old Coverage New Coverage Delta
pkg/apis/pipeline/v1/pipeline_types.go 96.9% 97.1% 0.2
pkg/apis/pipeline/v1/pipeline_validation.go 99.0% 99.0% -0.1
pkg/apis/pipeline/v1beta1/pipeline_conversion.go 96.1% 95.4% -0.7
pkg/apis/pipeline/v1beta1/pipeline_types.go 96.3% 96.6% 0.3
pkg/apis/pipeline/v1beta1/pipeline_validation.go 99.0% 99.0% -0.1
pkg/reconciler/pipelinerun/resources/apply.go 97.5% 98.0% 0.6

@tekton-robot
Copy link
Collaborator

The following is the coverage report on the affected files.
Say /test pull-tekton-pipeline-go-coverage to re-run this coverage report

File Old Coverage New Coverage Delta
pkg/apis/pipeline/v1/pipeline_types.go 96.9% 97.1% 0.2
pkg/apis/pipeline/v1/pipeline_validation.go 99.0% 99.0% -0.1
pkg/apis/pipeline/v1beta1/pipeline_conversion.go 96.1% 95.4% -0.7
pkg/apis/pipeline/v1beta1/pipeline_types.go 96.3% 96.6% 0.3
pkg/apis/pipeline/v1beta1/pipeline_validation.go 99.0% 99.0% -0.1
pkg/reconciler/pipelinerun/resources/apply.go 97.5% 98.0% 0.6

@tekton-robot
Copy link
Collaborator

The following is the coverage report on the affected files.
Say /test pull-tekton-pipeline-go-coverage-df to re-run this coverage report

File Old Coverage New Coverage Delta
pkg/apis/pipeline/v1/pipeline_types.go 96.9% 97.1% 0.2
pkg/apis/pipeline/v1/pipeline_validation.go 99.0% 99.0% -0.1
pkg/apis/pipeline/v1beta1/pipeline_conversion.go 96.1% 95.4% -0.7
pkg/apis/pipeline/v1beta1/pipeline_types.go 96.3% 96.6% 0.3
pkg/apis/pipeline/v1beta1/pipeline_validation.go 99.0% 99.0% -0.1
pkg/reconciler/pipelinerun/resources/apply.go 97.5% 98.0% 0.6

@EmmaMunley EmmaMunley force-pushed the pipeline-conversion-matrix-include branch from c5ff9b3 to fbea2b2 Compare March 9, 2023 22:08
@tekton-robot
Copy link
Collaborator

The following is the coverage report on the affected files.
Say /test pull-tekton-pipeline-go-coverage to re-run this coverage report

File Old Coverage New Coverage Delta
pkg/apis/pipeline/v1/pipeline_types.go 96.9% 97.1% 0.2
pkg/apis/pipeline/v1/pipeline_validation.go 99.0% 99.0% -0.1
pkg/apis/pipeline/v1beta1/pipeline_conversion.go 96.1% 96.4% 0.3
pkg/apis/pipeline/v1beta1/pipeline_types.go 96.3% 96.6% 0.3
pkg/apis/pipeline/v1beta1/pipeline_validation.go 99.0% 99.0% -0.1
pkg/reconciler/pipelinerun/resources/apply.go 97.5% 98.0% 0.6

@tekton-robot
Copy link
Collaborator

The following is the coverage report on the affected files.
Say /test pull-tekton-pipeline-go-coverage-df to re-run this coverage report

File Old Coverage New Coverage Delta
pkg/apis/pipeline/v1/pipeline_types.go 96.9% 97.1% 0.2
pkg/apis/pipeline/v1/pipeline_validation.go 99.0% 99.0% -0.1
pkg/apis/pipeline/v1beta1/pipeline_conversion.go 96.1% 96.4% 0.3
pkg/apis/pipeline/v1beta1/pipeline_types.go 96.3% 96.6% 0.3
pkg/apis/pipeline/v1beta1/pipeline_validation.go 99.0% 99.0% -0.1
pkg/reconciler/pipelinerun/resources/apply.go 97.5% 98.0% 0.6

@EmmaMunley EmmaMunley force-pushed the pipeline-conversion-matrix-include branch from fbea2b2 to 3c23992 Compare March 9, 2023 22:54
@EmmaMunley
Copy link
Contributor Author

/ unhold

1 similar comment
@EmmaMunley
Copy link
Contributor Author

/ unhold

@tekton-robot
Copy link
Collaborator

The following is the coverage report on the affected files.
Say /test pull-tekton-pipeline-go-coverage to re-run this coverage report

File Old Coverage New Coverage Delta
pkg/apis/pipeline/v1/pipeline_types.go 96.9% 97.1% 0.2
pkg/apis/pipeline/v1/pipeline_validation.go 99.0% 99.0% -0.1
pkg/apis/pipeline/v1beta1/pipeline_conversion.go 96.1% 96.4% 0.3
pkg/apis/pipeline/v1beta1/pipeline_types.go 96.3% 96.6% 0.3
pkg/apis/pipeline/v1beta1/pipeline_validation.go 99.0% 99.0% -0.1
pkg/reconciler/pipelinerun/resources/apply.go 97.5% 98.0% 0.6

@tekton-robot
Copy link
Collaborator

The following is the coverage report on the affected files.
Say /test pull-tekton-pipeline-go-coverage-df to re-run this coverage report

File Old Coverage New Coverage Delta
pkg/apis/pipeline/v1/pipeline_types.go 96.9% 97.1% 0.2
pkg/apis/pipeline/v1/pipeline_validation.go 99.0% 99.0% -0.1
pkg/apis/pipeline/v1beta1/pipeline_conversion.go 96.1% 96.4% 0.3
pkg/apis/pipeline/v1beta1/pipeline_types.go 96.3% 96.6% 0.3
pkg/apis/pipeline/v1beta1/pipeline_validation.go 99.0% 99.0% -0.1
pkg/reconciler/pipelinerun/resources/apply.go 97.5% 98.0% 0.6

@EmmaMunley EmmaMunley force-pushed the pipeline-conversion-matrix-include branch from 3c23992 to 6447e5d Compare March 9, 2023 23:06
[TEP-0090: Matrix] introduced `Matrix` to the `PipelineTask` specification such that the `PipelineTask` executes a list of `TaskRuns` or `Runs` in parallel with the specified list of inputs for a `Parameter` or with different combinations of the inputs for a set of `Parameters`.

To build on this, Tep-0018 introduced Matrix.Include, which allows passing in a specific combinations of `Parameters` into the `Matrix`.

This commit enables pipeline to handle matrix include params

Note: This feature is still in preview mode. Implementation logic will be added in subsequent commits.
@EmmaMunley EmmaMunley force-pushed the pipeline-conversion-matrix-include branch from 6447e5d to c9a3de6 Compare March 9, 2023 23:10
@tekton-robot
Copy link
Collaborator

The following is the coverage report on the affected files.
Say /test pull-tekton-pipeline-go-coverage-df to re-run this coverage report

File Old Coverage New Coverage Delta
pkg/apis/pipeline/v1/pipeline_types.go 96.9% 97.1% 0.2
pkg/apis/pipeline/v1/pipeline_validation.go 99.0% 99.0% -0.1
pkg/apis/pipeline/v1beta1/pipeline_conversion.go 96.1% 96.4% 0.3
pkg/apis/pipeline/v1beta1/pipeline_types.go 96.3% 96.6% 0.3
pkg/apis/pipeline/v1beta1/pipeline_validation.go 99.0% 99.0% -0.1
pkg/reconciler/pipelinerun/resources/apply.go 97.5% 98.0% 0.6

@tekton-robot
Copy link
Collaborator

The following is the coverage report on the affected files.
Say /test pull-tekton-pipeline-go-coverage to re-run this coverage report

File Old Coverage New Coverage Delta
pkg/apis/pipeline/v1/pipeline_types.go 96.9% 97.1% 0.2
pkg/apis/pipeline/v1/pipeline_validation.go 99.0% 99.0% -0.1
pkg/apis/pipeline/v1beta1/pipeline_conversion.go 96.1% 96.4% 0.3
pkg/apis/pipeline/v1beta1/pipeline_types.go 96.3% 96.6% 0.3
pkg/apis/pipeline/v1beta1/pipeline_validation.go 99.0% 99.0% -0.1
pkg/reconciler/pipelinerun/resources/apply.go 97.5% 98.0% 0.6

@tekton-robot
Copy link
Collaborator

The following is the coverage report on the affected files.
Say /test pull-tekton-pipeline-go-coverage-df to re-run this coverage report

File Old Coverage New Coverage Delta
pkg/apis/pipeline/v1/pipeline_types.go 96.9% 97.1% 0.2
pkg/apis/pipeline/v1/pipeline_validation.go 99.0% 99.0% -0.1
pkg/apis/pipeline/v1beta1/pipeline_conversion.go 96.1% 96.4% 0.3
pkg/apis/pipeline/v1beta1/pipeline_types.go 96.3% 96.6% 0.3
pkg/apis/pipeline/v1beta1/pipeline_validation.go 99.0% 99.0% -0.1
pkg/reconciler/pipelinerun/resources/apply.go 97.5% 98.0% 0.6

@JeromeJu
Copy link
Member

JeromeJu commented Mar 10, 2023

Thanks for all the changes Emma! 😺

/unhold
(the command needs to be right after the slash)

@tekton-robot tekton-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Mar 10, 2023
@@ -259,6 +259,16 @@ func (m *Matrix) convertTo(ctx context.Context, sink *v1.Matrix) {
param.convertTo(ctx, &new)
sink.Params = append(sink.Params, new)
}
if m.hasInclude() {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: the check here might also be removed

Copy link
Member

@jerop jerop left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"handling matrix include params" is ambiguous, let's clarify what this means in the commit and PR description 🙏🏾

looking into the PR itself, it seems there are three things happening:

  • refactoring extraction of parameters from a pipelinetask
  • conversion of matrix.include to/from v1
  • substituting result references in matrix.include.params

I recommend splitting this PR into three PRs such that each does only one thing 🙏🏾

@lbernick
Copy link
Member

@EmmaMunley related to Jerop's comment, with a lot of background copy-pasted between matrix include commit messages, it's hard to tell what's actually changing as a result of the commit. Can you focus the commit messages more on the specific changes being introduced?

@EmmaMunley
Copy link
Contributor Author

Thank you for the feedback and suggestions. @JeromeJu @JeromeJu @lbernick

I have addressed the comments and broken this out into multiple PRs and shortened the commit messages to be more specific.

cc: @QuanZhang-William

@EmmaMunley EmmaMunley closed this Mar 10, 2023
@JeromeJu
Copy link
Member

thanks for breaking the PRs up Emma!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/feature Categorizes issue or PR as related to a new feature. release-note-none Denotes a PR that doesnt merit a release note. size/XL Denotes a PR that changes 500-999 lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants