Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[TEP074] Remove Storage pipelineResources #6014

Closed
wants to merge 2 commits into from

Conversation

JeromeJu
Copy link
Member

@JeromeJu JeromeJu commented Jan 20, 2023

Changes

Remove Storage PipelineResources

This commit removes the Storage Resources support.
This PR removes github.com/tektoncd/pipeline/pkg/apis/resource/v1alpha1/storage,
pkg/artifacts and the respective test cases.

Submitter Checklist

As the author of this PR, please check off the items in this checklist:

  • Has Docs included if any changes are user facing
  • Has Tests included if any functionality added or changed
  • Follows the commit message standard
  • Meets the Tekton contributor standards (including
    functionality, content, code)
  • Has a kind label. You can add one by adding a comment on this PR that contains /kind <type>. Valid types are bug, cleanup, design, documentation, feature, flake, misc, question, tep
  • Release notes block below has been updated with any user facing changes (API changes, bug fixes, changes requiring upgrade notices or deprecation warnings)
  • Release notes contains the string "action required" if the change requires additional action from users switching to the new release

Release Notes

action required: please migrate off of `storage` `pipelineresources` as it is removed, please refer to the doc at https://github.com/tektoncd/pipeline/blob/main/docs/pipelineresources.md#replacing-an-storage-resource
`artifacts` apis are removed with the setup for pvc/buckets

@tekton-robot tekton-robot added do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. release-note-none Denotes a PR that doesnt merit a release note. labels Jan 20, 2023
@tekton-robot tekton-robot added the size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. label Jan 20, 2023
@tekton-robot
Copy link
Collaborator

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
To complete the pull request process, please assign pritidesai after the PR has been reviewed.
You can assign the PR to them by writing /assign @pritidesai in a comment when ready.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@tekton-robot tekton-robot added the kind/misc Categorizes issue or PR as a miscellaneuous one. label Jan 20, 2023
@tekton-robot
Copy link
Collaborator

The following is the coverage report on the affected files.
Say /test pull-tekton-pipeline-go-coverage to re-run this coverage report

File Old Coverage New Coverage Delta
pkg/apis/resource/v1alpha1/pipelineresource_validation.go 93.8% 89.5% -4.3
pkg/reconciler/taskrun/resources/input_resources.go 90.9% 81.6% -9.3
pkg/reconciler/taskrun/resources/output_resource.go 89.1% 86.2% -2.9

@tekton-robot
Copy link
Collaborator

The following is the coverage report on the affected files.
Say /test pull-tekton-pipeline-go-coverage-df to re-run this coverage report

File Old Coverage New Coverage Delta
pkg/apis/resource/v1alpha1/pipelineresource_validation.go 93.8% 89.5% -4.3
pkg/reconciler/taskrun/resources/input_resources.go 90.9% 81.6% -9.3
pkg/reconciler/taskrun/resources/output_resource.go 89.1% 86.2% -2.9

@JeromeJu JeromeJu changed the title [WIP] Remove Storage pipelineResources [WIP] [TEP074] Remove Storage pipelineResources Jan 20, 2023
@tekton-robot tekton-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jan 24, 2023
@lbernick lbernick added this to the Pipelines v0.45 milestone Jan 24, 2023
@JeromeJu JeromeJu force-pushed the rm-storage-resources branch 2 times, most recently from ab4bd05 to 00b8456 Compare February 2, 2023 13:02
@tekton-robot tekton-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Feb 2, 2023
@tekton-robot
Copy link
Collaborator

The following is the coverage report on the affected files.
Say /test pull-tekton-pipeline-go-coverage to re-run this coverage report

File Old Coverage New Coverage Delta
pkg/apis/resource/v1alpha1/pipelineresource_validation.go 93.8% 89.5% -4.3
pkg/reconciler/pipelinerun/pipelinerun.go 86.6% 87.0% 0.4
pkg/reconciler/pipelinerun/resources/input_output_steps.go 96.9% 96.0% -0.9
pkg/reconciler/taskrun/resources/input_resources.go 90.9% 82.9% -8.1
pkg/reconciler/taskrun/resources/output_resource.go 89.1% 86.2% -2.9
test/controller.go 29.1% 27.8% -1.3

@tekton-robot
Copy link
Collaborator

The following is the coverage report on the affected files.
Say /test pull-tekton-pipeline-go-coverage-df to re-run this coverage report

File Old Coverage New Coverage Delta
pkg/apis/resource/v1alpha1/pipelineresource_validation.go 93.8% 89.5% -4.3
pkg/reconciler/pipelinerun/pipelinerun.go 86.6% 87.0% 0.4
pkg/reconciler/pipelinerun/resources/input_output_steps.go 96.9% 96.0% -0.9
pkg/reconciler/taskrun/resources/input_resources.go 90.9% 82.9% -8.1
pkg/reconciler/taskrun/resources/output_resource.go 89.1% 86.2% -2.9
test/controller.go 29.1% 27.8% -1.3

@tekton-robot
Copy link
Collaborator

The following is the coverage report on the affected files.
Say /test pull-tekton-pipeline-go-coverage to re-run this coverage report

File Old Coverage New Coverage Delta
pkg/apis/resource/v1alpha1/pipelineresource_validation.go 93.8% 89.5% -4.3
pkg/reconciler/pipelinerun/pipelinerun.go 86.6% 87.0% 0.4
pkg/reconciler/pipelinerun/resources/input_output_steps.go 96.9% 96.0% -0.9
pkg/reconciler/taskrun/resources/input_resources.go 90.9% 82.9% -8.1
pkg/reconciler/taskrun/resources/output_resource.go 89.1% 86.2% -2.9
test/controller.go 29.1% 27.8% -1.3

@tekton-robot
Copy link
Collaborator

The following is the coverage report on the affected files.
Say /test pull-tekton-pipeline-go-coverage-df to re-run this coverage report

File Old Coverage New Coverage Delta
pkg/apis/resource/v1alpha1/pipelineresource_validation.go 93.8% 89.5% -4.3
pkg/reconciler/pipelinerun/pipelinerun.go 86.6% 87.0% 0.4
pkg/reconciler/pipelinerun/resources/input_output_steps.go 96.9% 96.0% -0.9
pkg/reconciler/taskrun/resources/input_resources.go 90.9% 82.9% -8.1
pkg/reconciler/taskrun/resources/output_resource.go 89.1% 86.2% -2.9
test/controller.go 29.1% 27.8% -1.3

@@ -948,7 +937,7 @@ func (c *Reconciler) createTaskRun(ctx context.Context, taskRunName string, para
tr.Annotations[workspace.AnnotationAffinityAssistantName] = getAffinityAssistantName(pipelinePVCWorkspaceName, pr.Name)
}

resources.WrapSteps(&tr.Spec, rpt.PipelineTask, rpt.ResolvedTaskResources.Inputs, rpt.ResolvedTaskResources.Outputs, storageBasePath)
resources.WrapSteps(&tr.Spec, rpt.PipelineTask, rpt.ResolvedTaskResources.Inputs, rpt.ResolvedTaskResources.Outputs)
Copy link
Member Author

@JeromeJu JeromeJu Feb 2, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since removing the artifacts/pvc-bucket would worth to clean up in a separate PR that before the removal of storage resource as the path of the output/input resources, I have been trying to come up with a better way for this removal:

For the WrapSteps, I would like to check if it is w to remove this in this PR or if it is worth changing the corresponding test cases while it is going to be removed eventually?

But if so, it will be related with removing/ cleaning up the pipeline/pkg/reconciler/pipelinerun/resources/input_output_steps.go 🤔

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@JeromeJu are you still having trouble with this bit? I think the git pipelineresource also uses artifact storage; it might be easier to not delete artifact storage in this PR or to delete the git resource and the storage resource in the same PR

Copy link
Member Author

@JeromeJu JeromeJu Feb 13, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @lbernick , that makes sense. Since Git and Storage pipelineresources both use artifact storage, we would have to remove them together to avoid the situation where we are introducing changes of Git resources removal in this Storage removal PR along.

Therefore, I tried to merge the changes and restore the parts on Git resources to be restored for image type at #6150.

And if that looks good I will try to make Image Resources removal as the last PR along with the removal of the generic functions as the removal of both Storage and Git resources has already been a large chunk.

@tekton-robot tekton-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Feb 3, 2023
@JeromeJu JeromeJu force-pushed the rm-storage-resources branch 2 times, most recently from 2f142e1 to 9d37509 Compare February 7, 2023 02:13
@tekton-robot tekton-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Feb 7, 2023
@tekton-robot
Copy link
Collaborator

The following is the coverage report on the affected files.
Say /test pull-tekton-pipeline-go-coverage to re-run this coverage report

File Old Coverage New Coverage Delta
pkg/apis/resource/v1alpha1/pipelineresource_validation.go 88.0% 75.0% -13.0
pkg/reconciler/pipelinerun/pipelinerun.go 87.2% 87.7% 0.5
pkg/reconciler/pipelinerun/resources/input_output_steps.go 96.9% 96.0% -0.9
pkg/reconciler/taskrun/resources/input_resources.go 90.9% 82.9% -8.1
pkg/reconciler/taskrun/resources/output_resource.go 89.1% 86.2% -2.9
test/controller.go 29.1% 27.8% -1.3

@tekton-robot
Copy link
Collaborator

The following is the coverage report on the affected files.
Say /test pull-tekton-pipeline-go-coverage-df to re-run this coverage report

File Old Coverage New Coverage Delta
pkg/apis/resource/v1alpha1/pipelineresource_validation.go 88.0% 75.0% -13.0
pkg/reconciler/pipelinerun/pipelinerun.go 87.2% 87.7% 0.5
pkg/reconciler/pipelinerun/resources/input_output_steps.go 96.9% 96.0% -0.9
pkg/reconciler/taskrun/resources/input_resources.go 90.9% 82.9% -8.1
pkg/reconciler/taskrun/resources/output_resource.go 89.1% 86.2% -2.9
test/controller.go 29.1% 27.8% -1.3

@tekton-robot
Copy link
Collaborator

The following is the coverage report on the affected files.
Say /test pull-tekton-pipeline-go-coverage to re-run this coverage report

File Old Coverage New Coverage Delta
pkg/apis/resource/v1alpha1/pipelineresource_validation.go 88.0% 75.0% -13.0
pkg/reconciler/pipelinerun/pipelinerun.go 87.2% 87.7% 0.5
pkg/reconciler/pipelinerun/resources/input_output_steps.go 96.9% 96.0% -0.9
pkg/reconciler/taskrun/resources/input_resources.go 90.9% 82.9% -8.1
pkg/reconciler/taskrun/resources/output_resource.go 89.1% 86.2% -2.9
test/controller.go 29.1% 27.8% -1.3

@tekton-robot
Copy link
Collaborator

The following is the coverage report on the affected files.
Say /test pull-tekton-pipeline-go-coverage to re-run this coverage report

File Old Coverage New Coverage Delta
pkg/apis/resource/v1alpha1/pipelineresource_validation.go 88.0% 75.0% -13.0
pkg/reconciler/pipelinerun/pipelinerun.go 87.2% 87.7% 0.5
pkg/reconciler/pipelinerun/resources/input_output_steps.go 96.9% 96.0% -0.9
pkg/reconciler/taskrun/resources/input_resources.go 90.9% 77.1% -13.8
pkg/reconciler/taskrun/resources/output_resource.go 89.1% 82.8% -6.4
test/controller.go 29.1% 27.8% -1.3

@tekton-robot
Copy link
Collaborator

The following is the coverage report on the affected files.
Say /test pull-tekton-pipeline-go-coverage-df to re-run this coverage report

File Old Coverage New Coverage Delta
pkg/apis/resource/v1alpha1/pipelineresource_validation.go 88.0% 75.0% -13.0
pkg/reconciler/pipelinerun/pipelinerun.go 87.2% 87.7% 0.5
pkg/reconciler/pipelinerun/resources/input_output_steps.go 96.9% 96.0% -0.9
pkg/reconciler/taskrun/resources/input_resources.go 90.9% 77.1% -13.8
pkg/reconciler/taskrun/resources/output_resource.go 89.1% 82.8% -6.4
test/controller.go 29.1% 27.8% -1.3

@tekton-robot
Copy link
Collaborator

The following is the coverage report on the affected files.
Say /test pull-tekton-pipeline-go-coverage to re-run this coverage report

File Old Coverage New Coverage Delta
pkg/apis/resource/v1alpha1/pipelineresource_validation.go 88.0% 75.0% -13.0
pkg/reconciler/pipelinerun/pipelinerun.go 87.2% 86.9% -0.3
pkg/reconciler/pipelinerun/resources/input_output_steps.go 96.9% 96.0% -0.9
pkg/reconciler/taskrun/resources/input_resources.go 90.9% 77.1% -13.8
pkg/reconciler/taskrun/resources/output_resource.go 89.1% 82.8% -6.4
test/controller.go 29.1% 27.8% -1.3

@tekton-robot
Copy link
Collaborator

The following is the coverage report on the affected files.
Say /test pull-tekton-pipeline-go-coverage-df to re-run this coverage report

File Old Coverage New Coverage Delta
pkg/apis/resource/v1alpha1/pipelineresource_validation.go 88.0% 75.0% -13.0
pkg/reconciler/pipelinerun/pipelinerun.go 87.2% 86.9% -0.3
pkg/reconciler/pipelinerun/resources/input_output_steps.go 96.9% 96.0% -0.9
pkg/reconciler/taskrun/resources/input_resources.go 90.9% 77.1% -13.8
pkg/reconciler/taskrun/resources/output_resource.go 89.1% 82.8% -6.4
test/controller.go 29.1% 27.8% -1.3

@tekton-robot
Copy link
Collaborator

The following is the coverage report on the affected files.
Say /test pull-tekton-pipeline-go-coverage to re-run this coverage report

File Old Coverage New Coverage Delta
pkg/apis/resource/v1alpha1/pipelineresource_validation.go 88.0% 75.0% -13.0
pkg/reconciler/pipelinerun/pipelinerun.go 87.2% 86.9% -0.3
pkg/reconciler/pipelinerun/resources/input_output_steps.go 96.9% 96.0% -0.9
pkg/reconciler/taskrun/resources/input_resources.go 90.9% 77.1% -13.8
pkg/reconciler/taskrun/resources/output_resource.go 89.1% 82.8% -6.4
test/controller.go 29.1% 27.8% -1.3

@tekton-robot
Copy link
Collaborator

The following is the coverage report on the affected files.
Say /test pull-tekton-pipeline-go-coverage-df to re-run this coverage report

File Old Coverage New Coverage Delta
pkg/apis/resource/v1alpha1/pipelineresource_validation.go 88.0% 75.0% -13.0
pkg/reconciler/pipelinerun/pipelinerun.go 87.2% 86.9% -0.3
pkg/reconciler/pipelinerun/resources/input_output_steps.go 96.9% 96.0% -0.9
pkg/reconciler/taskrun/resources/input_resources.go 90.9% 77.1% -13.8
pkg/reconciler/taskrun/resources/output_resource.go 89.1% 82.8% -6.4
test/controller.go 29.1% 27.8% -1.3

@JeromeJu
Copy link
Member Author

JeromeJu commented Feb 9, 2023

/retest

@JeromeJu JeromeJu changed the title [WIP] [TEP074] Remove Storage pipelineResources [TEP074] Remove Storage pipelineResources Feb 9, 2023
@tekton-robot tekton-robot added release-note-action-required Denotes a PR that introduces potentially breaking changes that require user action. and removed do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. release-note-none Denotes a PR that doesnt merit a release note. labels Feb 9, 2023
This commit removes the Storage Resources support.
This PR removes `github.com/tektoncd/pipeline/pkg/apis/resource/v1alpha1/storage`,
`pkg/artifacts` and the respective test cases.
@tekton-robot
Copy link
Collaborator

The following is the coverage report on the affected files.
Say /test pull-tekton-pipeline-go-coverage-df to re-run this coverage report

File Old Coverage New Coverage Delta
pkg/apis/resource/v1alpha1/pipelineresource_validation.go 88.0% 75.0% -13.0
pkg/reconciler/pipelinerun/pipelinerun.go 87.2% 86.9% -0.3
pkg/reconciler/pipelinerun/resources/input_output_steps.go 96.9% 96.0% -0.9
pkg/reconciler/taskrun/resources/input_resources.go 90.9% 77.1% -13.8
pkg/reconciler/taskrun/resources/output_resource.go 89.1% 82.8% -6.4
test/controller.go 29.1% 27.8% -1.3

@tekton-robot
Copy link
Collaborator

The following is the coverage report on the affected files.
Say /test pull-tekton-pipeline-go-coverage to re-run this coverage report

File Old Coverage New Coverage Delta
pkg/apis/resource/v1alpha1/pipelineresource_validation.go 88.0% 75.0% -13.0
pkg/reconciler/pipelinerun/pipelinerun.go 87.2% 86.9% -0.3
pkg/reconciler/pipelinerun/resources/input_output_steps.go 96.9% 96.0% -0.9
pkg/reconciler/taskrun/resources/input_resources.go 90.9% 77.1% -13.8
pkg/reconciler/taskrun/resources/output_resource.go 89.1% 82.8% -6.4
test/controller.go 29.1% 27.8% -1.3

@tekton-robot
Copy link
Collaborator

The following is the coverage report on the affected files.
Say /test pull-tekton-pipeline-go-coverage-df to re-run this coverage report

File Old Coverage New Coverage Delta
pkg/apis/resource/v1alpha1/pipelineresource_validation.go 88.0% 75.0% -13.0
pkg/reconciler/pipelinerun/pipelinerun.go 87.2% 86.9% -0.3
pkg/reconciler/pipelinerun/resources/input_output_steps.go 96.9% 96.0% -0.9
pkg/reconciler/taskrun/resources/input_resources.go 90.9% 77.1% -13.8
pkg/reconciler/taskrun/resources/output_resource.go 89.1% 82.8% -6.4
test/controller.go 29.1% 27.8% -1.3

@JeromeJu
Copy link
Member Author

This PR is ready for review, thanks!

PTAL /assign @afrittoli @lbernick

@JeromeJu
Copy link
Member Author

/assign @Yongxuanzhang

| `name` | The name of the resource. |
| `type` | Type value of `"gcs"`. |
| `location` | The fully qualified address of the blob storage. |

#### Variables for the `Cluster` type
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it looks like there's some cloudevent and cluster docs hanging around here

@@ -948,7 +937,7 @@ func (c *Reconciler) createTaskRun(ctx context.Context, taskRunName string, para
tr.Annotations[workspace.AnnotationAffinityAssistantName] = getAffinityAssistantName(pipelinePVCWorkspaceName, pr.Name)
}

resources.WrapSteps(&tr.Spec, rpt.PipelineTask, rpt.ResolvedTaskResources.Inputs, rpt.ResolvedTaskResources.Outputs, storageBasePath)
resources.WrapSteps(&tr.Spec, rpt.PipelineTask, rpt.ResolvedTaskResources.Inputs, rpt.ResolvedTaskResources.Outputs)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@JeromeJu are you still having trouble with this bit? I think the git pipelineresource also uses artifact storage; it might be easier to not delete artifact storage in this PR or to delete the git resource and the storage resource in the same PR

@@ -1204,3 +1205,30 @@ func tearDownV1Beta1CustomTask(ctx context.Context, t *testing.T, c *clients, na

tearDown(ctx, t, c, namespace)
}

// updateConfigMap updates the config map for specified @name with values. We can't use the one from knativetest because
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

was this moved from somewhere else or just a weird rebase?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It used to be in the deleted artifact_bucekt_test.go and referenced here.

@@ -64,13 +64,6 @@ metadata:
name: %s
namespace: %s
spec:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

seems like the changes in this file are mainly related to git resources

@@ -55,6 +55,7 @@ const (
scmRemoteUserPassword = "ab_d1234HIJKL"
// Defined in git-resolver/gitea.yaml's "gitea" StatefulSet, in the env for the "configure-gitea" init container
scmGiteaAdminPassword = "giteaPassword1234"
systemNamespace = "tekton-pipelines"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what's the reason for this change?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is moved from the deleted artifact_bucket_test.go

@JeromeJu
Copy link
Member Author

Closing as #6150 supersedes this PR

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/misc Categorizes issue or PR as a miscellaneuous one. release-note-action-required Denotes a PR that introduces potentially breaking changes that require user action. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants