Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make PipelineResources (mostly) immutable #1279

Closed
bobcatfish opened this issue Sep 5, 2019 · 2 comments
Closed

Make PipelineResources (mostly) immutable #1279

bobcatfish opened this issue Sep 5, 2019 · 2 comments

Comments

@bobcatfish
Copy link
Collaborator

Expected Behavior

When a Task (say "task B") in a Pipeline gets a PipelineResource from another Task (say "task A"), this means that Task B wants to use a version of the PipelineResource which has been changed by the previous Task A (as described in #1076: "Resources in general should represent reconciliation between a system in the outside world and a filesystem injected into a Tekton Task.").

This means:

  • Task B doesn't actually want to use the original PipelineResource, Task B wants to use the changed PipelineResource
  • If another Task, e.g. Task C, also wanted to use the same PipelineResource, but not "from" Task A, it would want to use the original PipelineResource, not the new one

i.e. Using a PipelineResource as an output should create a new PipelineResource

Actual Behavior

As described in #1076 we create PVCs to hold PipelineResources when from is used (and even when it isn't, but that's another story...) and when a PipelineResource is an output, we copy the data from the local filesystem to the PVC, then use that data as the source for any downstream Tasks that need this PipelineResource as an input.

This means that fan-in behavior is actually undefined, e.g. in this scenario:

//                        pipeline-task-1
//                       /               \
//   pipeline-task-2-parallel-1    pipeline-task-2-parallel-2
//                       \                /
//                        pipeline-task-3

Say that pipeline-task-2-parallel-1 and pipeline-task-2-parallel-2 both used the same PipelineResource as an output (e.g. wrote the file "foo" to disk), and pipeline-task-3 used it as an input.

Which "foo" would pipeline-task-3 get? It's undefined.

Additional Info

We may want the Volumes (#1184) PipelineResource type to be an exception to this b/c folks (e.g. jenkins X) have use cases where they want to just pass data between Tasks without having to upload it somewhere in between. However this implies that fan-in should not be allowed for Tasks using Volumes.

@bobcatfish
Copy link
Collaborator Author

@bobcatfish
Copy link
Collaborator Author

New proposal from @sbwsg for re-designing PipelineResources: https://docs.google.com/document/d/1euQ_gDTe_dQcVeX4oypODGIQCAkUaMYQH5h7SaeFs44/edit# (for #1076)

The proposal in that doc includes immutable PipelineResources. Closing this issue to reduce the number of issues we need to update with the same information.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant