You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
With the upcoming change to the behavior of output resources: #1122
We now have a potential ease-of-use issue. Consider the common case of a Task that takes an input resource and produces an output resource.
Description
If the Output Resource is brand new, the Task is responsible for seeding the output directory with all the information it needs. This makes sense. However, if the Output resource represents something that already exists (a PR, a Git repo, a GCS directory, etc.), the Task must now seed that directory with the existing information before making any changes.
Example
If the task wants to do something like add a label, it must do the following:
Copy the input to the output
Add a label
Forgetting to do the first one would result in all the other labels and information being deleted.
If a task wants to upload a file to a GCS bucket, it must first copy all the files from that bucket into the output directory, otherwise the GCS resource would delete all the contents.
Proposal
We should extend the Resource interface to allow Output resources to add directory initialization steps. Today all Output resources create an empty directory by default, but if we move this into the Resource interface we can allow this behavior to be customized per-resource.
With the upcoming change to the behavior of output resources: #1122
We now have a potential ease-of-use issue. Consider the common case of a Task that takes an input resource and produces an output resource.
Description
If the Output Resource is brand new, the Task is responsible for seeding the output directory with all the information it needs. This makes sense. However, if the Output resource represents something that already exists (a PR, a Git repo, a GCS directory, etc.), the Task must now seed that directory with the existing information before making any changes.
Example
If the task wants to do something like add a label, it must do the following:
Forgetting to do the first one would result in all the other labels and information being deleted.
If a task wants to upload a file to a GCS bucket, it must first copy all the files from that bucket into the output directory, otherwise the GCS resource would delete all the contents.
Proposal
We should extend the Resource interface to allow Output resources to add directory initialization steps. Today all Output resources create an empty directory by default, but if we move this into the Resource interface we can allow this behavior to be customized per-resource.
This has implications on #238
cc @vdemeester
The text was updated successfully, but these errors were encountered: