Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Proposal: Allow Output Resources to add pre-run containers #1214

Closed
dlorenc opened this issue Aug 19, 2019 · 2 comments
Closed

Proposal: Allow Output Resources to add pre-run containers #1214

dlorenc opened this issue Aug 19, 2019 · 2 comments

Comments

@dlorenc
Copy link
Contributor

dlorenc commented Aug 19, 2019

With the upcoming change to the behavior of output resources: #1122

We now have a potential ease-of-use issue. Consider the common case of a Task that takes an input resource and produces an output resource.

Description

If the Output Resource is brand new, the Task is responsible for seeding the output directory with all the information it needs. This makes sense. However, if the Output resource represents something that already exists (a PR, a Git repo, a GCS directory, etc.), the Task must now seed that directory with the existing information before making any changes.

Example

If the task wants to do something like add a label, it must do the following:

  • Copy the input to the output
  • Add a label

Forgetting to do the first one would result in all the other labels and information being deleted.

If a task wants to upload a file to a GCS bucket, it must first copy all the files from that bucket into the output directory, otherwise the GCS resource would delete all the contents.

Proposal

We should extend the Resource interface to allow Output resources to add directory initialization steps. Today all Output resources create an empty directory by default, but if we move this into the Resource interface we can allow this behavior to be customized per-resource.

This has implications on #238

cc @vdemeester

@dlorenc
Copy link
Contributor Author

dlorenc commented Aug 19, 2019

It might also be enough to instead just treat the Output Resource as an input resource for initialization purposes.

@dlorenc
Copy link
Contributor Author

dlorenc commented Oct 7, 2019

This was fixed wtih #1215

@dlorenc dlorenc closed this as completed Oct 7, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant