Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

dcpy: Add capability to scrape input datasets from recipe #978

Draft
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

sf-dcp
Copy link
Contributor

@sf-dcp sf-dcp commented Jul 5, 2024

It would be nice to automatically ingest input datasets for product recipes. Not all inputs can be automated: it depends on its source type. But we can create a fn that will try to extract as many as possible.

For now, it's meant to be a draft PR until we fully migrate to ingest code.

TODO:

  • modify fn to set recipe path based on product's name. For example, when input = db-pluto, recipe path is ./products/db-pluto/recipe.yml
  • write tests
  • GHA?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Love it. I do think this code should live somewhere within builds - even if it's doing ingest logic, it's a logical step that happens at the beginning of the build cycle and obviously is quite coupled to recipes which are really build entities

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Status: New
Development

Successfully merging this pull request may close these issues.

2 participants