Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature: Maintain link between kedro data catalog and .dvc files #16

Open
1 of 17 tasks
shaunc opened this issue Feb 12, 2022 · 0 comments
Open
1 of 17 tasks

Feature: Maintain link between kedro data catalog and .dvc files #16

shaunc opened this issue Feb 12, 2022 · 0 comments
Labels
feature tracking issue for feature
Milestone

Comments

@shaunc
Copy link
Collaborator

shaunc commented Feb 12, 2022

In order to track data used in kedro pipelines, we need to maintain a link between the kedro data catalog and data tracked by dvc.

See discussion.

For each supported entry in catalog, we create .dvc as described in discussion. We

Cases:

  • where is configuration: default or configured in different location?
  • kedro environment present in
    • base
    • local
    • test
  • dvc status
    • no .dvc
    • .dvc already exists
  • whether we run via command line or in hook
    • command line
    • on hook after_catalog_created:
      • same as kedro dvc update without --force

Scenarios:

  • kedro dvc update
    • if not exists, create
    • if exists, and corresponds, do nothing
    • if exists and different, error
  • kedro dvc update --force
    • same as above, except if exists and different, overwrite
  • environment besides base.

Tasks:

@shaunc shaunc assigned shaunc, 7UR7L3 and ElijahCFisher and unassigned shaunc Feb 12, 2022
@shaunc shaunc added the feature tracking issue for feature label Feb 12, 2022
@shaunc shaunc added this to the Stage 1 milestone Feb 12, 2022
@shaunc shaunc moved this to Todo in Kedro-dvc stage 1 Feb 19, 2022
@shaunc shaunc added the breakdown Break down issues w/ >1D expected implementation label Feb 19, 2022
@shaunc shaunc moved this from Todo to In Progress in Kedro-dvc stage 1 Mar 4, 2022
@shaunc shaunc removed the breakdown Break down issues w/ >1D expected implementation label Mar 6, 2022
@shaunc shaunc moved this from In Progress to Todo in Kedro-dvc stage 1 Mar 14, 2022
@shaunc shaunc moved this from Todo to In Progress in Kedro-dvc stage 1 May 6, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature tracking issue for feature
Projects
Status: In Progress
Development

No branches or pull requests

3 participants