Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

include column lineage in dataset resource #2148

Merged
merged 1 commit into from
Oct 11, 2022

Conversation

pawel-big-lebowski
Copy link
Collaborator

Signed-off-by: Pawel Leszczynski leszczynski.pawel@gmail.com

Problem

Column lineage should be returned within dataset resource.

Closes: #2113

Solution

  • create method in ColumnLineageService which enriches Dataset with column lineage information,
  • use the method in DatasetResource.

Note: All database schema changes require discussion. Please link the issue for context.

Checklist

  • You've signed-off your work
  • Your changes are accompanied by tests (if relevant)
  • Your change contains a small diff and is self-contained
  • You've updated any relevant documentation (if relevant)
  • You've updated the CHANGELOG.md with details about your change under the "Unreleased" section (if relevant, depending on the change, this may not be necessary)
  • You've versioned your .sql database schema migration according to Flyway's naming convention (if relevant)
  • You've included a header in any source code files (if relevant)

@codecov
Copy link

codecov bot commented Sep 29, 2022

Codecov Report

Merging #2148 (496566e) into main (aa7a47d) will not change coverage.
The diff coverage is n/a.

❗ Current head 496566e differs from pull request most recent head 1ed1720. Consider uploading reports for the commit 1ed1720 to get more accurate results

@@            Coverage Diff            @@
##               main    #2148   +/-   ##
=========================================
  Coverage     76.33%   76.33%           
  Complexity     1099     1099           
=========================================
  Files           214      214           
  Lines          5139     5139           
  Branches        407      407           
=========================================
  Hits           3923     3923           
  Misses          762      762           
  Partials        454      454           

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

@pawel-big-lebowski pawel-big-lebowski force-pushed the column-lineage-dataset-resource branch from 70db39e to 91d30f0 Compare September 29, 2022 14:18
@pawel-big-lebowski pawel-big-lebowski force-pushed the column-lineage-graph-endpoint branch 2 times, most recently from c65ecdd to 791e1bf Compare September 30, 2022 12:13
@pawel-big-lebowski pawel-big-lebowski force-pushed the column-lineage-dataset-resource branch from 91d30f0 to 5d49c51 Compare September 30, 2022 12:35
@pawel-big-lebowski pawel-big-lebowski force-pushed the column-lineage-dataset-resource branch 2 times, most recently from 6c35e81 to 8b669df Compare September 30, 2022 14:01
@pawel-big-lebowski pawel-big-lebowski force-pushed the column-lineage-dataset-resource branch from 8b669df to d4f0890 Compare October 5, 2022 07:05
@pawel-big-lebowski pawel-big-lebowski force-pushed the column-lineage-graph-endpoint branch 2 times, most recently from b79d712 to 8e66689 Compare October 7, 2022 10:07
Base automatically changed from column-lineage-graph-endpoint to main October 7, 2022 12:53
@pawel-big-lebowski pawel-big-lebowski force-pushed the column-lineage-dataset-resource branch from d4f0890 to 950e293 Compare October 7, 2022 12:57
@pawel-big-lebowski pawel-big-lebowski marked this pull request as ready for review October 10, 2022 08:05
@pawel-big-lebowski pawel-big-lebowski force-pushed the column-lineage-dataset-resource branch from 950e293 to 1ed1720 Compare October 10, 2022 12:17
@mobuchowski mobuchowski requested a review from wslulciuc October 10, 2022 15:07
Copy link
Contributor

@mobuchowski mobuchowski left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Few minor nits, besides LGTM

api/src/main/java/marquez/db/ColumnLineageDao.java Outdated Show resolved Hide resolved
api/src/test/java/marquez/DatasetIntegrationTest.java Outdated Show resolved Hide resolved
@pawel-big-lebowski pawel-big-lebowski force-pushed the column-lineage-dataset-resource branch from 1ed1720 to 2239520 Compare October 11, 2022 08:57
Signed-off-by: Pawel Leszczynski <leszczynski.pawel@gmail.com>
@pawel-big-lebowski pawel-big-lebowski force-pushed the column-lineage-dataset-resource branch from 2239520 to b665ad6 Compare October 11, 2022 09:25
@pawel-big-lebowski pawel-big-lebowski merged commit 6fd6416 into main Oct 11, 2022
@pawel-big-lebowski pawel-big-lebowski deleted the column-lineage-dataset-resource branch October 11, 2022 09:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Column lineage as a dataset resource
3 participants