Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Gen new uuid for etl_menus ol events #2519

Merged
merged 1 commit into from
Jun 15, 2023
Merged

Conversation

wslulciuc
Copy link
Member

@wslulciuc wslulciuc commented Jun 15, 2023

Problem

The OL quickstart produces an invalid lineage graph if Marquez was seeded using the seed cmd (see related PR OpenLineage/docs#183). That is, in the lineage graph below, the job my-job should not contain the out edge public.menus as an output dataset (linking the two subgraphs):

Screenshot 2023-06-15 at 12 47 42

Solution

Believe it or not, the invalid lineage graph is actually, valid! It's the result of a runID collision. That is, the OL events for my-job in the quickstart uses the runID d46e465b-d358-4d32-83d4-df660ff614dd, the same runID d46e465b-d358-4d32-83d4-df660ff614dd as job etl_menus in metadata.json. Therefore, the backend assumes the outputs have changed for etl_menus. This PR generates a new runID 4d3b8069-69b6-4708-ade0-3275112c9f04 for etl_menus to render the expected lineage graph:

Screen Shot 2023-06-15 at 2 31 36 PM

Checklist

  • You've signed-off your work
  • Your changes are accompanied by tests (if relevant)
  • Your change contains a small diff and is self-contained
  • You've updated any relevant documentation (if relevant)
  • You've included a one-line summary of your change for the CHANGELOG.md (Depending on the change, this may not be necessary).
  • You've versioned your .sql database schema migration according to Flyway's naming convention (if relevant)
  • You've included a header in any source code files (if relevant)

Signed-off-by: wslulciuc <willy@datakin.com>
@boring-cyborg boring-cyborg bot added the docker label Jun 15, 2023
@codecov
Copy link

codecov bot commented Jun 15, 2023

Codecov Report

Merging #2519 (a8883ca) into main (613b6fc) will not change coverage.
The diff coverage is n/a.

@@            Coverage Diff            @@
##               main    #2519   +/-   ##
=========================================
  Coverage     83.80%   83.80%           
  Complexity     1233     1233           
=========================================
  Files           235      235           
  Lines          5625     5625           
  Branches        270      270           
=========================================
  Hits           4714     4714           
  Misses          767      767           
  Partials        144      144           

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

Copy link
Member

@phixMe phixMe left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, looks good.

Copy link
Collaborator

@merobi-hub merobi-hub left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

awesome, thank you

@wslulciuc wslulciuc merged commit e1886e6 into main Jun 15, 2023
@wslulciuc wslulciuc deleted the bug/broken-ol-quickstart branch June 15, 2023 23:59
Xavier-Cliquennois pushed a commit to Xavier-Cliquennois/marquez that referenced this pull request Jul 26, 2023
Signed-off-by: wslulciuc <willy@datakin.com>
Signed-off-by: Xavier-Cliquennois <xavier.cliquennois@wearegraphite.io>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants