Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Runs row reduction #2041

Merged
merged 2 commits into from
Jul 26, 2022
Merged

Runs row reduction #2041

merged 2 commits into from
Jul 26, 2022

Conversation

collado-mike
Copy link
Collaborator

Problem

A lot of the runs queries we use unnecessarily join with the jobs_view when no job information is necessary. As part of the effort to remove the proliferation of job names, I reduced the scope of RunRow to exclude job information and changed consumers to depend on RunRow rather than ExtendedRunRow where only the run information was needed (often only the run UUID).

Note: All database schema changes require discussion. Please link the issue for context.

Checklist

  • You've signed-off your work
  • Your changes are accompanied by tests (if relevant)
  • Your change contains a small diff and is self-contained
  • You've updated any relevant documentation (if relevant)
  • You've updated the CHANGELOG.md with details about your change under the "Unreleased" section (if relevant, depending on the change, this may not be necessary)
  • You've versioned your .sql database schema migration according to Flyway's naming convention (if relevant)
  • You've included a header in any source code files (if relevant)

…possible

Signed-off-by: Michael Collado <collado.mike@gmail.com>
Signed-off-by: Michael Collado <collado.mike@gmail.com>
@collado-mike collado-mike requested a review from wslulciuc July 20, 2022 21:44
@codecov
Copy link

codecov bot commented Jul 20, 2022

Codecov Report

Merging #2041 (5c88d5b) into main (ccbdb96) will decrease coverage by 0.02%.
The diff coverage is 85.93%.

@@             Coverage Diff              @@
##               main    #2041      +/-   ##
============================================
- Coverage     78.83%   78.81%   -0.03%     
+ Complexity     1015     1013       -2     
============================================
  Files           199      200       +1     
  Lines          5557     5579      +22     
  Branches        421      422       +1     
============================================
+ Hits           4381     4397      +16     
- Misses          726      730       +4     
- Partials        450      452       +2     
Impacted Files Coverage Δ
api/src/main/java/marquez/db/JobVersionDao.java 91.04% <0.00%> (+1.18%) ⬆️
api/src/main/java/marquez/db/RunDao.java 92.50% <ø> (-0.53%) ⬇️
api/src/main/java/marquez/db/models/RunRow.java 100.00% <ø> (+10.00%) ⬆️
...src/main/java/marquez/db/mappers/RunRowMapper.java 70.83% <70.83%> (ø)
...rc/main/java/marquez/db/models/ExtendedRunRow.java 63.63% <83.33%> (+13.63%) ⬆️
api/src/main/java/marquez/db/OpenLineageDao.java 95.75% <100.00%> (-0.03%) ⬇️
.../src/main/java/marquez/service/DatasetService.java 89.18% <100.00%> (ø)
api/src/main/java/marquez/service/JobService.java 91.66% <100.00%> (ø)
.../main/java/marquez/service/OpenLineageService.java 92.55% <100.00%> (+0.24%) ⬆️
api/src/main/java/marquez/service/RunService.java 87.83% <100.00%> (+0.16%) ⬆️
... and 1 more

📣 Codecov can now indicate which changes are the most critical in Pull Requests. Learn more

@collado-mike collado-mike mentioned this pull request Jul 20, 2022
7 tasks
Copy link
Member

@wslulciuc wslulciuc left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice bit of cleanup with a huge impact 👌

@wslulciuc wslulciuc merged commit ee44ae0 into main Jul 26, 2022
@wslulciuc wslulciuc deleted the runs_row_reduction branch July 26, 2022 20:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants