-
Notifications
You must be signed in to change notification settings - Fork 323
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Ensure job data in lineage query is not null or empty #2253
Conversation
Signed-off-by: wslulciuc <willy@datakin.com>
Codecov Report
@@ Coverage Diff @@
## main #2253 +/- ##
============================================
- Coverage 77.07% 76.99% -0.08%
- Complexity 1170 1171 +1
============================================
Files 222 222
Lines 5317 5326 +9
Branches 425 426 +1
============================================
+ Hits 4098 4101 +3
- Misses 747 751 +4
- Partials 472 474 +2
📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more |
Signed-off-by: wslulciuc <willy@datakin.com>
Signed-off-by: wslulciuc <willy@datakin.com>
Signed-off-by: wslulciuc <willy@datakin.com>
Signed-off-by: wslulciuc <willy@datakin.com>
…ided Signed-off-by: wslulciuc <willy@datakin.com>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nice! 🥳
Signed-off-by: wslulciuc <willy@datakin.com>
Problem
An unknown edge case in our lineageAPI exists when querying for a job
uuid
that has no lineage when callingLineageDao.getLineage()
, yet is associated with a given datasetD
. That is, given a datasetD
, a reverse job lookup is performed usingLineageDao.getJobFromInputOrOutput()
(which returns the jobuuid
that produced datasetD
) but when querying for the lineage data for the returned job an empty set is returned, therefore resulting in a backend exception (=5xx
status code) when invokingLineageDao.getJobFromInputOrOutput()
.Solution
This PR ensures the lineage data for a job is not empty (an empty lineage graph is returned instead) and introduces logs to better understand the scenario in which this unknown edge case occurs for debugging purposes. This PR also adds a API check to ensure the query param
nodeID
provided in the lineageAPI call exists.Checklist
CHANGELOG.md
with details about your change under the "Unreleased" section (if relevant, depending on the change, this may not be necessary).sql
database schema migration according to Flyway's naming convention (if relevant)