Fill run state of the parent run when it is created by child run #2296
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Signed-off-by: Minkyu Park minkyu.park.200@gmail.com
Problem
Airflow integration is not sending the run event for the DAG but it is enclosed as a
parent
facet of task runs events, and when the marquez receives OpenLineage event withparent
facet, it creates parent job and run if they don't exist. However, the created parent run is missing its run state, and it makes a little weird when getting the runs for the DAG job. For example when the marquez receives theSTART
event, the the parent run is being created and its status isRUNNING
but it does not have astartedAt
information.Solution
This PR adds a run state to the parent run at its creation time, honoring the way that it determines the state of the parent run. If the parent run is created with the
START
event, its run state becomesRUNNING
and it addsstarted_at
to the database. If the parent run is created with theCOMPLETE
event, its run state becomesCOMPLETE
and it addsended_at
to the database.Checklist
CHANGELOG.md
with details about your change under the "Unreleased" section (if relevant, depending on the change, this may not be necessary).sql
database schema migration according to Flyway's naming convention (if relevant)