You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Collecting table state change information. - Table state change is a custom facet within outputs section of OpenLineage events.
Add new column operation to dataset_versions table
Update marquez.db.OpenLineageDao#updateBaseMarquezModel and marquez.db.OpenLineageDao#upsertLineageDataset methods to store table operation when present.
Exposing with API
Method marquez.api.DatasetResource#getDataset should be modified to contain optionally the latest table state change information.
Presenting on Web UI
This information should be shown on the popup box with dataset info similarly to "Run duration" & "Run State"
As we discussed offline, a good approach to rename might be to add the original dataset as an input and the target dataset as an output, while also creating a new version of the original dataset with a drop operation. From the UI perspective, users might see the original dataset as no longer existing, while the lineage persists to the new table.
Some of the OpenLineage integrations can report table operations, such as
We should record these operations in Marquez and reflect the new state of the table after the operation has completed.
The text was updated successfully, but these errors were encountered: