Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add experimental support for emitting lineage events #16242

Merged
merged 29 commits into from
Dec 12, 2024
Merged

Conversation

abrookins
Copy link
Collaborator

@abrookins abrookins commented Dec 6, 2024

This PR adds experimental support for emitting lineage events, starting with lineage tracking for flow and task results.

We attempt to make the least-invasive changes possible by keeping logic related to lineage event construction in a utility module rather than giving Block storage classes the ability to produce lineage events or event URIs.

NOTE: Lineage events are only consumed by Prefect Cloud at this time, and the Cloud consumer is still under development.

Closes CLOUD-705, CLOUD-791

Checklist

  • This pull request references any related issue by including "closes <link to issue>"
    • If no issue exists and your change is not a small fix, please create an issue first.
  • If this pull request adds new functionality, it includes unit tests that cover the changes
  • If this pull request removes docs files, it includes redirect settings in mint.json.
  • If this pull request adds functions or classes, it includes helpful docstrings.

This PR adds experimental support for emitting lineage events,
starting with lineage tracking for flow and task results.

NOTE: Lineage events are only consumed by Prefect Cloud at this
time, and the Cloud consumer is still under development.

Closes CLOUD-705, CLOUD-791
Copy link

codspeed-hq bot commented Dec 9, 2024

CodSpeed Performance Report

Merging #16242 will not alter performance

Comparing emit-lineage-events (05059bf) with main (dce8eca)

Summary

✅ 3 untouched benchmarks

Copy link
Member

@desertaxle desertaxle left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One request on the name of the new module, but otherwise LGTM!

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since stuff in the module appears to only be for internal use, could you name this module _experimental to communicate its internal nature? An added benefit is that our pyright type completeness checks will ignore this module if it's prefixed with an underscore.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Totally! Thanks @desertaxle.

@zhen0
Copy link
Member

zhen0 commented Dec 12, 2024

I'm getting the following error when I try run a flow using this branch with PREFECT_EXPERIMENTS_LINEAGE_EVENTS_ENABLED set to true:

Encountered an error while persisting result: object str can't be used in 'await' expression Execution will continue, but the result has not been persisted

@abrookins
Copy link
Collaborator Author

@zhen0 Can you share the flow?

@zhen0
Copy link
Member

zhen0 commented Dec 12, 2024

@zhen0 Can you share the flow?

Yep - let me see if I can simplify to an MRE

@zhen0
Copy link
Member

zhen0 commented Dec 12, 2024

After digging in further I can see this happening in main so likely not connected to this PR. 👍

@abrookins abrookins merged commit f57b48c into main Dec 12, 2024
38 checks passed
@abrookins abrookins deleted the emit-lineage-events branch December 12, 2024 18:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants