-
Notifications
You must be signed in to change notification settings - Fork 1.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add experimental support for emitting lineage events #16242
Conversation
This PR adds experimental support for emitting lineage events, starting with lineage tracking for flow and task results. NOTE: Lineage events are only consumed by Prefect Cloud at this time, and the Cloud consumer is still under development. Closes CLOUD-705, CLOUD-791
CodSpeed Performance ReportMerging #16242 will not alter performanceComparing Summary
|
…efect into emit-lineage-events
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
One request on the name of the new module, but otherwise LGTM!
src/prefect/experimental/__init__.py
Outdated
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Since stuff in the module appears to only be for internal use, could you name this module _experimental
to communicate its internal nature? An added benefit is that our pyright
type completeness checks will ignore this module if it's prefixed with an underscore.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Totally! Thanks @desertaxle.
I'm getting the following error when I try run a flow using this branch with
|
@zhen0 Can you share the flow? |
Yep - let me see if I can simplify to an MRE |
After digging in further I can see this happening in main so likely not connected to this PR. 👍 |
This PR adds experimental support for emitting lineage events, starting with lineage tracking for flow and task results.
We attempt to make the least-invasive changes possible by keeping logic related to lineage event construction in a utility module rather than giving
Block
storage classes the ability to produce lineage events or event URIs.NOTE: Lineage events are only consumed by Prefect Cloud at this time, and the Cloud consumer is still under development.
Closes CLOUD-705, CLOUD-791
Checklist
<link to issue>
"mint.json
.