Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fix(spark): Finegrained lineage is emitted on the DataJob and not on the emitted Datasets. #11956

Merged
merged 15 commits into from
Jan 6, 2025
Merged

fix(spark): Finegrained lineage is emitted on the DataJob and not on the emitted Datasets. #11956

merged 15 commits into from
Jan 6, 2025

Conversation

treff7es
Copy link
Contributor

@treff7es treff7es commented Nov 26, 2024 •
edited
Loading

  • Major changes:
    • Finegrained lineage is emitted on the DataJob, not the emitted Datasets. This is the correct behaviour which was not correct earlier. This causes earlier emitted fine-grained lineages to not be overwritten by the new ones.
      You can remove the old lineages by setting spark.datahub.legacyLineageCleanup.enabled=true. Make sure you have the latest server if you enable it with patch support. (this was introduced since 0.2.17-rc5)
  • Changes:
    • Add option to remove legacy lineages from older Spark Plugin runs. This will remove those lineages from the Datasets which it adds to DataJob -> spark.datahub.legacyLineageCleanup.enabled
  • Fixes:
    • Fix handling map transformation in the lineage. Earlier, it generated the wrong lineage for map transformation.

Checklist

  • The PR conforms to DataHub's Contributing Guideline (particularly Commit Message Format)
  • Links to related issues (if applicable)
  • Tests for the changes have been added/updated (if applicable)
  • Docs related to the changes have been added/updated (if applicable). If a new feature has been added a Usage Guide has been added for the same.
  • For any breaking change/potential downtime/deprecation/big changes an entry has been made in Updating DataHub

@github-actions github-actions bot added the ingestion PR or Issue related to the ingestion of metadata label Nov 26, 2024
@datahub-cyborg datahub-cyborg bot added the needs-review Label for PRs that need review from a maintainer. label Nov 26, 2024
@treff7es
- Add DataJobInputOutputPatchBuilder
7c452c0 
- Fix finegrained lineage generation
@treff7es treff7es changed the title fix(spark): Handle spark rdd map where input and output is the same fix(spark): Fix Finegrained lineage and handle spark rdd map where input and output is the same Dec 4, 2024
asikowitz
asikowitz reviewed Dec 4, 2024
View reviewed changes
Copy link
Collaborator

@asikowitz asikowitz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A couple initial nits

...src/main/java/com/linkedin/metadata/aspect/patch/builder/DataJobInputOutputPatchBuilder.java Outdated Show resolved Hide resolved
...src/main/java/com/linkedin/metadata/aspect/patch/builder/DataJobInputOutputPatchBuilder.java Outdated Show resolved Hide resolved
@datahub-cyborg datahub-cyborg bot added pending-submitter-response Issue/request has been reviewed but requires a response from the submitter and removed needs-review Label for PRs that need review from a maintainer. labels Dec 4, 2024
@datahub-cyborg datahub-cyborg bot added needs-review Label for PRs that need review from a maintainer. and removed pending-submitter-response Issue/request has been reviewed but requires a response from the submitter labels Dec 5, 2024
@treff7es
Generating lineage on DataJob and not on Dataset which is the support…
719ece1 
…ed and correct way

Add option to incrementally remove lineages which was set by the old plugin
@treff7es
Merge branch 'master' into spark_map_fix
c5bceb3
@datahub-cyborg datahub-cyborg bot added pending-submitter-merge and removed needs-review Label for PRs that need review from a maintainer. labels Dec 13, 2024
@treff7es treff7es changed the title fix(spark): Fix Finegrained lineage and handle spark rdd map where input and output is the same fix(spark): Finegrained lineage is emitted on the DataJob and not on the emitted Datasets. Jan 6, 2025
@treff7es treff7es merged commit 3316d40 into datahub-project:master Jan 6, 2025
38 checks passed
llance pushed a commit to llance/datahub that referenced this pull request Jan 13, 2025
@treff7es @llance
fix(spark): Finegrained lineage is emitted on the DataJob and not on …
c551785 
…the emitted Datasets. (datahub-project#11956)
llance pushed a commit to llance/datahub that referenced this pull request Jan 13, 2025
@treff7es @llance
fix(spark): Finegrained lineage is emitted on the DataJob and not on …
d83bee5 
…the emitted Datasets. (datahub-project#11956)
brock-acryl pushed a commit to brock-acryl/datahub that referenced this pull request Jan 30, 2025
@treff7es @chakru-r
fix(spark): Finegrained lineage is emitted on the DataJob and not on …
41921da 
…the emitted Datasets. (datahub-project#11956)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@asikowitz asikowitz asikowitz approved these changes

Assignees
No one assigned
Labels
ingestion PR or Issue related to the ingestion of metadata pending-submitter-merge
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@treff7es @asikowitz