Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

tolerate null transformation attrs in ColumnLineageInputField #2600

Conversation

davidjgoss
Copy link
Contributor

@davidjgoss davidjgoss commented Aug 17, 2023

Problem

The transformationType and transformationDescription attributes for column-level lineage are optional at the database level and can be safely omitted when pushing in lineage events.

However, because they're marked as non-nullable in the Java client's representation, an exception is thrown when deserialising a response from the column lineage endpoint (thanks to Lombok-generated Objects.requireNonNull in the constructor).

Solution

Remove the @NonNull annotation from the client class and the @NotNull from the model class. Add a test that would fail without this change. Validated by running the stack locally.

One-line summary:
tolerate null transformation attrs in field model

Checklist

  • You've signed-off your work
  • Your changes are accompanied by tests (if relevant)
  • Your change contains a small diff and is self-contained
  • You've updated any relevant documentation (if relevant)
  • You've included a one-line summary of your change for the CHANGELOG.md (Depending on the change, this may not be necessary).
  • You've versioned your .sql database schema migration according to Flyway's naming convention (if relevant)
  • You've included a header in any source code files (if relevant)

@boring-cyborg boring-cyborg bot added api API layer changes client/java labels Aug 17, 2023
@davidjgoss davidjgoss force-pushed the bug/java-client-transformation-nullable branch 2 times, most recently from 82c81da to 73ebf93 Compare August 17, 2023 09:09
@codecov
Copy link

codecov bot commented Aug 17, 2023

Codecov Report

Merging #2600 (2ddca22) into main (e48ac69) will not change coverage.
The diff coverage is n/a.

@@            Coverage Diff            @@
##               main    #2600   +/-   ##
=========================================
  Coverage     83.30%   83.30%           
  Complexity     1287     1287           
=========================================
  Files           243      243           
  Lines          5935     5935           
  Branches        279      279           
=========================================
  Hits           4944     4944           
  Misses          844      844           
  Partials        147      147           

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

@pawel-big-lebowski
Copy link
Collaborator

Indeed, transformation attrs are not required within the spec -> https://github.com/OpenLineage/OpenLineage/blob/main/spec/facets/ColumnLineageDatasetFacet.json#L51
so they shouldn't be obligatory in Marquez.

Copy link
Collaborator

@pawel-big-lebowski pawel-big-lebowski left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me!

Signed-off-by: David Goss <david.goss@matillion.com>
@davidjgoss davidjgoss force-pushed the bug/java-client-transformation-nullable branch from 73ebf93 to 2ddca22 Compare August 22, 2023 07:30
Copy link
Member

@wslulciuc wslulciuc left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM 👍

@wslulciuc wslulciuc merged commit e85127c into MarquezProject:main Aug 22, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
api API layer changes client/java
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants