Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

The deleteNonMatched incremental table option #1758

Open
AaronRobins opened this issue Jun 25, 2024 · 0 comments
Open

The deleteNonMatched incremental table option #1758

AaronRobins opened this issue Jun 25, 2024 · 0 comments

Comments

@AaronRobins
Copy link

Hey Dataform Team,

First let me say thanks for building a super useful platform big fan.

disclaimer - this may be a terrible idea, I am not across the full repository and its use cases.

  1. Currently, incremental tables hold onto rows which are in the target, but are no longer in the source
  2. This is because of the MERGE operation which is 'under the hood'
  3. Although its not a common use case, it can happen a row is dropped from source data and we too want that row dropped in the target table

My proposed solution would be:

  • Allow for a config item called deleteNonMatched which is a boolean and only available for incremental tables
  • If this config item is set to true, then the compiler will add in the following line to the stored procedure
WHEN NOT MATCHED BY SOURCE THEN DELETE

Currently the only workaround solutions I have found it to use a post operation or abandon the Incremental table all together and use a custom operation instead.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants