Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Re-evaluate skip normalization if no new data feature #17578

Closed
gosusnp opened this issue Oct 4, 2022 · 5 comments
Closed

Re-evaluate skip normalization if no new data feature #17578

gosusnp opened this issue Oct 4, 2022 · 5 comments
Labels
frozen Not being actively worked on team/platform-move

Comments

@gosusnp
Copy link
Contributor

gosusnp commented Oct 4, 2022

What

We introduced a feature to skip normalization in certain cases #9672.
However, following a reset, normalization should run as well because some clean up on the destination side are only done by normalization (https://github.com/airbytehq/oncall/issues/741)
This behavior has been disabled by #17574 as a mitigation.

Also for context, there might be some discussions about how to better handle resets which may impact this feature
(slack thread + https://github.com/airbytehq/airbyte-internal-issues/issues/999)

@gosusnp
Copy link
Contributor Author

gosusnp commented Oct 4, 2022

One option might be to not skip normalization following a reset as well.

@killthekitten
Copy link
Contributor

killthekitten commented Jan 11, 2023

@gosusnp I wanted to highlight #9663 as an additional variable for your evaluation. Currently, the default behavior is to run normalization even after the sync has failed, which under certain circumstances erases data from the destination on a failure.

@rach
Copy link

rach commented Apr 12, 2023

This issue make airbyte nearly unusable and very unstable. I think that's ok with having sync falling, but having it wiping the data is a big issue which make the tool unreliable

@tsh356
Copy link

tsh356 commented Sep 3, 2023

Is this anywhere on the roadmap?

@bleonard bleonard added the frozen Not being actively worked on label Mar 22, 2024
@davinchia
Copy link
Contributor

Hi all, this is automatically handled by our Destinations today - any sort of typing and deduping is only run if needed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
frozen Not being actively worked on team/platform-move
Projects
None yet
Development

No branches or pull requests

7 participants