Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Sync Mode: Incremental | Deduped without history #5618

Closed
m-ronchi opened this issue Aug 25, 2021 · 2 comments
Closed

Sync Mode: Incremental | Deduped without history #5618

m-ronchi opened this issue Aug 25, 2021 · 2 comments
Labels
type/enhancement New feature or request

Comments

@m-ronchi
Copy link
Contributor

Tell us about the problem you're trying to solve

for data that is already historic (e.g. google ads reports or facebook marketing insights) we may not need to have an additional _scd table.
furthermore, with a large lookback window and frequent syncs, it is possible that values within the lookback window change slightly between each sync, resulting in a huge amount of _scd data (e.g. daily syncs with 28-day lookback can be 28x more data in the _scd table than in the final table. hourly or 5 minute sync interval will just explode that number)

Describe the solution you’d like

an option in the UI to not persist the _scd table (i.e. mark it as 'ephemeral' in dbt)

Describe the alternative you’ve considered or used

customize the dbt project and replace basic normalization, but it doesn't work on kube (#5091) and it is also clunkier to use on multiple connections

Additional context

I am trying to reduce the amount of excess data stored on redshift, this + #2227 would be great

@m-ronchi m-ronchi added the type/enhancement New feature or request label Aug 25, 2021
@ChristopheDuong
Copy link
Contributor

THis is probably duplicating this issue: #3487

@marcosmarxm
Copy link
Member

Closing as duplicated. @m-ronchi let me know if you think #3487 is not equal to your request.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
type/enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

3 participants