Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

minimize downtime for incremental models with full-refresh #525

Closed
drewbanin opened this issue Sep 12, 2017 · 4 comments · Fixed by #1682
Closed

minimize downtime for incremental models with full-refresh #525

drewbanin opened this issue Sep 12, 2017 · 4 comments · Fixed by #1682
Labels
bug Something isn't working enhancement New feature or request

Comments

@drewbanin
Copy link
Contributor

drewbanin commented Sep 12, 2017

incremental models should:

  1. build
  2. drop
  3. rename

0.8.3 and 0.9.0a1 instead do drop, build, rename, leading to a potentially significant amount of time during which the model is unavailable for querying

@drewbanin drewbanin added the enhancement New feature or request label Sep 12, 2017
@igrayson
Copy link
Contributor

igrayson commented Sep 13, 2017

Should this be labeled as a bug? As of 0.8.3, the new behavior no longer matches our docs @ https://dbt.readme.io/v0.8/reference#section-refresh-incremental-models ; specifically, --full-refresh doesn't cause dbt to treat models as table models (table models continue to have the desired behavior of build/drop/rename in 0.8.3).

If you provide the --full-refresh argument to dbt run, dbt will treat incremental models as table models.

This also means that 0.8.3+ users can temporarily configure an incremental model with materialized: table to get a working --full-refresh semantic.

@drewbanin drewbanin added the bug Something isn't working label Sep 13, 2017
@drewbanin
Copy link
Contributor Author

Thanks @igrayson

@drewbanin drewbanin added this to the Adapter Parity milestone Mar 6, 2018
@cmcarthur cmcarthur modified the milestones: Adapter Parity, 0.10.2 Apr 26, 2018
@drewbanin
Copy link
Contributor Author

see also: #653

@drewbanin
Copy link
Contributor Author

The 0.13.x/0.14.0 implementation of incremental models for redshift/postgres is not transactionally consistent if an error occurs in an incremental model in --full-refresh mode. We should try to fix that at the same time we make minimize the incremental downtime for rebuilds

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working enhancement New feature or request
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants