[CT-1301] Behavior change: full_refresh: true config should respect --full-refresh, -f flag

[jtcohen6]

https://docs.getdbt.com/reference/resource-configs/full_refresh

Current behavior

When configured, full_refresh: true|false means: "always/never full-refresh, and always ignore the --full-refresh flag"

Added way back in #2438 / v0.18

Proposed change

• full_refresh: true should mean "always respect the --full-refresh flag"
• full_refresh: false should mean "never full refresh, under any circumstances"

That's trickier for intuition, and harder to explain, but definitely more desirable in terms of the behavior it enables. I'm not sure why you'd always want to full-refresh an incremental model; at that point it's really just a table materialization.

If this is set at the project level:

models:
  +full_refresh: false

It's not possible to override that setting for one particular model ("actually I want this model to respect the --full-refresh flag"). This doesn't work as I'd hoped it might:

{{ config(materialized = 'incremental', full_refresh = none) }}

Is this a breaking change?

Only for people who currently make use of full_refresh: true. Again, I'm not actually sure what the use case for doing that would be!

[b-per]

The only reason I could think of people using full_refresh: true would be from some env_var(), e.g. always doing a full-refresh in Dev (and maybe limiting the number of rows) but keeping it incremental in Prod. This looks very niche though. Most of the time that I need this parameter it is to set it to false to prevent FR.

[github-actions]

This issue has been marked as Stale because it has been open for 180 days with no activity. If you would like the issue to remain open, please comment on the issue or else it will be closed in 7 days.

[b-per]

I think that it is worth keeping this issue active

[github-actions]

This issue has been marked as Stale because it has been open for 180 days with no activity. If you would like the issue to remain open, please comment on the issue or else it will be closed in 7 days.

[b-per]

Commenting to keep it on! I still think that we could improve incremental models handling.

[jtcohen6]

@b-per I agree! I think we'd want to find some way to change the default here, without breaking existing users' projects. If you can think of a clever way to do that, without requiring another opt-in config (full_refresh_config_respect_flag: true), I'm all ears.

This behavior would then also diverge from --store-failures / store_failures, where the node-level config always takes precedence over the flag. Should we change that behavior too, to match?

[andrii0yerko]

I had faced the same problem.

Did anyone find any workarounds to achieve respect for the --full-refresh flag?

I tried things like

{{
    config(
        materialized='incremental',
        full_refresh = flags.FULL_REFRESH,
        ...
}}

but it doesn't work. Seems that full_refresh cannot be set dynamically for some reasons