[CT-3401] [Bug] dbt retry fails to apply --full-refresh to downstream models and vars

[lennons301]

Is this a new bug in dbt-core?

• I believe this is a new bug in dbt-core
• I have searched the existing issues, and I could not find an existing issue for this bug

Current Behavior

dbt retry fails to pass --full-refresh flag from the original invocation to all models in the retry DAG after the first one

Haven't used dbt retry in earlier versions so unable to say if this is a regression or not

Expected Behavior

dbt retry creates a new DAG starting from the point of failure, continuing with everything downstream

All flags/args passed in the previous invocation should be applied to all models in the retry DAG

Steps To Reproduce

1. Create 2 incremental models. The second model should reference the first. In both models log the value of is_incremental with {{ log('is_incremental is' ~ is_incremental(), info=True)}}

2. Create a compilation error in model 1 and invoke dbt run --select +model_2 --full-refresh

This should set is_incremental() to False and

• produce a compilation error in model 1
• skip model 2

3. Correct your compilation error in model 1 and invoke dbt retry

Model one will now run, logging is_incremental = False (expected behaviour)
Model two will now run, logging is_incremental = True (not desired behaviour)

Relevant log output

'''
dbt retry
10:49:37  Running with dbt=1.6.0
10:49:37  Registered adapter: trino=1.6.0
10:49:37  is_incremental isFalse
10:49:37  Found 2 models
10:49:37  
10:49:38  Concurrency: 1 threads (target='local')
10:49:38  
10:49:38  1 of 2 START sql incremental model model_1  [RUN]
10:49:38  is_incremental isFalse
10:49:41  1 of 2 OK created sql incremental model model_1 [SUCCESS in 2.77s]
10:49:41  2 of 2 START sql incremental model model_2  [RUN]
10:49:41  is_incremental isTrue
'''

Environment

- OS: Ubuntu 20.04.6
- Python:3.10.13
- dbt:1.6.0

Which database adapter are you using with dbt?

other (mention it in "Additional Context")

Additional Context

Using dbt-trino 1.6.0

[lennons301]

This also seems to apply to vars passed in with --vars '{var_name: var_value}' - except the vars appear not be passed to the retry DAG at all, wheras --full-refresh is applied to the first task in a retry DAG, but not to everything downstream

[graciegoheen]

Notes from refinement:

• we decided that you should be able to override the path configs the the retry command
• for every other flag though, whatever was the setting for the original invocation, use the same flags for retry
• they're being retrieved from the args dict in the results.json (maybe these aren't complete? we see full-refresh and vars in this)
• https://github.com/dbt-labs/dbt-core/blob/main/core/dbt/task/retry.py#L110-L111

• dbt-core/core/dbt/task/retry.py

  Line 104 in b1b115d

  previous_args = {

• full-refresh isn't a flag, it's a CLI arg

[jtcohen6]

Silly repro case:

select {{ var('something') }} as {{ flags.FULL_REFRESH }}

$ dbt run --full-refresh --vars 'something: else'
$ mkdir state
$ mv target/run_results.json state/
$ dbt retry --state state/

After a bit of poking, I'm seeing the issue crop up here, when we initialize Flags from the dictionary of combined_args:

dbt-core/core/dbt/task/retry.py

Lines 108 to 110 in 1c9cec1

	combined_args = {**previous_args, **current_args}
	retry_flags = Flags.from_dict(cli_command, combined_args)

ipdb> combined_args['full_refresh']
True
ipdb> combined_args['vars']
{'something': 'else'}
ipdb> cli_command
<Command.RUN: 'run'>
ipdb> retry_flags
Flags()
ipdb> retry_flags.full_refresh
False
ipdb> retry_flags.vars
{'something': 'else'}

The combined_args dict has the appropriate values of --full-refresh and --vars. The retry_flags dict overwrites the value of full_refresh with the default. Even though the value of vars looks correct, we're not appropriately patching cli_vars somewhere, so we still encounter this runtime error:

Compilation Error in model my_model (models/my_model.sql)
  Required var 'something' not found in config:
  Vars supplied to my_model = {}

[aranke]

Discussed offline with @jtcohen6, comment doesn't apply but leaving for posterity.

Here's the list where we can add in --full-refresh to resolve this issue:

dbt-core/core/dbt/task/retry.py

Lines 22 to 31 in 3f7f7de

	OVERRIDE_PARENT_FLAGS = {
	"log_path",
	"output_path",
	"profiles_dir",
	"profiles_dir_exists_false",
	"project_dir",
	"defer_state",
	"deprecated_state",
	"target_path",
	}

[graciegoheen]

@aranke In case you didn't see the notes from refinement - "We decided that you should be able to override the path configs the the retry command. For every other flag though, whatever was the setting for the original invocation, use the same flags for retry."

This issue is about both full_refresh and vars

[jtcohen6]

From live discussion: Let's investigate the fix, and backport to v1.7 only. Rationale: v1.6 is now under "critical" support, and it is very much still usable, just with this known bug. Users will be able to upgrade to v1.7 for the fix.