[CT-3294] [Bug] dbt retry does not respect target-path CLI flag / env variable

[kevin-chao]

Is this a new bug in dbt-core?

• I believe this is a new bug in dbt-core
• I have searched the existing issues, and I could not find an existing issue for this bug

Current Behavior

The current implementation of dbt retry seems to only look for run_results.json when:

• The default target-path is set (e.g. no configuration)
• target-path is set within dbt_project.yml

While the target-path configuration in dbt_project.yml can solve my issue today, I recognize that it has been deprecated and will likely no longer work in future versions. The other options I have are the CLI flag / environment variable, which when tested produce two different errors that boil down to these configurations not being supported.

Expected Behavior

When using dbt retry, I should be able to:

• Surface a flag like dbt retry --target-path /path/to/artifacts
• Set an environment variable like DBT_TARGET_PATH=/path/to/artifacts

Steps To Reproduce

1. Create a new file directory in dbt project (for me test_artifacts) and move run_results.json to that folder.
2. Set a flag or env var
   3a. dbt retry --target-path test_artifacts

➜ dbt retry --target-path test_artifacts
Usage: dbt retry [OPTIONS]
Try 'dbt retry -h' for help.

Error: No such option: --target-path Did you mean --target?

3b. In my .zshrc, set export DBT_TARGET_PATH=~/path/to/dbt/test_artifacts, source shell and run dbt retry

➜ dbt retry      
15:34:02  Running with dbt=1.6.4
15:34:02  Registered adapter: snowflake=1.6.4
15:34:06  Encountered an error:
Runtime Error
  Could not find previous run in 'target' target directory

4. In dbt_project.yml, set target-path: test_artifacts, run dbt retry

➜ dbt retry                   
15:39:54  Running with dbt=1.6.4
15:39:54  [WARNING]: Deprecated functionality
The `target-path` config in `dbt_project.yml` has been deprecated, and will no
longer be supported in a future version of dbt-core. If you wish to write dbt
artifacts to a custom directory, please use the --target-path CLI flag or
DBT_TARGET_PATH env var instead.
15:39:54  Registered adapter: snowflake=1.6.4
15:39:54  Unable to do partial parsing because saved manifest not found. Starting full parse.
15:40:30  Found 2577 models, 61 snapshots, 3577 tests, 90 seeds, 4 operations, 1783 sources, 0 exposures, 0 metrics, 985 macros, 0 groups, 0 semantic models
...
Successful run

Relevant log output

No response

Environment

- OS: MacOS, Sonoma 14.0
- Python: 3.10.4
- dbt: snowflake=1.6.4

Which database adapter are you using with dbt?

snowflake

Additional Context

No response

[dbeatty10]

Thanks for raising this @kevin-chao !

This looks like it was solved in dbt-core 1.6.6 by #8803:

• Override path-like args in dbt retry (#8682)

Could you give this a shot and see if it solves this for you?

[kevin-chao]

Hey @dbeatty10, thanks for the quick triage! I gave it a try with dbt-core==1.6.6, but no dice. Here are some logs when using DBT_TARGET_PATH as an environment variable (dbt retry doesn't take the --target-path flag) that I'm hoping helps debug.

When using dbt retry set to test/ directory

============================== 14:52:08.998563 | d7da2086-a55a-48f9-b343-9a8bda1ab450 ==============================
�[0m14:52:08.998563 [info ] [MainThread]: Running with dbt=1.6.6
�[0m14:52:08.998951 [debug] [MainThread]: running dbt with arguments {'printer_width': '80', 'indirect_selection': 'eager', 'write_json': 'True', 'log_cache_events': 'False', 'partial_parse': 'True', 'cache_selected_only': 'False', 'profiles_dir': '/Users/kevin/.dbt', 'debug': 'False', 'version_check': 'True', 'log_path': '/Users/kevin/dbt/logs', 'fail_fast': 'False', 'warn_error': 'None', 'use_colors': 'True', 'use_experimental_parser': 'False', 'no_print': 'None', 'quiet': 'False', 'log_format': 'default', 'static_parser': 'True', 'invocation_command': 'dbt retry', 'introspect': 'True', 'target_path': 'None', 'warn_error_options': 'WarnErrorOptions(include=[], exclude=[])', 'send_anonymous_usage_stats': 'True'}
�[0m14:52:09.186531 [debug] [MainThread]: Sending event: {'category': 'dbt', 'action': 'project_id', 'label': 'd7da2086-a55a-48f9-b343-9a8bda1ab450', 'context': [<snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x105ca58b0>]}
�[0m14:52:09.197814 [debug] [MainThread]: Sending event: {'category': 'dbt', 'action': 'adapter_info', 'label': 'd7da2086-a55a-48f9-b343-9a8bda1ab450', 'context': [<snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x107803a60>]}
�[0m14:52:09.198372 [info ] [MainThread]: Registered adapter: snowflake=1.6.4
�[0m14:52:09.240377 [debug] [MainThread]: checksum: 546b81fb56652c304d87abd676e84d4737d8a0c6b62160f4a6e79dcddbc842bb, vars: {}, profile: , target: , version: 1.6.6
�[0m14:52:10.576116 [debug] [MainThread]: Partial parsing enabled: 0 files deleted, 0 files added, 0 files changed.
�[0m14:52:10.576597 [debug] [MainThread]: Partial parsing enabled, no changes found, skipping parsing
�[0m14:52:11.008990 [debug] [MainThread]: Sending event: {'category': 'dbt', 'action': 'load_project', 'label': 'd7da2086-a55a-48f9-b343-9a8bda1ab450', 'context': [<snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x125f3eee0>]}
�[0m14:52:12.206905 [error] [MainThread]: Encountered an error:
Runtime Error
  Could not find previous run in 'target' target directory
�[0m14:52:12.207506 [debug] [MainThread]: Command `dbt retry` failed at 14:52:12.207445 after 3.24 seconds
�[0m14:52:12.207786 [debug] [MainThread]: Sending event: {'category': 'dbt', 'action': 'invocation', 'label': 'end', 'context': [<snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x1034eeb80>, <snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x152d58fd0>, <snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x152d58fa0>]}
�[0m14:52:12.208054 [debug] [MainThread]: Flushing usage events
�[0m14:53:55.114401 [debug] [MainThread]: Sending event: {'category': 'dbt', 'action': 'invocation', 'label': 'start', 'context': [<snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x105c2efa0>, <snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x108431ee0>, <snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x1084465b0>]}

When using dbt run set to test/ directory:

============================== 14:53:55.117193 | 4ad12cdc-9bf6-4e32-be39-78f854e8e0d2 ==============================
�[0m14:53:55.117193 [info ] [MainThread]: Running with dbt=1.6.6
�[0m14:53:55.117858 [debug] [MainThread]: running dbt with arguments {'printer_width': '80', 'indirect_selection': 'eager', 'write_json': 'True', 'log_cache_events': 'False', 'partial_parse': 'True', 'cache_selected_only': 'False', 'warn_error': 'None', 'version_check': 'True', 'profiles_dir': '/Users/kevin/.dbt', 'log_path': '/Users/kevin/dbt/logs', 'fail_fast': 'False', 'debug': 'False', 'use_colors': 'True', 'use_experimental_parser': 'False', 'no_print': 'None', 'quiet': 'False', 'log_format': 'default', 'introspect': 'True', 'warn_error_options': 'WarnErrorOptions(include=[], exclude=[])', 'static_parser': 'True', 'target_path': '/Users/kevin/dbt/test', 'invocation_command': 'dbt run -s stg_google_ads__account_history', 'send_anonymous_usage_stats': 'True'}
�[0m14:53:55.316020 [debug] [MainThread]: Sending event: {'category': 'dbt', 'action': 'project_id', 'label': '4ad12cdc-9bf6-4e32-be39-78f854e8e0d2', 'context': [<snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x108431880>]}
�[0m14:53:55.327699 [debug] [MainThread]: Sending event: {'category': 'dbt', 'action': 'adapter_info', 'label': '4ad12cdc-9bf6-4e32-be39-78f854e8e0d2', 'context': [<snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x108f8ebe0>]}
�[0m14:53:55.328393 [info ] [MainThread]: Registered adapter: snowflake=1.6.4
�[0m14:53:55.371382 [debug] [MainThread]: checksum: 546b81fb56652c304d87abd676e84d4737d8a0c6b62160f4a6e79dcddbc842bb, vars: {}, profile: , target: , version: 1.6.6
�[0m14:53:55.373585 [info ] [MainThread]: Unable to do partial parsing because saved manifest not found. Starting full parse.
�[0m14:53:55.373942 [debug] [MainThread]: Sending event: {'category': 'dbt', 'action': 'partial_parser', 'label': '4ad12cdc-9bf6-4e32-be39-78f854e8e0d2', 'context': [<snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x108f93850>]}
�[0m14:54:08.840438 [debug] [MainThread]: Sending event: {'category': 'dbt', 'action': 'experimental_parser', 'label': '4ad12cdc-9bf6-4e32-be39-78f854e8e0d2', 'context': [<snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x10cce7c70>]}
...
�[0m14:54:34.093047 [debug] [MainThread]: Command end result
�[0m14:54:34.729373 [info ] [MainThread]: 
�[0m14:54:34.729889 [info ] [MainThread]: �[32mCompleted successfully�[0m
�[0m14:54:34.730174 [info ] [MainThread]: 
�[0m14:54:34.730419 [info ] [MainThread]: Done. PASS=1 WARN=0 ERROR=0 SKIP=0 TOTAL=1
�[0m14:54:34.730822 [debug] [MainThread]: Command `dbt run` succeeded at 14:54:34.730767 after 39.65 seconds
�[0m14:54:34.731093 [debug] [MainThread]: Sending event: {'category': 'dbt', 'action': 'invocation', 'label': 'end', 'context': [<snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x105c2efa0>, <snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x108f8ebe0>, <snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x12adc2430>]}
�[0m14:54:34.731360 [debug] [MainThread]: Flushing usage events

Notably in the debug statement, the environment variable selectively shows up for target-path. I can also confirm that dbt run created files in /dbt/test but did not for dbt retry

[dbeatty10]

Thanks for giving that a try @kevin-chao !

I tried it myself and was able to see the same thing as you.

It seems like the missing piece might be what you called out in your original issue:

• dbt retry doesn't support specifying the --target-path flag here

Possible solution

In my local testing with an editable version of dbt-core, both --target-path and DBT_TARGET_PATH worked when I added this single line within here:

@p.target_path

Caveat

I get the following warning (which I think is expected due to #8638 and #8160 (comment):

19:01:40  Warning: The state and target directories are the same: 'test_2'. This could lead to missing changes due to overwritten state including non-idempotent retries.

Toggle to see full log output

(core_editable) $ dbt retry --target-path test_2
19:01:40  Running with dbt=1.8.0-a1
19:01:40  Registered adapter: postgres=1.8.0-a1
19:01:40  Warning: The state and target directories are the same: 'test_2'. This could lead to missing changes due to overwritten state including non-idempotent retries.
19:01:40  Found 1 model, 0 sources, 0 exposures, 0 metrics, 401 macros, 0 groups, 0 semantic models
19:01:40  
19:01:40  Concurrency: 5 threads (target='postgres')
19:01:40  
19:01:40  1 of 1 START sql view model dbt_dbeatty.my_model ............................... [RUN]
19:01:40  1 of 1 OK created sql view model dbt_dbeatty.my_model .......................... [CREATE VIEW in 0.12s]
19:01:40  
19:01:40  Finished running 1 view model in 0 hours 0 minutes and 0.33 seconds (0.33s).
19:01:41  
19:01:41  Completed successfully
19:01:41  
19:01:41  Done. PASS=1 WARN=0 ERROR=0 SKIP=0 TOTAL=1

Acceptance criteria for implementation

• Include a functional test for dbt retry --target-path
• Include a functional test for export DBT_TARGET_PATH=something and dbt retry
• Make sure both tests fail
• Add --target-path to dbt retry by adding @p.target_path here
• Make sure both tests pass

[owenprough-sift]

My organization encountered this today and it's a bummer that the workaround is "use a deprecated dbt_project.yml configuration option". I'm looking forward to the resolution of this issue.

[Ynabokov]

My team has also encountered this issue after updating dbt from 1.4.4 to 1.7.6. We would really appreciate the fix for this bug.

[ChenyuLInx]

@graciegoheen As of the change we added here, you can specify just --state path and it should find the result.json in the state dir and use that.

[dbeatty10]

As @ChenyuLInx already mentioned, the snippet here shows why providing --state works.

I did a write-up in #9575 (comment) to give examples why we should enable DBT_TARGET_PATH and --target-path.

If we implement #9575, then it would resolve #8948 as well.

[the4thamigo-uk]

Hi, Ive hit an issue that relies on this fix. I was wondering when this is likely to be released?

[graciegoheen]

@the4thamigo-uk our next 1.7 patch is currently scheduled for wednesday!