Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Regression] Existing Dynamic Tables Are Being Recreated Even Without Full Refresh #1267

Closed
2 tasks done
gseda19 opened this issue Dec 9, 2024 · 0 comments · Fixed by #1268
Closed
2 tasks done

[Regression] Existing Dynamic Tables Are Being Recreated Even Without Full Refresh #1267

gseda19 opened this issue Dec 9, 2024 · 0 comments · Fixed by #1268
Labels

Comments

@gseda19
Copy link

gseda19 commented Dec 9, 2024

Is this a regression in a recent version of dbt-snowflake?

  • I believe this is a regression in dbt-snowflake functionality
  • I have searched the existing issues, and I could not find an existing issue for this regression

Current Behavior

Dynamic tables are being recreated even without using full-refresh. This behavior is observed in dbt Cloud versionless and dbt Core (tested with dbt-core 1.8.9 and dbt-snowflake 1.9.0rc1).

This is not the case for dbt Cloud v1.7 and dbt Core using dbt-snowflake 1.8.4

cc: @amychen1776

Expected/Previous Behavior

When dbt run or dbt build is run, the existing dynamic tables should not be recreated without the full-refresh flag.

Steps To Reproduce

Using dbt Cloud versionless or dbt Core (dbt-snowflake 1.9.0rc1), run a dbt run or dbt build then check the logs and see that the existing dynamic tables are having CREATE OR REPLACE code.

dbt Cloud Versionless:
dynamic_table_2.sql:

{{ config(
    materialized="dynamic_table",
    target_lag="downstream",
    snowflake_warehouse="MY_WAREHOUSE",
    refresh_mode="AUTO", 

) }}
select * from my_first_dbt_model

Logs:

08:56:29 Applying ALTER to: "MY_DATABASE"."DBT_GSEDA"."DYNAMIC_TABLE_2"
08:56:29 Applying REPLACE to: "MY_DATABASE"."DBT_GSEDA"."DYNAMIC_TABLE_2"
08:56:29 Writing runtime sql for node "model.my_new_project.dynamic_table_2"
08:56:29 Using snowflake connection "model.my_new_project.dynamic_table_2"
08:56:29 On model.my_new_project.dynamic_table_2: /* {"app": "dbt", "dbt_version": "2024.11.25+e9c6e22", "profile_name": "user", "target_name": "default", "node_id": "model.my_new_project.dynamic_table_2"} */
create or replace dynamic table MY_DATABASE.dbt_gseda.dynamic_table_2
        target_lag = 'downstream'
        warehouse = MY_WAREHOUSE
        refresh_mode = AUTO

        initialize = ON_CREATE

        as (
            
select * from my_first_dbt_model
        )
08:56:30 SQL status: SUCCESS 1 in 1.239 seconds
08:56:30 6 of 17 OK created sql dynamic_table model dbt_gseda.dynamic_table_2 ........... [SUCCESS 1 in 3.98s]

dbt Core (adapter version: 1.9.0rc1; dbt version: 1.8.9):
dynamic_table_2_core.sql:

{{
    config(
        materialized="dynamic_table",
        target_lag="downstream",
        snowflake_warehouse="MY_WAREHOUSE",
        refresh_mode="AUTO",
        initialize="on_create"
    )
}}
select * from my_first_dbt_model

Logs:

08:54:39  Applying ALTER to: "MY_DATABASE"."DBT_GSEDA_CORE"."DYNAMIC_TABLE_2_CORE"
08:54:39  Applying REPLACE to: "MY_DATABASE"."DBT_GSEDA_CORE"."DYNAMIC_TABLE_2_CORE"
08:54:39  Writing runtime sql for node "model.snowflake_core_2.dynamic_table_2_core"
08:54:39  Using snowflake connection "model.snowflake_core_2.dynamic_table_2_core"
08:54:39  On model.snowflake_core_2.dynamic_table_2_core: /* {"app": "dbt", "dbt_version": "1.8.9", "profile_name": "snowflake_core_2", "target_name": "dev", "node_id": "model.snowflake_core_2.dynamic_table_2_core"} */
create or replace dynamic table MY_DATABASE.DBT_GSEDA_CORE.dynamic_table_2_core
        target_lag = 'downstream'
        warehouse = MY_WAREHOUSE
        refresh_mode = AUTO

        initialize = ON_CREATE

        as (


select * from my_first_dbt_model
        )
08:54:41  SQL status: SUCCESS 1 in 2.863 seconds
08:54:41  Sending event: {'category': 'dbt', 'action': 'run_model', 'label': 'd0e5e21f-fbcb-4222-911f-5d7faa085964', 'context': [<snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x108caf110>]}
08:54:41  1 of 10 OK created sql dynamic_table model DBT_GSEDA_CORE.dynamic_table_2_core . [SUCCESS 1 in 4.29s]

Please note that CREATE OR REPLACE is only present on subsequent runs but on initial run, it's just CREATE which I think is expected:

create dynamic table MY_DATABASE.dbt_gseda.dynamic_table_2
        target_lag = 'downstream'
        warehouse = MY_WAREHOUSE
        refresh_mode = AUTO

        initialize = ON_CREATE

        as (
            
select * from my_first_dbt_model
        )
08:08:42 Opening a new connection, currently in state init
08:08:45 SQL status: SUCCESS 1 in 3.161 seconds
08:08:45 6 of 17 OK created sql dynamic_table model dbt_gseda.dynamic_table_2 ........... [SUCCESS 1 in 3.26s]

Relevant log output

No response

Environment

- OS:
- Python:
- dbt-core (working version): dbt cloud v1.7 
- dbt-snowflake (working version): dbt-snowflake 1.8.4
- dbt-core (regression version): Versionless
- dbt-snowflake (regression version): 1.9.0rc1

Additional Context

No response

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant