Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature] dbt_clone sf implementation #664

Merged
merged 32 commits into from
Jul 11, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
32 commits
Select commit Hold shift + click to select a range
032b5b5
update RELEASE_BRANCH env
McKnight-42 Apr 14, 2023
16470c2
Merge branch 'main' of gist.github.com:dbt-labs/dbt-snowflake
McKnight-42 Apr 25, 2023
4c475f1
Merge branch 'main' of gist.github.com:dbt-labs/dbt-snowflake
McKnight-42 May 25, 2023
81d5d5d
Merge branch 'main' of gist.github.com:dbt-labs/dbt-snowflake
McKnight-42 May 26, 2023
f1d8078
Merge branch 'main' of gist.github.com:dbt-labs/dbt-snowflake
McKnight-42 May 30, 2023
2773177
Merge branch 'main' of gist.github.com:dbt-labs/dbt-snowflake
McKnight-42 May 30, 2023
77a2b66
Merge branch 'main' of gist.github.com:dbt-labs/dbt-snowflake
McKnight-42 May 30, 2023
d9d3d2d
Merge branch 'main' of gist.github.com:dbt-labs/dbt-snowflake
McKnight-42 Jun 1, 2023
a51efc2
Merge branch 'main' of gist.github.com:dbt-labs/dbt-snowflake
McKnight-42 Jun 2, 2023
df68367
Merge branch 'main' of gist.github.com:dbt-labs/dbt-snowflake
McKnight-42 Jun 12, 2023
d9ab1ac
Merge branch 'main' of gist.github.com:dbt-labs/dbt-snowflake
McKnight-42 Jun 14, 2023
942473f
Merge branch 'main' of gist.github.com:dbt-labs/dbt-snowflake
McKnight-42 Jun 16, 2023
c5db89f
initial implementation of snowflake dbt_clone macros and test design …
McKnight-42 Jun 16, 2023
949a242
prove tests throw expected values if expected use caes bases on bool …
McKnight-42 Jun 21, 2023
c79fb2c
prove tests throw expected values if expected use caes bases on bool …
McKnight-42 Jun 21, 2023
7c03678
Merge branch 'main' of gist.github.com:dbt-labs/dbt-snowflake into fe…
McKnight-42 Jun 21, 2023
470937b
start to move towards importing adapter version of test
McKnight-42 Jun 21, 2023
246f48f
macro name fix, file rename
McKnight-42 Jun 21, 2023
13edf7c
change state_relation to defer_relation to match change in dbt-core
McKnight-42 Jun 21, 2023
08eef26
change pointer in dev-requirements and add changelog
McKnight-42 Jun 22, 2023
f11b037
Merge branch 'main' into feature/dbt-clone-sf
McKnight-42 Jun 22, 2023
06daf13
Merge branch 'main' of gist.github.com:dbt-labs/dbt-snowflake into fe…
McKnight-42 Jun 26, 2023
6a1840b
Merge branch 'feature/dbt-clone-sf' of gist.github.com:dbt-labs/dbt-s…
McKnight-42 Jun 26, 2023
2414226
add clean_up method to drop alt schema names after tests run
McKnight-42 Jun 26, 2023
157fe40
Merge branch 'main' into feature/dbt-clone-sf
McKnight-42 Jun 28, 2023
f8fef89
Merge branch 'main' of gist.github.com:dbt-labs/dbt-snowflake into fe…
McKnight-42 Jun 29, 2023
24b8e31
change pointer back to main
McKnight-42 Jun 29, 2023
dd54c4c
Merge branch 'feature/dbt-clone-sf' of gist.github.com:dbt-labs/dbt-s…
McKnight-42 Jun 29, 2023
18c7f49
Merge branch 'main' into feature/dbt-clone-sf
aranke Jul 6, 2023
5197f44
Merge branch 'main' into feature/dbt-clone-sf
McKnight-42 Jul 7, 2023
38d60fc
Merge branch 'main' into feature/dbt-clone-sf
McKnight-42 Jul 11, 2023
a121695
clone transient table test
McKnight-42 Jul 11, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions .changes/unreleased/Features-20230622-114621.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
kind: Features
body: add dbt-snowflake portion of dbt_clone functionality
time: 2023-06-22T11:46:21.399594-05:00
custom:
Author: McKnight-42 aranke
Issue: "7256"
11 changes: 11 additions & 0 deletions dbt/include/snowflake/macros/materializations/clone.sql
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
{% macro snowflake__can_clone_table() %}
{{ return(True) }}
{% endmacro %}

{% macro snowflake__create_or_replace_clone(this_relation, defer_relation) %}
create or replace
{{ "transient" if config.get("transient", true) }}
table {{ this_relation }}
clone {{ defer_relation }}
{{ "copy grants" if config.get("copy_grants", false) }}
{% endmacro %}
83 changes: 83 additions & 0 deletions tests/functional/adapter/dbt_clone/test_dbt_clone.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,83 @@
import pytest
import shutil
import os
from copy import deepcopy
from dbt.tests.util import run_dbt
from dbt.tests.adapter.dbt_clone.test_dbt_clone import BaseClonePossible


class TestSnowflakeClonePossible(BaseClonePossible):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do we need to test transient and copy_grants?

Copy link
Contributor Author

@McKnight-42 McKnight-42 Jul 11, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this admittedly is more pulled from Jeremey's draft pr where he created a snowflake implementation where he does talk about the use case for transient https://github.com/dbt-labs/dbt-core/pull/7258/files#diff-073e6ed96ac92033f0b921e061b47226b87d4f358350a1ed94fc94165f247b7aR39

while in the docs does mention copy grants

@pytest.fixture(autouse=True)
def clean_up(self, project):
yield
with project.adapter.connection_named("__test"):
relation = project.adapter.Relation.create(
database=project.database, schema=f"{project.test_schema}_SEEDS"
)
project.adapter.drop_schema(relation)

relation = project.adapter.Relation.create(
database=project.database, schema=project.test_schema
)
project.adapter.drop_schema(relation)

pass


table_model_1_sql = """
{{ config(
materialized='table',
transient=true,
) }}

select 1 as fun
"""


class TestSnowflakeCloneTrainsentTable:
@pytest.fixture(scope="class")
def models(self):
return {
"table_model.sql": table_model_1_sql,
}

@pytest.fixture(scope="class")
def other_schema(self, unique_schema):
return unique_schema + "_other"

@pytest.fixture(scope="class")
def profiles_config_update(self, dbt_profile_target, unique_schema, other_schema):
outputs = {"default": dbt_profile_target, "otherschema": deepcopy(dbt_profile_target)}
outputs["default"]["schema"] = unique_schema
outputs["otherschema"]["schema"] = other_schema
return {"test": {"outputs": outputs, "target": "default"}}

def copy_state(self, project_root):
state_path = os.path.join(project_root, "state")
if not os.path.exists(state_path):
os.makedirs(state_path)
shutil.copyfile(
f"{project_root}/target/manifest.json", f"{project_root}/state/manifest.json"
)

def run_and_save_state(self, project_root, with_snapshot=False):
results = run_dbt(["run"])
assert len(results) == 1
assert not any(r.node.deferred for r in results)

self.copy_state(project_root)

def test_can_clone_transient_table(self, project, other_schema):
project.create_test_schema(other_schema)
self.run_and_save_state(project.project_root)

clone_args = [
"clone",
"--state",
"state",
"--target",
"otherschema",
]

results = run_dbt(clone_args)
assert len(results) == 1