-
Notifications
You must be signed in to change notification settings - Fork 179
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Feature] dbt_clone sf implementation #664
Merged
Merged
Changes from all commits
Commits
Show all changes
32 commits
Select commit
Hold shift + click to select a range
032b5b5
update RELEASE_BRANCH env
McKnight-42 16470c2
Merge branch 'main' of gist.github.com:dbt-labs/dbt-snowflake
McKnight-42 4c475f1
Merge branch 'main' of gist.github.com:dbt-labs/dbt-snowflake
McKnight-42 81d5d5d
Merge branch 'main' of gist.github.com:dbt-labs/dbt-snowflake
McKnight-42 f1d8078
Merge branch 'main' of gist.github.com:dbt-labs/dbt-snowflake
McKnight-42 2773177
Merge branch 'main' of gist.github.com:dbt-labs/dbt-snowflake
McKnight-42 77a2b66
Merge branch 'main' of gist.github.com:dbt-labs/dbt-snowflake
McKnight-42 d9d3d2d
Merge branch 'main' of gist.github.com:dbt-labs/dbt-snowflake
McKnight-42 a51efc2
Merge branch 'main' of gist.github.com:dbt-labs/dbt-snowflake
McKnight-42 df68367
Merge branch 'main' of gist.github.com:dbt-labs/dbt-snowflake
McKnight-42 d9ab1ac
Merge branch 'main' of gist.github.com:dbt-labs/dbt-snowflake
McKnight-42 942473f
Merge branch 'main' of gist.github.com:dbt-labs/dbt-snowflake
McKnight-42 c5db89f
initial implementation of snowflake dbt_clone macros and test design …
McKnight-42 949a242
prove tests throw expected values if expected use caes bases on bool …
McKnight-42 c79fb2c
prove tests throw expected values if expected use caes bases on bool …
McKnight-42 7c03678
Merge branch 'main' of gist.github.com:dbt-labs/dbt-snowflake into fe…
McKnight-42 470937b
start to move towards importing adapter version of test
McKnight-42 246f48f
macro name fix, file rename
McKnight-42 13edf7c
change state_relation to defer_relation to match change in dbt-core
McKnight-42 08eef26
change pointer in dev-requirements and add changelog
McKnight-42 f11b037
Merge branch 'main' into feature/dbt-clone-sf
McKnight-42 06daf13
Merge branch 'main' of gist.github.com:dbt-labs/dbt-snowflake into fe…
McKnight-42 6a1840b
Merge branch 'feature/dbt-clone-sf' of gist.github.com:dbt-labs/dbt-s…
McKnight-42 2414226
add clean_up method to drop alt schema names after tests run
McKnight-42 157fe40
Merge branch 'main' into feature/dbt-clone-sf
McKnight-42 f8fef89
Merge branch 'main' of gist.github.com:dbt-labs/dbt-snowflake into fe…
McKnight-42 24b8e31
change pointer back to main
McKnight-42 dd54c4c
Merge branch 'feature/dbt-clone-sf' of gist.github.com:dbt-labs/dbt-s…
McKnight-42 18c7f49
Merge branch 'main' into feature/dbt-clone-sf
aranke 5197f44
Merge branch 'main' into feature/dbt-clone-sf
McKnight-42 38d60fc
Merge branch 'main' into feature/dbt-clone-sf
McKnight-42 a121695
clone transient table test
McKnight-42 File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,6 @@ | ||
kind: Features | ||
body: add dbt-snowflake portion of dbt_clone functionality | ||
time: 2023-06-22T11:46:21.399594-05:00 | ||
custom: | ||
Author: McKnight-42 aranke | ||
Issue: "7256" |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,11 @@ | ||
{% macro snowflake__can_clone_table() %} | ||
{{ return(True) }} | ||
{% endmacro %} | ||
|
||
{% macro snowflake__create_or_replace_clone(this_relation, defer_relation) %} | ||
create or replace | ||
{{ "transient" if config.get("transient", true) }} | ||
table {{ this_relation }} | ||
clone {{ defer_relation }} | ||
{{ "copy grants" if config.get("copy_grants", false) }} | ||
{% endmacro %} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,83 @@ | ||
import pytest | ||
import shutil | ||
import os | ||
from copy import deepcopy | ||
from dbt.tests.util import run_dbt | ||
from dbt.tests.adapter.dbt_clone.test_dbt_clone import BaseClonePossible | ||
|
||
|
||
class TestSnowflakeClonePossible(BaseClonePossible): | ||
@pytest.fixture(autouse=True) | ||
def clean_up(self, project): | ||
yield | ||
with project.adapter.connection_named("__test"): | ||
relation = project.adapter.Relation.create( | ||
database=project.database, schema=f"{project.test_schema}_SEEDS" | ||
) | ||
project.adapter.drop_schema(relation) | ||
|
||
relation = project.adapter.Relation.create( | ||
database=project.database, schema=project.test_schema | ||
) | ||
project.adapter.drop_schema(relation) | ||
|
||
pass | ||
|
||
|
||
table_model_1_sql = """ | ||
{{ config( | ||
materialized='table', | ||
transient=true, | ||
) }} | ||
|
||
select 1 as fun | ||
""" | ||
|
||
|
||
class TestSnowflakeCloneTrainsentTable: | ||
@pytest.fixture(scope="class") | ||
def models(self): | ||
return { | ||
"table_model.sql": table_model_1_sql, | ||
} | ||
|
||
@pytest.fixture(scope="class") | ||
def other_schema(self, unique_schema): | ||
return unique_schema + "_other" | ||
|
||
@pytest.fixture(scope="class") | ||
def profiles_config_update(self, dbt_profile_target, unique_schema, other_schema): | ||
outputs = {"default": dbt_profile_target, "otherschema": deepcopy(dbt_profile_target)} | ||
outputs["default"]["schema"] = unique_schema | ||
outputs["otherschema"]["schema"] = other_schema | ||
return {"test": {"outputs": outputs, "target": "default"}} | ||
|
||
def copy_state(self, project_root): | ||
state_path = os.path.join(project_root, "state") | ||
if not os.path.exists(state_path): | ||
os.makedirs(state_path) | ||
shutil.copyfile( | ||
f"{project_root}/target/manifest.json", f"{project_root}/state/manifest.json" | ||
) | ||
|
||
def run_and_save_state(self, project_root, with_snapshot=False): | ||
results = run_dbt(["run"]) | ||
assert len(results) == 1 | ||
assert not any(r.node.deferred for r in results) | ||
|
||
self.copy_state(project_root) | ||
|
||
def test_can_clone_transient_table(self, project, other_schema): | ||
project.create_test_schema(other_schema) | ||
self.run_and_save_state(project.project_root) | ||
|
||
clone_args = [ | ||
"clone", | ||
"--state", | ||
"state", | ||
"--target", | ||
"otherschema", | ||
] | ||
|
||
results = run_dbt(clone_args) | ||
assert len(results) == 1 |
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
do we need to test
transient
andcopy_grants
?There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
this admittedly is more pulled from Jeremey's draft pr where he created a snowflake implementation where he does talk about the use case for
transient
https://github.com/dbt-labs/dbt-core/pull/7258/files#diff-073e6ed96ac92033f0b921e061b47226b87d4f358350a1ed94fc94165f247b7aR39while in the docs does mention
copy grants