dbt clone command #3742

graciegoheen · 2023-07-14T00:06:48Z

What are you changing in this pull request and why?

Add docs for new dbt clone command

Closes #3607

Outstanding:

add more of the narrative around "why"/ use cases / places where cloud shines
add versioning components to the project-state page
add to https://docs.getdbt.com/docs/deploy/slim-ci-jobs page
ensure i've created links correctly

Previews

Checklist

Add versioning components, as described in Versioning Docs
Review the Content style guide and About versioning so my content adheres to these guidelines.
Add a checklist item for anything that needs to happen before this PR is merged, such as "needs technical review" or "change base branch."
Add page to website/sidebars.js
Provide a unique filename for the new page

netlify · 2023-07-14T00:07:01Z

✅ Deploy Preview for docs-getdbt-com ready!

Name	Link
🔨 Latest commit	`bd77486`
🔍 Latest deploy log	https://app.netlify.com/sites/docs-getdbt-com/deploys/64b71496a3e9630008057a74
😎 Deploy Preview	https://deploy-preview-3742--docs-getdbt-com.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site configuration.

runleonarun

Hey @graciegoheen just two tiny link fixes to fix your staging preview. We use the document root which starts with /docs or /reference or /guides.

website/docs/reference/commands/clone.md

website/docs/reference/node-selection/syntax.md

jtcohen6

nice solid start!!

I'd also like to see something in the docs on deferral: when/why would I use deferral versus cloning? Where are there two separate approaches to achieving the same end? (see internal notion thread)

website/docs/reference/node-selection/syntax.md

website/docs/reference/commands/clone.md

Co-authored-by: Leona B. Campbell <3880403+runleonarun@users.noreply.github.com>

Co-authored-by: Jeremy Cohen <jeremy@dbtlabs.com>

aranke

Great start, thanks for taking this on @graciegoheen!

website/docs/reference/node-selection/syntax.md

website/docs/reference/commands/clone.md

aranke · 2023-07-14T15:01:46Z

website/docs/reference/commands/clone.md

+- Otherwise, dbt will create a simple pointer view (`select * from` the other object)
+
+Note: 
+- The state to clone from is based on the location of nodes in the manifest provided to `--state`.


Something else to note here is that you cannot use the same target directory for both the source and target due to <placeholder>.

Will it literally not allow the target and source to be the same? Or would it yield surprising results?

How is something like this typically called out in the docs @runleonarun

@graciegoheen I would add this tidbit as a sentence in the very first paragraph. Maybe by rephrasing as what you should do instead of what you shouldn't, for example:

The dbt clone command clones selected nodes from the specified state to the target schema(s). You must always use different directories for source and target otherwise you will get an error message/overwrite the source. This command makes use of the clone materialization:

(hopefully you just get an error @aranke??)

You get a very weird error that source and target states are the same, and so it's a no-op.

I ran into this during local development, so think this warning needs to be surfaced loudly.

Sounds like we want a dbt-core follow-on issue?

^ yes and add callout to the docs? or yes then we don't need this callout?

I'd say "yes, and" — then when the issue is closed, we can remove the callout from the docs

docs are updated - @dbeatty10 or @aranke are one of y'all able to open up a core issue to improve the error message?

@graciegoheen I just now opened a minimal issue here: dbt-labs/dbt-core#8160

website/docs/reference/commands/clone.md

… with `dbt clone`

website/docs/reference/commands/clone.md

graciegoheen · 2023-07-18T00:54:32Z

nice solid start!!

I'd also like to see something in the docs on deferral: when/why would I use deferral versus cloning? Where are there two separate approaches to achieving the same end? (see internal notion thread)

@jtcohen6 were you imagining this callout be added to the defer docs, or to the clone docs, or to both? or a new page altogether?

jtcohen6 · 2023-07-18T10:14:20Z

@jtcohen6 were you imagining this callout be added to the defer docs, or to the clone docs, or to both? or a new page altogether?

How about this:

A sentence addition to the "defer" page, stating that cloning is an alternative to deferral and links to the "clone" command page
A slightly longer addition (paragraph) on the "clone" command page: "When would I use cloning instead of deferral?" Deferral is the feature that's more established (been around for longer), and at first glance it offers a strictly better/cheaper alternative to clone (which requires some compute + creation of more DWH objects), but has its own advantages worth explaining/considering

mirnawong1 · 2023-07-18T16:29:11Z

website/docs/reference/commands/clone.md

+- You may want to specify a higher number of [threads](/docs/running-a-dbt-project/using-threads) to decrease execution time since individual clone statements are independent of one another.
+
+The `clone` command is useful for:
+- blue/green continuous deployment


would it be helpful to link to this so there's context? is that the right link?

yes, good call! that's a fairly old post at this point, but until we have a newer one, it's good to provide some definition/context

How do we feel about this being a snowflake-specific article? I think blue/green continuous blue/green deployment will be different depending on the warehouse being used (zero copy cloning vs. pointer views)

Even better point. Blue/green is really only enabled by table clones. There's also ~~hackery~~ cleverness in that post around overriding ref so that when the schemas get swapped, the views stay pointed to the same database.

Maybe better to skip it for now, and just say something like:

blue/green continuous deployment (on data warehouses that support cloning tables)

Or remove the bullet entirely, until we have a more up-to-date and thorough dev blog article that we could link to

Just updated - let me know what you think

aranke · 2023-07-18T18:37:46Z

@jtcohen6 were you imagining this callout be added to the defer docs, or to the clone docs, or to both? or a new page altogether?

To-do for @dbeatty10 and I: link to blog post highlighting differences when that's ready

jtcohen6

One small comment (I changed my mind about the source/target mismatch error) -- then this lgtm!

website/docs/reference/commands/clone.md

jtcohen6 · 2023-07-18T21:19:58Z

website/docs/reference/commands/clone.md

+For example, by creating actual data warehouse objects, `dbt clone` allows you to test out your code changes on downstream dependencies _outside of dbt_ (such as a BI tool). 
+
+As another example, you could `clone` your modified incremental models as the first step of your dbt Cloud CI job to prevent costly `full-refresh` builds for warehouses that support zero-copy cloning.


… into dbt-clone

graciegoheen added 3 commits July 13, 2023 14:52

add initial clone updates

f2c0b62

added initial clone command page

eb71a3b

update project state page

db5b2cc

graciegoheen requested review from dbeatty10 and aranke July 14, 2023 00:06

github-actions bot added content Improvements or additions to content size: small This change will take 1 to 2 days to address labels Jul 14, 2023

Merge branch 'current' into dbt-clone

e575d98

graciegoheen requested a review from jtcohen6 July 14, 2023 00:10

runleonarun reviewed Jul 14, 2023

View reviewed changes

website/docs/reference/commands/clone.md Outdated Show resolved Hide resolved

website/docs/reference/node-selection/syntax.md Outdated Show resolved Hide resolved

jtcohen6 reviewed Jul 14, 2023

View reviewed changes

dbeatty10 and others added 3 commits July 14, 2023 08:44

Fix link

98ff0b0

Co-authored-by: Leona B. Campbell <3880403+runleonarun@users.noreply.github.com>

Update link

184a085

Co-authored-by: Leona B. Campbell <3880403+runleonarun@users.noreply.github.com>

Update explanations for zero-copy cloning (or not)

14d9c4c

Co-authored-by: Jeremy Cohen <jeremy@dbtlabs.com>

aranke reviewed Jul 14, 2023

View reviewed changes

dbeatty10 reviewed Jul 14, 2023

View reviewed changes

website/docs/reference/commands/clone.md Outdated Show resolved Hide resolved

dbeatty10 reviewed Jul 14, 2023

View reviewed changes

website/docs/reference/commands/clone.md Outdated Show resolved Hide resolved

dbeatty10 added 2 commits July 14, 2023 10:17

Link to docs for specifying / establishing state

b1f734f

Remove note in favor of link to docs for "establishing state"

3f46631

dbeatty10 reviewed Jul 17, 2023

View reviewed changes

website/docs/reference/commands/clone.md Outdated Show resolved Hide resolved

Incorporate feedback on phrasing for blue/green continuous deployment…

d9bcdb7

… with `dbt clone`

dbeatty10 reviewed Jul 17, 2023

View reviewed changes

website/docs/reference/commands/clone.md Outdated Show resolved Hide resolved

dbeatty10 and others added 2 commits July 17, 2023 09:59

Incorporate feedback for delineating source and target environments

a96502e

Merge branch 'current' into dbt-clone

6def43d

graciegoheen marked this pull request as ready for review July 18, 2023 00:51

graciegoheen requested a review from a team as a code owner July 18, 2023 00:51

jtcohen6 mentioned this pull request Jul 18, 2023

Revise v1.6 migration guide #3728

Merged

matthewshaver requested review from dbeatty10, runleonarun and aranke July 18, 2023 14:53

mirnawong1 reviewed Jul 18, 2023

View reviewed changes

add initial defer docs

30a856a

github-actions bot added size: medium This change will take up to a week to address and removed size: small This change will take 1 to 2 days to address labels Jul 18, 2023

graciegoheen and others added 3 commits July 18, 2023 13:30

add caveats for useful bullets

e1b9972

error note

1a087c7

Merge branch 'current' into dbt-clone

4a844c1

graciegoheen requested a review from jtcohen6 July 18, 2023 21:05

jtcohen6 approved these changes Jul 18, 2023

View reviewed changes

graciegoheen added 2 commits July 18, 2023 14:22

error message begone

c42a530

Merge branch 'dbt-clone' of https://github.com/dbt-labs/docs.getdbt.com…

b3002bf

… into dbt-clone

graciegoheen self-assigned this Jul 18, 2023

Merge branch 'current' into dbt-clone

bd77486

graciegoheen merged commit b6d1d47 into current Jul 18, 2023

graciegoheen deleted the dbt-clone branch July 18, 2023 22:46

dbeatty10 mentioned this pull request Jul 19, 2023

[CT-2850] [Feature] Raise a warning when --state path == --target-path dbt-labs/dbt-core#8160

Closed

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

dbt clone command #3742

dbt clone command #3742

graciegoheen commented Jul 14, 2023 •

edited by dbeatty10

Loading

netlify bot commented Jul 14, 2023 •

edited

Loading

runleonarun left a comment

jtcohen6 left a comment

aranke left a comment

aranke Jul 14, 2023 •

edited

Loading

dbeatty10 Jul 17, 2023

graciegoheen Jul 18, 2023

runleonarun Jul 18, 2023

aranke Jul 18, 2023 •

edited

Loading

jtcohen6 Jul 18, 2023

graciegoheen Jul 18, 2023

jtcohen6 Jul 18, 2023

graciegoheen Jul 18, 2023

dbeatty10 Jul 19, 2023

graciegoheen commented Jul 18, 2023 •

edited

Loading

jtcohen6 commented Jul 18, 2023 •

edited

Loading

mirnawong1 Jul 18, 2023

jtcohen6 Jul 18, 2023

graciegoheen Jul 18, 2023

jtcohen6 Jul 18, 2023

graciegoheen Jul 18, 2023

aranke commented Jul 18, 2023

jtcohen6 left a comment

jtcohen6 Jul 18, 2023

		For example, by creating actual data warehouse objects, `dbt clone` allows you to test out your code changes on downstream dependencies _outside of dbt_ (such as a BI tool).

		As another example, you could `clone` your modified incremental models as the first step of your dbt Cloud CI job to prevent costly `full-refresh` builds for warehouses that support zero-copy cloning.

dbt clone command #3742

dbt clone command #3742

Conversation

graciegoheen commented Jul 14, 2023 • edited by dbeatty10 Loading

What are you changing in this pull request and why?

Previews

Checklist

netlify bot commented Jul 14, 2023 • edited Loading

✅ Deploy Preview for docs-getdbt-com ready!

runleonarun left a comment

Choose a reason for hiding this comment

jtcohen6 left a comment

Choose a reason for hiding this comment

aranke left a comment

Choose a reason for hiding this comment

aranke Jul 14, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aranke Jul 18, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

graciegoheen commented Jul 18, 2023 • edited Loading

jtcohen6 commented Jul 18, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aranke commented Jul 18, 2023

jtcohen6 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

graciegoheen commented Jul 14, 2023 •

edited by dbeatty10

Loading

netlify bot commented Jul 14, 2023 •

edited

Loading

aranke Jul 14, 2023 •

edited

Loading

aranke Jul 18, 2023 •

edited

Loading

graciegoheen commented Jul 18, 2023 •

edited

Loading

jtcohen6 commented Jul 18, 2023 •

edited

Loading