Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add callout for net new snapshots #6521

Merged
merged 43 commits into from
Dec 4, 2024
Merged
Show file tree
Hide file tree
Changes from 26 commits
Commits
Show all changes
43 commits
Select commit Hold shift + click to select a range
befa7b5
add callout for net new
mirnawong1 Nov 21, 2024
0697083
Merge branch 'current' into net-new-snapshots
mirnawong1 Nov 25, 2024
0ff5771
Merge branch 'current' into net-new-snapshots
mirnawong1 Nov 25, 2024
4127104
Merge branch 'current' into net-new-snapshots
mirnawong1 Nov 25, 2024
5a736c5
Merge branch 'current' into net-new-snapshots
mirnawong1 Nov 25, 2024
6746196
Merge branch 'current' into net-new-snapshots
mirnawong1 Nov 25, 2024
64d56e2
Merge branch 'current' into net-new-snapshots
mirnawong1 Nov 25, 2024
0347fd3
turn to snippet
mirnawong1 Nov 25, 2024
576af99
Update website/docs/reference/snapshot-configs.md
mirnawong1 Nov 25, 2024
2532cad
update grace feedback
mirnawong1 Nov 25, 2024
472d5c0
update'
mirnawong1 Nov 25, 2024
eee70d8
Merge branch 'current' into net-new-snapshots
mirnawong1 Nov 26, 2024
02fd890
Merge branch 'current' into net-new-snapshots
mirnawong1 Nov 26, 2024
1ca7c4e
Merge branch 'current' into net-new-snapshots
mirnawong1 Nov 26, 2024
43386c6
Merge branch 'current' into net-new-snapshots
mirnawong1 Nov 26, 2024
832ef12
Merge branch 'current' into net-new-snapshots
mirnawong1 Nov 26, 2024
91c3bb9
Merge branch 'current' into net-new-snapshots
mirnawong1 Nov 27, 2024
14690e0
Merge branch 'current' into net-new-snapshots
mirnawong1 Nov 27, 2024
7e68e91
update to snapshot configs
mirnawong1 Nov 27, 2024
6d2a38d
Merge branch 'current' into net-new-snapshots
mirnawong1 Nov 27, 2024
38762c1
Update website/docs/reference/snapshot-configs.md
mirnawong1 Nov 27, 2024
bdf2850
Update website/docs/reference/snapshot-configs.md
mirnawong1 Nov 27, 2024
6a84a7f
Update website/docs/reference/snapshot-configs.md
mirnawong1 Nov 27, 2024
37efd7a
Update website/snippets/_snapshots-yaml-config.md
mirnawong1 Nov 27, 2024
7eb3bf1
Merge branch 'current' into net-new-snapshots
mirnawong1 Nov 27, 2024
2fba6cc
Merge branch 'current' into net-new-snapshots
mirnawong1 Nov 27, 2024
ad3b3bd
Merge branch 'current' into net-new-snapshots
mirnawong1 Nov 28, 2024
5d3b48f
update
mirnawong1 Nov 28, 2024
b84f4a5
remove iimport
mirnawong1 Nov 28, 2024
915e396
Update snapshot-configs.md
mirnawong1 Nov 28, 2024
61d3792
Update snapshot-configs.md
mirnawong1 Nov 28, 2024
079378f
Merge branch 'current' into net-new-snapshots
mirnawong1 Nov 29, 2024
4c2708d
Merge branch 'current' into net-new-snapshots
mirnawong1 Nov 29, 2024
5732ff5
Merge branch 'current' into net-new-snapshots
mirnawong1 Nov 29, 2024
a8760c4
Merge branch 'current' into net-new-snapshots
mirnawong1 Nov 29, 2024
7dd144d
Merge branch 'current' into net-new-snapshots
mirnawong1 Dec 3, 2024
c491818
add link
mirnawong1 Dec 3, 2024
8382836
Update website/docs/reference/snapshot-configs.md
mirnawong1 Dec 3, 2024
a9d4cc5
Update website/docs/reference/snapshot-configs.md
mirnawong1 Dec 3, 2024
944fb50
doug's feedback
mirnawong1 Dec 3, 2024
c19c1d8
empty
mirnawong1 Dec 3, 2024
839c99d
Merge branch 'current' into net-new-snapshots
mirnawong1 Dec 4, 2024
a475180
tweaks
mirnawong1 Dec 4, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
35 changes: 26 additions & 9 deletions website/docs/reference/snapshot-configs.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,21 +7,18 @@

import ConfigResource from '/snippets/_config-description-resource.md';
import ConfigGeneral from '/snippets/_config-description-general.md';
import SnapshotsYamlConfig from '/snippets/_snapshots-yaml-config.md';


## Related documentation
* [Snapshots](/docs/build/snapshots)
* The `dbt snapshot` [command](/reference/commands/snapshot)

<!--
Parts of a snapshot:
- name
- query
-->

## Available configurations
### Snapshot-specific configurations


<ConfigResource meta={frontMatter.meta} />

<VersionBlock lastVersion="1.8">
Expand Down Expand Up @@ -78,8 +75,9 @@
[+](/reference/resource-configs/plus-prefix)[strategy](/reference/resource-configs/strategy): timestamp | check
[+](/reference/resource-configs/plus-prefix)[updated_at](/reference/resource-configs/updated_at): <column_name>
[+](/reference/resource-configs/plus-prefix)[check_cols](/reference/resource-configs/check_cols): [<column_name>] | all
[+](/reference/resource-configs/plus-prefix)[invalidate_hard_deletes](/reference/resource-configs/invalidate_hard_deletes) : true | false
[+](/reference/resource-configs/plus-prefix)[snapshot_meta_column_names](/reference/resource-configs/snapshot_meta_column_names): {<dictionary>}
[+](/reference/resource-configs/plus-prefix)[invalidate_hard_deletes](/reference/resource-configs/invalidate_hard_deletes) : true | false
[+](/reference/resource-configs/plus-prefix)[dbt_valid_to_current](/reference/resource-configs/dbt_valid_to_current): <string>
mirnawong1 marked this conversation as resolved.
Show resolved Hide resolved
```

</File>
Expand Down Expand Up @@ -112,8 +110,9 @@
[strategy](/reference/resource-configs/strategy): timestamp | check
[updated_at](/reference/resource-configs/updated_at): <column_name>
[check_cols](/reference/resource-configs/check_cols): [<column_name>] | all
[snapshot_meta_column_names](/reference/resource-configs/snapshot_meta_column_names): {<dictionary>}
[invalidate_hard_deletes](/reference/resource-configs/invalidate_hard_deletes) : true | false
[snapshot_meta_column_names](/reference/resource-configs/snapshot_meta_column_names): {<dictionary>}
[dbt_valid_to_current](/reference/resource-configs/dbt_valid_to_current): <string>
```
</File>

Expand All @@ -125,7 +124,8 @@

<VersionBlock firstVersion="1.9">

Configurations can be applied to snapshots using the [YAML syntax](/docs/build/snapshots), available in Versionless and dbt v1.9 and higher, in the `snapshot` directory file.
<SnapshotsYamlConfig />


</VersionBlock>

Expand All @@ -150,11 +150,27 @@

</Tabs>

### Snapshot configuration migration

The latest snapshot YAML configurations introduced in dbt v1.9 (such as [`snapshot_meta_column_names`](/reference/resource-configs/snapshot_meta_column_names) and [`dbt_valid_to_current`](/reference/resource-configs/dbt_valid_to_current)) are best suited for new snapshots. For existing snapshots, we recommend the following to avoid any inconsistencies in your snapshots:

Check warning on line 155 in website/docs/reference/snapshot-configs.md

View workflow job for this annotation

GitHub Actions / vale

[vale] website/docs/reference/snapshot-configs.md#L155

[custom.Typos] Oops there's a typo -- did you really mean 'v1.9'?
Raw output
{"message": "[custom.Typos] Oops there's a typo -- did you really mean 'v1.9'? ", "location": {"path": "website/docs/reference/snapshot-configs.md", "range": {"start": {"line": 155, "column": 59}}}, "severity": "WARNING"}
mirnawong1 marked this conversation as resolved.
Show resolved Hide resolved
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
The latest snapshot YAML configurations introduced in dbt v1.9 (such as [`snapshot_meta_column_names`](/reference/resource-configs/snapshot_meta_column_names) and [`dbt_valid_to_current`](/reference/resource-configs/dbt_valid_to_current)) are best suited for new snapshots. For existing snapshots, we recommend the following to avoid any inconsistencies in your snapshots:
The latest snapshot configurations introduced in dbt v1.9 (such as [`snapshot_meta_column_names`](/reference/resource-configs/snapshot_meta_column_names) and [`dbt_valid_to_current`](/reference/resource-configs/dbt_valid_to_current)) are best suited for new snapshots. For existing snapshots, we recommend the following to avoid any inconsistencies in your snapshots:


#### For new snapshots
- Use the latest snapshot YAML configurations when creating new snapshots that didn't exist previously.
mirnawong1 marked this conversation as resolved.
Show resolved Hide resolved

#### For existing snapshots
- Migrate tables &mdash; Migrate the previous snapshot to the new table schema and values:
- Create a backup copy of your snapshots.
- Use `alter` statements as needed (or a script to apply `alter` statements) to ensure table consistency.
- New YAML config &mdash; Convert the YAML configurations one at a time, testing as you go.
mirnawong1 marked this conversation as resolved.
Show resolved Hide resolved

If you use one of the latest configs, such as `dbt_valid_to_current`, without migrating your data, you may have mixed old and new data, leading to an incorrect downstream result.
mirnawong1 marked this conversation as resolved.
Show resolved Hide resolved


### General configurations

<ConfigGeneral />


<Tabs
groupId="config-languages"
defaultValue="project-yaml"
Expand All @@ -170,6 +186,7 @@

<VersionBlock firstVersion="1.9">


```yaml
snapshots:
[<resource-path>](/reference/resource-configs/resource-path):
Expand Down Expand Up @@ -256,7 +273,7 @@

<VersionBlock firstVersion="1.9">

Configurations can be applied to snapshots using [YAML syntax](/docs/build/snapshots), available in Versionless and dbt v1.9 and higher, in the `snapshot` directory file.
<SnapshotsYamlConfig />

</VersionBlock>

Expand Down
1 change: 1 addition & 0 deletions website/snippets/_snapshots-yaml-config.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Define snapshots using the latest [YAML configurations](/docs/build/snapshots#configuring-snapshots), available in Versionless dbt Cloud and Core v1.9 and higher, in the `snapshot` directory file.

Check warning on line 1 in website/snippets/_snapshots-yaml-config.md

View workflow job for this annotation

GitHub Actions / vale

[vale] website/snippets/_snapshots-yaml-config.md#L1

[custom.Typos] Oops there's a typo -- did you really mean 'v1.9'?
Raw output
{"message": "[custom.Typos] Oops there's a typo -- did you really mean 'v1.9'? ", "location": {"path": "website/snippets/_snapshots-yaml-config.md", "range": {"start": {"line": 1, "column": 147}}}, "severity": "WARNING"}
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is this about the YAML syntax?

I think we're getting two separate things muddied:

  • starting in 1.9 you can define snapshots in yaml
  • starting in 1.9 there are a handful of new configs that you can use, we recommend only using those configs on new snapshots or if you use them on old snapshots you will need to migrate

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ah ok thank you for clarifying! resovled this by having the below for snapshot specific configs and general configs:

snapshot specific configs

Starting from Versionless and Core v1.9, defining snapshots in a .sql file using a config block is a legacy method. You can define snapshots in YAML format using the latest snapshot-specific configurations. For new snapshots, we recommend using these latest configs. If applying them to existing snapshots, you'll need to migrate over.

general configs

Starting from Versionless and Core v1.9, defining snapshots in a .sql file using a config block is a legacy method. You can define snapshots in YAML format using the latest configurations. You can also continue using the legacy method for existing snapshots, but we recommend migrating over to the latest YAML format for better readability and maintainability.

3 changes: 2 additions & 1 deletion website/src/components/expandable/styles.module.css
Original file line number Diff line number Diff line change
Expand Up @@ -145,4 +145,5 @@
.headerText {
display: flex;
align-items: center;
}
}

Loading