[Epic] Snapshots as a First Class Citizen

[gshank]

An epic to improve the usability and accuracy of snapshots.

P0 - must

Beta Give feedback

1. Snapshot config doesn't work in schema.yml #4000
   snapshots user docs +2
   
2. Remove IntermediateSnapshotNode class #10326
   snapshots +1
3. [Investigation] Explore options for providing upgrade or migration paths (for snapshots) #10325
   snapshots +1
   
4. [Feature] Snapshots should respect generate schema/database name macro #10301
   enhancement user docs +2
   
5. Warn if updated_at field for snapshot is not same datatype as what's returned in snapshot_get_time() #10234
   snapshots user docs +2
   
6. Allow customizing names of metadata fields in snapshots #10185
   snapshots user docs +2
   
7. Allow custom date for dbt_valid_to in snapshots #10187
   snapshots user docs +2
   
8. new record in snapshot to track deleted state (is_deleted column, zombie records) #10235
   snapshots user docs +2
9. [CT-2701] [Bug] Snapshot dbt_updated_at not updated #7869
   snapshots user docs +2
10. [Feature] No more jinja block for snapshots - new snapshot design ideas #10246
    enhancement snapshots user docs +3
    

P1 - should

Beta Give feedback

1. [Feature] Allows snapshots to have a list as a unique key #9992
   enhancement snapshots +2
2. [Feature] option to generate dbt_scd_id as an integer column instead of a string for performance improvements #10300
   enhancement snapshots +2
3. New snapshot config to validate uniqueness before merge #10236
   snapshots user docs +2
4. [CT-2869] Add new snapshot records when source removes a field #8207
   Refinement enhancement snapshots +3

p2 - could

Beta Give feedback

1. [Feature] exclude option for check_cols snapshot #10438
   enhancement snapshots +2
2. [CT-2280] Tracking changes in the non-chronological insertions of data in dbt snapshots (check strategy) #7138
   enhancement snapshots +2

revisit, refine, prioritize

Beta Give feedback

1. [CT-2699] [Feature] "interactive" compile should include the compiled code of a snapshot #7867
   enhancement help_wanted snapshots +3
2. Incorrect identifiers in snapshot schema test when overriding the ref() macro #2848
   bug +1
3. [CT-137] [Bug] Snapshot job creates duplicate valid records for unique_key #4661
   bug snapshots +2
4. dbt snapshots to handle append tables #3878
   enhancement snapshots stale +3
5. [CT-2492] [Bug] Update snapshot logic to use dbt.hash instead of hardcoded md5 dbt-adapters#82
   Stale enhancement snapshots +3
6. create a snapshot on sources that already have some history #10320
   snapshots +1
7. [Feature] Support --empty for snapshots #10372
   enhancement snapshots unit tests +3
8. [CT-3252] [Feature] Add a way to refresh Snapshots in the database #8885
   enhancement snapshots wontfix +3

Discussions

Problems & (Potential) Solutions for Snapshots
Toggling values when snapshotting a table with a key violation

[danielefrigo]

another interesting customization could be the ability to configure the value for dbt_valid_to then opening a new record.
dbt default is null, but in many cases I found the need to have a far away timestamp (e.g. '2999-12-31').

[graciegoheen]

@danielefrigo Yes! This is something we'd like to do - issue here -> #10187

[jeremyyeo]

Snapshots currently don't support contracts. Would be great have snapshots support contracts/constraints just as models do.