Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Expanded deprecation_date docs: motivation, examples, tie-in to model governance #3745

Closed
1 task done
Tracked by #3362
jtcohen6 opened this issue Jul 14, 2023 · 0 comments · Fixed by #3754
Closed
1 task done
Tracked by #3362

Expanded deprecation_date docs: motivation, examples, tie-in to model governance #3745

jtcohen6 opened this issue Jul 14, 2023 · 0 comments · Fixed by #3754
Assignees
@dbeatty10
Labels
content Improvements or additions to content dbt-core v1.6 Docs impact for the v1.6 release (July 2023) improvement Use this when an area of the docs needs improvement as it's currently unclear
Milestone
dbt Core v1.6 Doc...

Comments

@jtcohen6
Copy link
Collaborator

jtcohen6 commented Jul 14, 2023 •
edited
Loading

Contributions

  • I have read the contribution docs, and understand what's expected of me.

Link to the page on docs.getdbt.com requiring updates

https://docs.getdbt.com/reference/resource-properties/deprecation_date

  • Link back to page providing thorough overview of model versions
  • What does "YAML DateTime format" mean? What are the viable options?
  • Add an example, including

https://docs.getdbt.com/docs/collaborate/govern/model-versions

These statements need updating, and should link back to the reference docs on deprecation_date:

In the future, dbt will also offer first-class support for deprecating models (dbt-labs/dbt-core#7433).

dbt will notify consumers of your versioned model when new versions become available, or (in the future) when they are slated for deprecation

Because dbt knows that these models are actually the same model, it can notify downstream consumers as new versions become available, and (in the future) as older versions are slated for deprecation.

What part(s) of the page would you like to see updated?

#3579 added the stub for the deprecation_date page, but we're missing a lot of important context & detail:

  • why would I set a deprecation_date for my models?
  • what happens after a model is deprecated? does it stop building/being selected? what are the warning/error messages I could expect to see, for myself and for downstream queriers?
  • how does this relate to model versions? do I have to be using model contracts/versions/etc to set a deprecation_date? (no!)
  • how does this relate to table expiration on BQ? (they're different things, but maybe you'd want to use them in synchrony)
  • is there specific selection syntax I can use, to select only deprecated models? (I don't think we've implemented this)
  • how can I clean up / remove tables from the DWH associated with deprecated models? (nothing built-in/automated, but is there a discourse/pattern we could point people toward?)

Additional information
@jtcohen6 jtcohen6 added content Improvements or additions to content improvement Use this when an area of the docs needs improvement as it's currently unclear idea Proposes an idea for new content dbt-core v1.6 Docs impact for the v1.6 release (July 2023) labels Jul 14, 2023
@jtcohen6 jtcohen6 added this to the dbt Core v1.6 Documentation milestone Jul 14, 2023
@jtcohen6 jtcohen6 mentioned this issue Jul 14, 2023
Closed
@dbeatty10 dbeatty10 self-assigned this Jul 14, 2023
@dbeatty10 dbeatty10 mentioned this issue Jul 15, 2023
Merged
1 task
@MichelleArk MichelleArk mentioned this issue Jul 18, 2023
Merged
4 tasks
dbeatty10 added a commit that referenced this issue Jul 18, 2023
@dbeatty10
Expanded docs for deprecation_date (#3754)
19db6f3 
## What are you changing in this pull request and why?

Resolves #3745

- Updated docs for [model
versions](https://docs.getdbt.com/docs/collaborate/govern/model-versions)
including link back to the reference docs on `deprecation_date`
- Add examples with different accepted formats, including an aware
datetime, a naive datetime, and a plain date
   - `1999-01-01 00:00:00.00+00:00`
   - `2023-12-31 23:59:59.99`
   - `2024-01-01`

## Previews
-
[deprecation_date](https://deploy-preview-3754--docs-getdbt-com.netlify.app/reference/resource-properties/deprecation_date)
-
[model-versions](https://deploy-preview-3754--docs-getdbt-com.netlify.app/docs/collaborate/govern/model-versions)

## 🎩 

<img width="651" alt="image"
src="https://github.com/dbt-labs/docs.getdbt.com/assets/44704949/17a3db91-5842-4611-9f66-60ac5c0605b2">

## Key questions to answer

### Why would I set a `deprecation_date` for my models?

Declaring a deprecation date is a way of signaling the maturity level of
a model, and communicating plans for long-term support and maintenance.
It provides a mechanism to communicate plans and timelines for
sunsetting models so they don't need to be maintained and supported
indefinitely and their build and storage costs can be freed up.

### What happens after a model is deprecated?

Does it stop building/being selected? What are the warning/error
messages I could expect to see, for myself and for downstream queriers?

Deprecated models can still continue to be built by producers and be
selected by consumers until they are
[disabled](https://docs.getdbt.com/reference/resource-configs/enabled)
or removed.

Models marked for deprecation will raise informative warnings when they
are parsed or referenced. (With cross-project references, dbt raises
these warnings in downstream projects that are ref'ing a deprecated
version of a public model from an upstream project.)

When a project references a model that's slated for deprecation, a
warning is generated. If it's a versioned model, with a newer version
available, then the warning says so. This added bit of communication,
from producers to consumers, is an advantage of using dbt's built-in
functionality around model versions to facilitate migrations.

Additionally,
[`WARN_ERROR_OPTIONS`](https://docs.getdbt.com/reference/global-configs/warnings)
gives a mechanism whereby users can promote these warnings to actual
runtime errors.

**Warning messages:**
* DeprecatedModel (warning when parsing a project that defines
deprecated model(s)): `[WARNING] Model {model_name}[.v{version} if
versioned] has passed its deprecation date of {deprecation_date}. This
model should be disabled or removed.`
* UpcomingDeprecationReference (warning when referencing a model with a
future deprecation date): `[WARNING] While compiling
'{this_model_name}': Found a reference to {ref_model_name}[.v{version}
if versioned], which is slated for deprecation on
'{ref_model_deprecation_date}'. [if versioned: A new version of
'{ref_model_name}' is available. Try it out: {{ ref('ref_model_package',
'ref_model_name', v='{ref_model_latest_version}') }}.`
* DeprecatedReference (warning when referencing a model with a past
deprecation date): `[WARNING] While compiling '{this_model_name}': Found
a reference to {ref_model_name}[.v{version} if versioned], which was
deprecated on '{ref_model_deprecation_date}'. [if versioned: A new
version of '{ref_model_name}' is available. Migrate now: {{
ref('ref_model_package', 'ref_model_name',
v='{ref_model_latest_version}') }}.`

### How does this relate to model versions?

Do I have to be using model contracts/versions/etc to set a
`deprecation_date`?

(no!)

Setting a `deprecation_date` works well in conjunction with other [model
governance](https://docs.getdbt.com/docs/collaborate/govern/about-model-governance)
features like [model
versions](https://docs.getdbt.com/docs/collaborate/govern/model-versions),
but can also be used independently from them.

### Is there specific selection syntax I can use, to select only
deprecated models?

(I don't think we've implemented this)

There is not specific [node selection
syntax](https://docs.getdbt.com/reference/node-selection/syntax) for
`deprecation_date`. [Programmatic
invocations](https://docs.getdbt.com/reference/programmatic-invocations)
is one way to identify deprecated models (potentially in conjunction
with [dbt list](https://docs.getdbt.com/reference/commands/list)). e.g.,
`dbt -q ls --output json --output-keys database schema alias
deprecation_date`.

### How can I clean up / remove tables from the DWH associated with
deprecated models?

(nothing built-in/automated, but is there a discourse/pattern we could
point people toward?)

Just like it [does not automatically drop relations when models are
deleted](https://docs.getdbt.com/faqs/models/removing-deleted-models),
dbt does not removed tables for deprecated models.

Strategies similar to
[here](https://discourse.getdbt.com/t/faq-cleaning-up-removed-models-from-your-production-schema/113)
or
[here](https://discourse.getdbt.com/t/clean-your-warehouse-of-old-and-deprecated-models/1547)
can be used to drop relations that have been deprecated and are no
longer in use.

### How does this relate to [table expiration on
BQ](https://docs.getdbt.com/reference/resource-configs/bigquery-configs#controlling-table-expiration)?

(they're different things, but maybe you'd want to use them in
synchrony)

dbt-bigquery can set an
[`hours_to_expiration`](https://docs.getdbt.com/reference/resource-configs/bigquery-configs#controlling-table-expiration)
that [translates to
`expiration_timestamp`](https://github.com/dbt-labs/dbt-bigquery/blob/ea258bb76169375ded8f7ff9e596a436a5ed165a/dbt/adapters/bigquery/impl.py#L844-L846)
(expired tables in BigQuery will be deleted and their storage
reclaimed).

dbt does not automatically synchronize `deprecation_date` and
`hours_to_expiration`, but users may want to coordinate them in some
fashion (such as setting a model to expire 48 hours after its
`deprecation_date`).

## Checklist
- [x] Review the [Content style
guide](https://github.com/dbt-labs/docs.getdbt.com/blob/current/contributing/content-style-guide.md)
and [About
versioning](https://github.com/dbt-labs/docs.getdbt.com/blob/current/contributing/single-sourcing-content.md#adding-a-new-version)
so my content adheres to these guidelines.
@runleonarun runleonarun removed the idea Proposes an idea for new content label Mar 13, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@dbeatty10 dbeatty10

Labels
content Improvements or additions to content dbt-core v1.6 Docs impact for the v1.6 release (July 2023) improvement Use this when an area of the docs needs improvement as it's currently unclear
Projects
None yet
Milestone
dbt Core v1.6 Documentation
Development

Successfully merging a pull request may close this issue.

Expanded docs for deprecation_date dbt-labs/docs.getdbt.com
3 participants
@runleonarun @jtcohen6 @dbeatty10