ADAP-814: Add support for replacing materialized views with tables/views and vice versa #8449

mikealfare · 2023-08-17T22:14:29Z

Problem

Materialized view support for dbt-postgres does not allow for replacing a materialized view with a table/view and vice versa.

Solution

Make the drop and rename statements generic across relations
Dispatch to the relation-specific statement for drop/rename to allow for variations in the statement
Allow a table/view to trigger the drop/rename of a materialized view
Allow a materialized view to trigger the drop/rename of a table/view

Checklist

I have read the contributing guide and understand what's expected of me
I have run this code in development and it appears to resolve the stated issue
This PR includes tests, or tests are not required/relevant for this PR
This PR has no interface changes (e.g. macros, cli, logs, json artifacts, config files, adapter interface, etc) or this PR has already received feedback and approval from Product or DX

github-actions · 2023-08-17T22:14:47Z

Thank you for your pull request! We could not find a changelog entry for this change. For details on how to document a change, see the contributing guide.

codecov · 2023-08-17T22:18:19Z

Codecov Report

Patch coverage: 100.00% and no project coverage change.

Comparison is base (4ec87a0) 86.36% compared to head (8b643bd) 86.36%.
Report is 14 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #8449   +/-   ##
=======================================
  Coverage   86.36%   86.36%           
=======================================
  Files         174      174           
  Lines       25575    25582    +7     
=======================================
+ Hits        22087    22095    +8     
+ Misses       3488     3487    -1

Flag	Coverage Δ
integration	`83.16% <100.00%> (-0.04%)`	⬇️
unit	`65.04% <100.00%> (-0.12%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files Changed	Coverage Δ
core/dbt/adapters/base/relation.py	`93.75% <100.00%> (+0.11%)`	⬆️

... and 31 files with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

This reverts commit 46e8aef.

This reverts commit d898469.

This reverts commit d755c51.

This reverts commit 03a15e7.

* apply reformatting changes only for #8449 * add logging back to get_create_materialized_view_as_sql * changie

# Conflicts: # core/dbt/include/global_project/macros/relations/drop.sql # core/dbt/include/global_project/macros/relations/rename.sql # plugins/postgres/dbt/include/postgres/macros/relations/materialized_view/alter.sql

…relations` so that generics are close to the relation specific macros that they reference; also aligns with adapter macro files structure, to look more familiar

…r adapters that have a unique list of relation types, e.g. dbt-snowflake

…nd can be addressed properly at a later date

…default from drop_relation

…t from dbt-core

* apply reformatting changes only for #8449 * add logging back to get_create_materialized_view_as_sql * changie

* Add compiled node properties to run_results.json * Include compiled-node attributes in run_results.json * Fix typo * Bump schema version of run_results * Fix test assertions * Update expected run_results to reflect new attributes * Code review changes * Fix mypy warnings for ManifestLoader.load() (#8443) * revert python version for docker images (#8445) * revert python version for docker images * add comment to not update python version, update changelog * Bumping version to 1.7.0b1 and generate changelog * [CT-3013] Fix parsing of `window_groupings` (#8454) * Update semantic model parsing tests to check measure non_additive_dimension spec * Make `window_groupings` default to empty list if not specified on `non_additive_dimension` * Add changie doc for `window_groupings` parsing fix * update `Number` class to handle integer values (#8306) * add show test for json data * oh changie my changie * revert unecessary cahnge to fixture * keep decimal class for precision methods, but return __int__ value * jerco updates * update integer type * update other tests * Update .changes/unreleased/Fixes-20230803-093502.yaml --------- Co-authored-by: Emily Rockman <emily.rockman@dbtlabs.com> * Improve docker image README (#8212) * Improve docker image README - Fix unnecessary/missing newline escapes - Remove double whitespace between parameters - 2-space indent for extra lines in image build commands * Add changelog entry for #8212 * ADAP-814: Refactor prep for MV updates (#8459) * apply reformatting changes only for #8449 * add logging back to get_create_materialized_view_as_sql * changie * swap trigger (#8463) * update the implementation template (#8466) * update the implementation template * add colon * Split tests into classes (#8474) * add flaky decorator * split up tests into classes * revert update agate for int (#8478) * updated typing and methods to meet mypy standards (#8485) * Convert error to conditional warning for unversioned contracted model, fix msg format (#8451) * first pass, tests need updates * update proto defn * fixing tests * more test fixes * finish fixing test file * reformat the message * formatting messages * changelog * add event to unit test * feedback on message structure * WIP * fix up event to take in all fields * fix test * Fix ambiguous reference error for duplicate model names across packages with tests (#8488) * Safely remove external nodes from manifest (#8495) * [CT-2840] Improved semantic layer protocol satisfaction tests (#8456) * Test `SemanticModel` satisfies protocol when none of it's `Optionals` are specified * Add tests ensuring SourceFileMetadata and FileSlice satisfiy DSI protocols * Add test asserting Defaults obj satisfies protocol * Add test asserting SemanticModel with optionals specified satisfies protocol * Split dimension protocol satisfaction tests into with and without optionals * Simplify DSI Protocol import strategy in protocol satisfaction tests * Add test asserting DimensionValidtyParams satisfies protocol * Add test asserting DimensionTypeParams satisfies protocol * Split entity protocol satisfaction tests into with and without optionals * Split measure protocol satisfication tests and add measure aggregation params satisficaition test * Split metric protocol satisfaction test into optional specified an unspecified Additionally, create where_filter pytest fixture * Improve protocol satisfaction tests for MetricTypeParams and sub protocols Specifically we added/improved protocol satisfaction tests for - MetricTypeParams - MetricInput - MetricInputMeasure - MetricTimeWindow * Convert to using mashumaro jsonschema with acceptable performance (#8437) * Regenerate run_results schema after merging in changes from main. --------- Co-authored-by: Gerda Shank <gerda@dbtlabs.com> Co-authored-by: Matthew McKnight <91097623+McKnight-42@users.noreply.github.com> Co-authored-by: Github Build Bot <buildbot@fishtownanalytics.com> Co-authored-by: Quigley Malcolm <QMalcolm@users.noreply.github.com> Co-authored-by: dave-connors-3 <73915542+dave-connors-3@users.noreply.github.com> Co-authored-by: Emily Rockman <emily.rockman@dbtlabs.com> Co-authored-by: Jaime Martínez Rincón <jaime@jamezrin.name> Co-authored-by: Mike Alfare <13974384+mikealfare@users.noreply.github.com> Co-authored-by: Michelle Ark <MichelleArk@users.noreply.github.com>

core/dbt/include/global_project/macros/relations/replace.sql

core/dbt/adapters/base/relation.py

mikealfare added 3 commits August 17, 2023 18:02

first draft of adding in table - materialized view swap

f0b07d0

Merge branch 'main' into feature/materialized-views/adap-814

24d9c42

tests pass

7372440

mikealfare self-assigned this Aug 17, 2023

cla-bot bot added the cla:yes label Aug 17, 2023

table/view/materialized view can all replace each other

d20f7dc

mikealfare changed the title ~~ADAP-816: Add support for replacing materialized views with tables~~ ADAP-814: Add support for replacing materialized views with tables Aug 17, 2023

mikealfare added 2 commits August 17, 2023 18:57

changie

28e1612

update renameable relations to a config

4483f6c

mikealfare changed the title ~~ADAP-814: Add support for replacing materialized views with tables~~ ADAP-814: Add support for replacing materialized views with tables/views and vice versa Aug 17, 2023

revert to just reformatting changes

03a15e7

mikealfare mentioned this pull request Aug 18, 2023

ADAP-814: Refactor prep for MV updates #8459

Merged

4 tasks

mikealfare added 7 commits August 18, 2023 16:07

revert to just reformatting changes

d755c51

revert to just reformatting changes

d898469

revert to just reformatting changes

46e8aef

Revert "revert to just reformatting changes"

3572593

This reverts commit 46e8aef.

Revert "revert to just reformatting changes"

4404b85

This reverts commit d898469.

Revert "revert to just reformatting changes"

e7909aa

This reverts commit d755c51.

Revert "revert to just reformatting changes"

71cc6ed

This reverts commit 03a15e7.

mikealfare changed the base branch from main to feature/materialized-views/adap-814-prep August 19, 2023 01:55

Base automatically changed from feature/materialized-views/adap-814-prep to main August 21, 2023 18:44

mikealfare added a commit that referenced this pull request Aug 21, 2023

ADAP-814: Refactor prep for MV updates (#8459)

ff98685

* apply reformatting changes only for #8449 * add logging back to get_create_materialized_view_as_sql * changie

mikealfare added 5 commits August 21, 2023 17:17

Merge branch 'main' into feature/materialized-views/adap-814

5dc4aea

# Conflicts: # core/dbt/include/global_project/macros/relations/drop.sql # core/dbt/include/global_project/macros/relations/rename.sql # plugins/postgres/dbt/include/postgres/macros/relations/materialized_view/alter.sql

migrate relations macros from macros/adapters/relations to `macros/…

38bbbdb

…relations` so that generics are close to the relation specific macros that they reference; also aligns with adapter macro files structure, to look more familiar

update comment to be more clear

a6c5c0e

loosen type on relations_that_can_be_renamed to StrEnum to account fo…

6dbaabb

…r adapters that have a unique list of relation types, e.g. dbt-snowflake

move drop macro to drop macro file

aaec703

mikealfare marked this pull request as ready for review August 25, 2023 21:53

mikealfare requested review from a team as code owners August 25, 2023 21:53

mikealfare requested review from nathaniel-may and gshank and removed request for gshank and nathaniel-may August 25, 2023 21:53

mikealfare added 4 commits August 28, 2023 16:29

remove view and table from generic create template, it's not needed a…

6159fd4

…nd can be addressed properly at a later date

align the behavior of get_drop_sql and drop_relation, adopt existing …

99a1bb4

…default from drop_relation

whitespace/readability

17aae86

add explicit ddl for drop statements instead of inheriting the defaul…

a31189e

…t from dbt-core

peterallenwebb pushed a commit that referenced this pull request Aug 30, 2023

ADAP-814: Refactor prep for MV updates (#8459)

7bb45d5

* apply reformatting changes only for #8449 * add logging back to get_create_materialized_view_as_sql * changie

mikealfare marked this pull request as draft August 31, 2023 15:07

mikealfare marked this pull request as ready for review August 31, 2023 20:51

mikealfare commented Aug 31, 2023

View reviewed changes

core/dbt/include/global_project/macros/relations/replace.sql Outdated Show resolved Hide resolved

mikealfare commented Aug 31, 2023

View reviewed changes

core/dbt/adapters/base/relation.py Outdated Show resolved Hide resolved

mikealfare added 3 commits August 31, 2023 17:50

update renameable relations attribute on BaseRelation

24ebc37

update replace macro dependent macros to align with naming standards

a7db607

update type for mashumaro, update related test

8b643bd

colin-rogers-dbt approved these changes Sep 1, 2023

View reviewed changes

mikealfare merged commit adfa322 into main Sep 1, 2023

mikealfare deleted the feature/materialized-views/adap-814 branch September 1, 2023 00:41

tlento mentioned this pull request Nov 6, 2023

Update typing-extensions version to >=4.4 #9012

Merged

5 tasks

dinigo mentioned this pull request Jan 16, 2024

Support BigQuery Materialized Views dbt-labs/dbt-bigquery#18

Closed

tlento mentioned this pull request Feb 26, 2024

Update dbt-semantic-interfaces dependency to compatible range #9671

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ADAP-814: Add support for replacing materialized views with tables/views and vice versa #8449

ADAP-814: Add support for replacing materialized views with tables/views and vice versa #8449

mikealfare commented Aug 17, 2023 •

edited

Loading

github-actions bot commented Aug 17, 2023

codecov bot commented Aug 17, 2023 •

edited

Loading

ADAP-814: Add support for replacing materialized views with tables/views and vice versa #8449

ADAP-814: Add support for replacing materialized views with tables/views and vice versa #8449

Conversation

mikealfare commented Aug 17, 2023 • edited Loading

Problem

Solution

Checklist

github-actions bot commented Aug 17, 2023

codecov bot commented Aug 17, 2023 • edited Loading

Codecov Report

mikealfare commented Aug 17, 2023 •

edited

Loading

codecov bot commented Aug 17, 2023 •

edited

Loading