Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix issues around new get_catalog_by_relations macro #8856

Merged
merged 5 commits into from
Oct 26, 2023

Conversation

peterallenwebb
Copy link
Contributor

@peterallenwebb peterallenwebb commented Oct 13, 2023

resolves #8846

Problem

An optional parameter was added to BaseAdapter.get_catalog() in 1.7.0rc1, which was unfortunately overridden by databricks, and may also be overridden in other adapter implementations.

Solution

This changeset reverts the function to its original signature, and adds a new function called BaseAdapter.get_filtered_catalog(). It also reworks the docs generate code which had relied previously relied on the new parameter.

Significantly, the filtering is now performed against a set of BaseRelation objects passed to the adapter, instead of a set of nodes. This reduces the number of concepts that leak into the adapter, which should not need to "know" about the dag or nodes.

Checklist

  • I have read the contributing guide and understand what's expected of me
  • I have run this code in development and it appears to resolve the stated issue
  • This PR includes tests, or tests are not required/relevant for this PR
  • This PR has no interface changes (e.g. macros, cli, logs, json artifacts, config files, adapter interface, etc) or this PR has already received feedback and approval from Product or DX
  • This PR includes type annotations for new and modified functions

@cla-bot cla-bot bot added the cla:yes label Oct 13, 2023
@github-actions
Copy link
Contributor

Thank you for your pull request! We could not find a changelog entry for this change. For details on how to document a change, see the contributing guide.

@codecov
Copy link

codecov bot commented Oct 13, 2023

Codecov Report

Attention: 5 lines in your changes are missing coverage. Please review.

Comparison is base (ef9d6a8) 86.31% compared to head (1c3128f) 86.44%.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #8856      +/-   ##
==========================================
+ Coverage   86.31%   86.44%   +0.13%     
==========================================
  Files         177      177              
  Lines       26385    26405      +20     
==========================================
+ Hits        22775    22827      +52     
+ Misses       3610     3578      -32     
Flag Coverage Δ
integration 83.23% <91.07%> (+0.13%) ⬆️
unit 64.80% <64.28%> (+0.21%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files Coverage Δ
core/dbt/adapters/base/impl.py 82.85% <94.59%> (+2.59%) ⬆️
core/dbt/task/generate.py 86.88% <84.21%> (-0.69%) ⬇️

... and 1 file with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@peterallenwebb peterallenwebb marked this pull request as ready for review October 24, 2023 21:15
@peterallenwebb peterallenwebb requested review from a team as code owners October 24, 2023 21:15
@peterallenwebb peterallenwebb merged commit a2a7b7d into main Oct 26, 2023
51 checks passed
@peterallenwebb peterallenwebb deleted the paw/fix-get-catalog branch October 26, 2023 19:23
@mikealfare
Copy link
Contributor

I think we might have introduced a pre-regression with this PR. See the details here: dbt-labs/dbt-snowflake#816.

peterallenwebb added a commit that referenced this pull request Oct 31, 2023
* Fix issues around new get_catalog_by_relations macro

* Add changelog entry

* Fix unit test.

* Additional unit testing
peterallenwebb added a commit that referenced this pull request Oct 31, 2023
* Fix issues around new get_catalog_by_relations macro (#8856)

* Fix issues around new get_catalog_by_relations macro

* Add changelog entry

* Fix unit test.

* Additional unit testing

* Fix cased comparison in catalog-retrieval function (#8940)

* Fix cased comparison in catalog-retrieval function.

* Fix cased comparison in catalog-retrieval function.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[CT-3216] [Regression] Calls to get_catalog are not all wrapped in the feature flag
3 participants