Adapter Maintainers: Upgrading to dbt-core 1.4.0 #6624

dataders · 2023-01-17T03:51:29Z

dataders
Jan 17, 2023
Maintainer

TL;DR Not much heavy lifting for this minor version. But FYI there will be more work for 1.5.0. Be warned!

This discussion is for communicating to adapter maintainers the scope of work needed to make use of the changes in 1.4.0. If you have questions and concerns, please ask them here for posterity.

For the guide to upgrading an adapter to be compatible with dbt-core=1.3.0, please see #6011.

The latest release cut for 1.4.0, dbt-core==1.4.0rc1, was published on January 11, 2023 (PyPI | Github). We are targeting releasing the official cut of 1.4.0 in time for January 25, 2023. We ask that adapter maintainers strive to release an official version of corresponding adapter package by February 8, 2023.

Upgrade dependencies

Be sure to also upgrade the version of dbt-tests-adapter

# for all package users
# dbt-core~=1.4.0rc1 until final release
dbt-core~=1.4.0

# for adapter tests from core
# dbt-tests-adapter~=1.4.0rc1
dbt-tests-adapter~=1.4.0 until final release

Add support for Python 3.11

The long wait is over for Guido's MSFT-funded foray into speeding up his pet project. However these performance improvements are not immediately realizble in dbt until key package dependencies first start supporting (and testing against) Python 3.11. As of the time of writing (1/16/23), most Python database driver libraries/packages support Python 3.11

If possible, please update your adapter to support Python 3.11 and also test against it. Verified adapters must test against this version, and adapters included in cloud must test against all currently supported minor version of the Python 3 (i.e. 3.7 though 3.11).

Consolidate timestamp functions & macros

Most of the work was done in core (#5521 https://github.com/dbt-labs/dbt-bigquery/pull/323/files). Feel free to dive in to the PRs and related issues, but my TL;DR is that just because macros had the same name across adapters didn't mean the implementation was the same or was guaranteed to return the same datatype. The consensus is now to standardize timestamp-related macros to make use of the convert_timezone() macro and built in Python logic.

There is a new adapter-zone test to implement to ensure things are hunky-dory:
dbt.tests.adapter.utils.test_timestamps.BaseCurrentTimestamps

Any adapter-specific override of the following time-related macros should now live in macros/utils/timestamps.sql
- current_timestamp
- snapshot_string_as_time
- snapshot_get_time

Relatedly, if you maintain a shim of dbt-utils that overrides these macros, changes similar to dbt-labs/dbt-utils#694 may be required

Replace deprecated exception functions

With the exception to some logging upgrades and small tweaks, dbt-core's Exceptions library (core/dbt/exceptions.py) has sat largely dormant for nearly three years. Strong exception handling interface is key to a solid API upon which other teams and tools can rely.

#6347
dbt-labs/dbt-snowflake#366

RuntimeException -> DbtRuntimeError
DatabaseException -> DbtDatabaseError
raise_compiler_error() -> CompilationError

Incremental Predicates

Incremental Predicates is a new feature for 1.4. See #5702 for more information

There are also tests (TestIncrementalPredicatesDeleteInsert, TestPredicatesDeleteInsert )that can be used to validate the implementation if your adapter supports either of the mentioned incremental strategies.

Additional Tests

Incremental Tests

If your adapter has a custom incremental "unique key" strategy and/or materializaiton TestIncrementalUniqueKey is a recommended approach
#4958

Query Comment Tests

If your data platform supports query comments, then #5971 migrates the related tests to the adapter zone for you to implement (see dbt-labs/dbt-redshift#197).

TestChangeRelationTypes

TODO explain the value of this test.... @colin-rogers-dbt? #6054 dbt-labs/dbt-redshift#203

Example adapter diffs

dbt-labs/dbt-redshift@v1.3.0...v1.4.0rc1
dbt-labs/dbt-bigquery@v1.3.0...v1.4.0rc1
dbt-labs/dbt-snowflake@v1.3.0...v1.4.0rc1
dbt-labs/dbt-spark@v1.3.0...v1.4.0rc1
dbt-labs/dbt-bigquery@v1.3.0...v1.4.0rc1

morsapaes · 2023-01-17T11:38:30Z

morsapaes
Jan 17, 2023

Thanks for the overview (as usual), @dataders!

There is a pending question from our side in terms of test coverage: is there a recommendation around what tests must be included in each external adapter's test suite? Since dbt-materialize tests run in our main CI pipeline and the adapter largely shims dbt-postgres, we'd like to keep things slim if things can be kept slim!

1 reply

dataders Jan 30, 2023
Maintainer Author

fair question. I can't speak to y'all's optimization tradeoffs. that's a "you problem" 😺 .

A confounding factor here is that this suite aren't unit tests, but rather integration tests in that they're useless without a data platform on which to run them. In theory you could just run the subset of adapter/ zone tests that are distinct from dbt-postgres's, but, what assurances do we have that the dbt-postgres code used as-in on Materialize's database will work?

See that all relevant tests have passed for each release is a requirement to be a "verified" core adapter. But that's only for the release commits, so perhaps there's tweaking of the GitHub actions such that the full suite runs only on commits associated with releases? otherwise it is the subset? 🤷🏻

That said, we're always looking to improve the value and UX of our adapter testing suite, so if there's specific feedback about tests that are too slow, let us know!

genzgd · 2023-02-02T16:24:40Z

genzgd
Feb 2, 2023

Is there discussion of this change somewhere?

quote_policy: SnowflakeQuotePolicy = field(default_factory=lambda: SnowflakeQuotePolicy())

Are all policies required to be a dataclasses.field now?

2 replies

colin-rogers-dbt Feb 2, 2023
Maintainer

This wasn't a change we wanted to make but a requirement for supporting py3.11, see: #6326

genzgd Feb 2, 2023

Ah, that makes sense for dataclasses, thanks!

nicor88 · 2023-02-03T15:26:18Z

nicor88
Feb 3, 2023

When attempting to upgrade the dbt-athena adapter (PR: dbt-labs/dbt-athena#134) I'm having few issues:

caplog.text used in our unit testing (with pytest) to catch some errors/messages when in debug mode, started to be empty (it was fully working on version 1.3.x) - could be that dbt is now flushing the caplog therefore is empty. Could be that logger.propagate = False on the logs are not push down.

~~some of our functions tests started to fail~~

FAILED tests/functional/adapter/utils/test_utils.py::TestArrayAppend::test_expected_actual - TypeError: '_MISSING_TYPE' object is not callable
FAILED tests/functional/adapter/utils/test_utils.py::TestArrayConstruct::test_expected_actual - TypeError: '_MISSING_TYPE' object is not callable
FAILED tests/functional/adapter/utils/test_utils.py::TestArrayConcat::test_expected_actual - TypeError: '_MISSING_TYPE' object is not callable
FAILED tests/functional/adapter/utils/test_utils.py::TestExcept::test_build_assert_equal - TypeError: '_MISSING_TYPE' object is not callable
FAILED tests/functional/adapter/utils/test_utils.py::TestIntersect::test_build_assert_equal - TypeError: '_MISSING_TYPE' object is not callable

~~Seems due to~~

@classmethod
  def get_default_include_policy(cls) -> Policy:
>       return cls._get_field_named("include_policy").default_factory()
E       TypeError: '_MISSING_TYPE' object is not callable

~~Any clue on what to fix here?~~
Fixed adding :

 @dataclass(frozen=True, eq=False, repr=False)
 class AthenaRelation(BaseRelation):
    quote_character: str = ""
    include_policy: Policy = field(default_factory=lambda: AthenaIncludePolicy())

Overall the core functionalities are working, but the above is preventing us to move further with merging and releasing to the new version.

5 replies

genzgd Feb 3, 2023

For the missing type do something like this in your Relation subclass:

@dataclass(frozen=True, eq=False, repr=False)
class ClickHouseRelation(BaseRelation):
    quote_policy: Policy = field(default_factory=lambda: ClickHouseQuotePolicy())
    include_policy: Policy = field(default_factory=lambda: ClickHouseIncludePolicy())

genzgd Feb 3, 2023

I had the same problem prompted my comment above: #6624 (comment)

nicor88 Feb 3, 2023

~~I tried with one of the failing tests and seems that worked.~~ All our functional testings are now working again.

dataders Feb 3, 2023
Maintainer Author

looks like this finding is worth adding the the main upgrade guide -- either of y'all mind writing up a snippet and I'll add it in?

nicor88 Feb 3, 2023

I think that you can pick what @genzgd posted

from dataclasses import dataclass, field

@dataclass(frozen=True, eq=False, repr=False)
class ClickHouseRelation(BaseRelation):
    quote_policy: Policy = field(default_factory=lambda: ClickHouseQuotePolicy())

ravjotbrar · 2023-02-22T00:46:41Z

ravjotbrar
Feb 22, 2023

Is the new tests/adapter/dbt/tests/adapter/utils/test_timestamps.py test only applicable to postgres and snowflake adapters? I see this in the changes for the relevant macros in timestamp.sql:

/* {#
    DEPRECATED: DO NOT USE IN NEW PROJECTS

    This is ONLY to handle the fact that Snowflake + Postgres had functionally
    different implementations of {{ dbt.current_timestamp }} + {{ dbt_utils.current_timestamp }}

    If you had a project or package that called {{ dbt_utils.current_timestamp() }}, you should
    continue to use this macro to guarantee identical behavior on those two databases.
#} */

0 replies

dataders · 2023-04-19T16:02:05Z

dataders
Apr 19, 2023
Maintainer Author

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Adapter Maintainers: Upgrading to dbt-core 1.4.0 #6624

{{title}}

Replies: 5 comments 8 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Select a reply

Adapter Maintainers: Upgrading to dbt-core 1.4.0 #6624

dataders Jan 17, 2023 Maintainer

Upgrade dependencies

Add support for Python 3.11

Consolidate timestamp functions & macros

Replace deprecated exception functions

Incremental Predicates

Additional Tests

Incremental Tests

Query Comment Tests

TestChangeRelationTypes

Example adapter diffs

Replies: 5 comments · 8 replies

dataders Jan 30, 2023 Maintainer Author

colin-rogers-dbt Feb 2, 2023 Maintainer

dataders Feb 3, 2023 Maintainer Author

dataders Apr 19, 2023 Maintainer Author

dataders
Jan 17, 2023
Maintainer

Replies: 5 comments 8 replies

dataders Jan 30, 2023
Maintainer Author

colin-rogers-dbt Feb 2, 2023
Maintainer

dataders Feb 3, 2023
Maintainer Author

dataders
Apr 19, 2023
Maintainer Author