Adapter Maintainers: Upgrading to dbt-core v1.7.0

[dataders]

Overview

This discussion is for communicating to adapter maintainers the scope of work needed to make use of the changes in 1.7.0. If you have questions and concerns, please ask them here for posterity.

Please consider this a living document between now and the date of final release. If there's something missing, please comment below!

Loom video overview (12 min)

Adapter Maintainers: Upgrading to dbt-core v1.7.0 - Watch Video

release timeline

The below table gives the milestones between up to and including the final release. It will be updated with each subsequent release.

• [CT-2894] Release v1.7.0b1 - August 17 #8261
• [CT-2895] Release v1.7.0rc1 - October 12 #8262
• [CT-2896] Release v1.7.0 (final) - November 2 #8263
• Release v1.7.0 to cloud October 30

prior maintainer upgrade guides

• Adapter Maintainers: Upgrading to dbt-core `1.6.0` #7958
• Adapter Maintainers: Upgrading to dbt-core 1.5.0 #7213
• Adapter Maintainers: Upgrading to dbt-core 1.4.0 #6624
• Adapter Maintainers: Upgrading to dbt-core 1.3.0 #6011
• Adapter Maintainers: Upgrading to dbt-core 1.2.0 #5468

Example Diffs

• dbt-labs/dbt-redshift@v1.6.0...1.7.latest
• dbt-labs/dbt-bigquery@v1.6.0...1.7.latest
• dbt-labs/dbt-snowflake@v1.6.0...1.7.latest
• dbt-labs/dbt-spark@v1.6.0...1.7.latest

TL;DR

Excluding the work to consume our refactoring work for materialized views, there should be no breaking changes. Please tell me if I'm wrong, but I believe that things should work by bumping package and dependency versions. I recommend adding tests. Everything below is nice-to-have first steps toward long-term stable interfaces we will develop in future versions.

Breaking changes

#8846 is a pre-regression in 1.7.0rc1 that will be remediated for the final release of 1.7.0. The issue only impacts adapters that have overridden BaseAdapter.get_catalog() method. However, this does not affect adapters that have overridden the get_catalog(), macro.

If you wish to temporarily unblock yourself, you may change your method's function signature to match that of the base class, but ultimately, this call will be wrapped in the capability flag so you only need update it if you plan on supporting the more precise get_catalog functionality.

Changes

new capability support structure for adapters

dbt-core 1.7.0 improves performance of two existing features: source freshness and catalog metadata fetching (see below for more). However, these improvements are only available contingent upon underlying data platform support.

Traditionally, this adapter interface to call this out would be a single jinja macro or python method that may be overridden accordingly, but the end result is one macro that varies in what it does across adapters.

Our solution here is to define adapter/database capability in a top-level structure, that then "dispatches" to the corresponding macro/function. This way there's a stronger separation of concerns.

A new static member variable on BaseAdapter called _capabilities with type CapabilityDict can be overriden by adapter implementations. This new member variable is introduced to more easily flag to dbt-core whether or not an adapter supports a given dbt feature as well as whether the database itself supports the feature. dbt/adapters/capability.py has great information on how it's structured.

From 1.7 forward, for a given relevant feature we'll define a new Capability within the Capability class, so that it may be defined within an adapter. The possible support states are defined within the Support class, namely: "Unknown", "Unsupported", "NotImplemented", "Versioned", or "Full".

If you as an adaper maintainer want to define that your data platform has does not support a new feature, SomeNewFeature, you'd define it as the below.

_capabilities: CapabilityDict = CapabilityDict(
    {Capability.SomeNewFeature: CapabilitySupport(support=Support.Unsupported)}
)

I'm very excited at the potential of this feature. I can imagine that much of an adapter might be abstracted into a structure like this, so that where previously a macro-override was required, now it's just another Dict entry.

metadata freshness checks

#7012 #8704 #8795 dbt-labs/dbt-snowflake#796

Excerpt from #7102

Currently, source freshness can only be determined by querying a column in a table. When there are a lot of tables, even with a high number of threads, the amount of time it takes to compute source freshness might be unacceptably long - it's effectively just metadata collation after all.
However, lots of databases track table modification times (although don't always isolate DDL changes from DML changes) and expose this via a metadata route. For example, BigQuery has the well-known INFORMATION_SCHEMA tables which expose various metadata attributes.

How to implement

Note

This is the first of the two new features enabled via the new Capability dict.

Warning

if you're overriding Adapter.calculate_freshness() the type signature has changed so please update accordingly: dbt-core#8795#discussion_r1355356418

If your data platform has Full support for this new functionality, you will need to:

1. add a new entry within the _capabilities Dict (see above section) to define. in this case, the feature is TableLastModifiedMetadata. See dbt-snowflake#796#impl.py for the change that's needed.
2. add a dispatched get_relation_last_modified() to define how to fetch this freshness metadata with a new macro. Note that default__get_relation_last_modified() is not implemented, so you must implement your own. See snowflake__get_relation_last_modified(). The function signature is similar to that of the below described get_catalog_tables_sql(), but returns a table with four columns:
   1. schema
   2. identifier,
   3. last_modified the actual freshness metadata of the table
   4. snapshotted_at current timestamp so we know when dbt checks for this

Freshness Tests

• TestGetLastRelationModified
• TestListRelationsWithoutCaching has been deprecated and broken into the following:
  • TestListRelationsWithoutCachingSingle
  • TestListRelationsWithoutCachingFull

Catalog fetch performance improvements

#8521 #8648 dbt-snowflake#758

Note

This is the second of the two new features enabled via the new Capability dict.

The performance of parsing projects and generating documentation has always become a bottleneck when one of the two conditions are met:

• Projects grow very large (10,000+ models), or
• Projects are defined within databases and schemas that contain many pre-existing objects

A change for 1.7 addresses the second scenario. The get_catalog() macro has always accepted a list of schemas as an argument. All the relations that exist for the given list of schemas are output, regardless of whether or not dbt interacts with this object.

The adapter implementation to solve this problem is three-fold:

1. Introduce a new macro, get_catalog_relations() which accepts a list of relations, rather than a list of schemas.
2. A new entry within the _capabilities Dict (see above section) to define whether support for SchemaMetadataByRelations is Full, Unsupported, or NotImplemented. See dbt-snowflake#758#impl.py for the change that's needed.
3. Decompose the existing get_catalog() macro in order to minimize redundancy with body of get_catalog_relations(). This introduces some additional macros:
   1. get_catalog_tables_sql() copied straight from pre-existing get_catalog() everything you would normally fetch from INFORMATION_SCHEMA.tables
   2. get_catalog_columns_sql() copied straight from pre-existing get_catalog() everything you would normally fetch from INFORMATION_SCHEMA.columns
   3. get_catalog_schemas_where_clause_sql(schemas) copied straight from pre-existing get_catalog(). This uses jinja to loop through the provided schema list and make a big WHERE clause of the form:
      WHERE info_schema.tables.table_schema IN "schema1" OR info_schema.tables.table_schema IN "schema2" [OR ...]
   4. get_catalog_relations_where_clause_sql(relations) this is likely the only new thing

get_catalog_relations_where_clause_sql

This macro is effectively an evolution of the old get_catalog WHERE clause, except now it has the following control flow.

Keep in mind that relation in this instance can be a standalone schema, not necessarily an object with a three-part identifier.

for relation provided list of relations
1. if relation has an identifier and a schema
    1. then write WHERE clause to filter on both
2. elif relation has a schema
    1. do what was normally done, filter where info_schema.table.table_schema == relation.schema 
3. else throw exception. Houston we do not have a something we can use.

The result of the above is that dbt can, with "laser-precision" fetch metadata for only the relations it needs, rather than the superset of "all the relations in all the schemas in which dbt has relations".

Catalog Query Structure

Where the below mentioned OBJECT_LIST is relations or schemas depending on the macro.

{% set query %}
    with tables as (
        {{ myadapter__get_catalog_tables_sql(information_schema) }}
        {{ myadapter__get_catalog_OBJECT_LIST_where_clause_sql(OBJECT_LIST) }}
    ),
    columns as (
        {{ myadapter__get_catalog_columns_sql(information_schema) }}
        {{ myadapter__get_catalog_OBJECT_LIST_where_clause_sql(OBJECT_LIST) }}
    )
    {{ myadapter__get_catalog_results_sql() }}
{%- endset -%}

Tests

TestDocsGenerateOverride in tests/functional/artifacts/test_override.py was modified to cover this new behavior. If you are not already implementing this, I suggest that you should.

Behavior of dbt show's --limit flag

#8496 #8641

dbt show shipped in 1.5.0 with a --limit flag that, when provided, would limit the number of results that dbt would grab to display. It does not modify the original query, which means that even if you provide --limit 5, the command will not complete until the entire underlying query is complete which can take a long time if the query's result set is large. This is especially evident because dbt show is now also used for dbt Cloud IDE's "preview" button.

The fix was #8641, which extends the implementation of the --limit flag to wrap the underlying query into a CTE and append a LIMIT {limit} clause.

No change is needed if the adapter's data platform supports ANSI SQL LIMIT clauses. Any changes can be made in a dispatched version of get_limit_subquery_sql(), see default__get_limit_subquery(). This was merged into 1.7.0, but also back-ported to 1.5 and 1.6, so if a change is needed, it will likely also need to be back-ported and patched accordingly.

dbt.tests.adapter.dbt_show.test_dbt_show contains new tests to ensure the new behavior functions properly:

• BaseShowSqlHeader
• BaseShowLimit

Migrate date_spine() Macro from dbt-utils to Core

#8172 #8616

Following on from initiative #5520, dbt-utils.date_spine() (and the macros on which it directly depends) now lives within dbt-core in order to better support the semantic layer, which requires a date spine to get started.

Macros that are now testable and overridable within an adapter:

• date_spine
• generate_series
• get_intervals_between
• get_powers_of_two

The default__ version of these macros is compatible with the adapters we currently support in dbt Cloud, with the exception of the following that should be migrated:

• Starburst/Trino, which has dbt-trino-utils.date_spine,
• Fabric which has tsql-utils.date_spine

Data Spine Tests

Important

Important! The following test cases must be added to the adapter to ensure compatibility now and moving forward. See #8616 changes to dbt/tests/adapter/utils/ directory for more info.

• TestDateSpine
• TestGenerateSeries
• TestGetIntervalsBetween
• TestGetPowersOfTwo

Storing Test Failures as View

#6914 #8653

There's a new config for tests, store_failures_as which can be table, view, or ephemeral. table and view will commonly be used to turn on the store failure feature. ephemeral is mostly used to turn off the store failures feature at a lower grain (e.g. model) after it has been turned on at a higher level (e.g. project). The new store_failures_as config, when set by the user, will override the existing config store_failures, which will be deprecated within the next two minor releases, i.e. either 1.8 or 1.9. If the user does not set store_failures_as, store_failures will continue to behave as it has in the past until deprecation.

Note

Provided that your adapter doesn't have a custom test materialization overriding Core's default, there shouldn't be work required here beyond adding the below tests. If the test materialization is overridden, then the change is to inspect config.get("store_failures", "table") and use that as the relation_type instead of a hard-coded "table". There could be other changes required, but that's the gist of it. I'll leave this as standalone section rather than listing the available tests just in case.

Store Failures Tests

• TestStoreTestFailuresAsInteractions
• TestStoreTestFailuresAsProjectLevelOff
• TestStoreTestFailuresAsProjectLevelView
• TestStoreTestFailuresAsGeneric
• TestStoreTestFailuresAsProjectLevelEphemeral
• TestStoreTestFailuresAsExceptions

Additional Tests

Important

These are new tests introduced into the adapter zone that you should have in your adapter.

Tests	Issue	PR	note
TestIncrementalForeignKeyConstraint	#8022	#8768	not relevant to adapters, no need to implement
TestCloneSameTargetAndState	#8160	#8638	dbt clone
SeedUniqueDelimiterTestBase TestSeedWithWrongDelimiter TestSeedWithEmptyDelimiter	#3990	#7186	custom-delimiter for seeds

Materialized Views Refactor

Note

Under construction

Materialized views introduced a few pieces of purely new functionality within dbt-core. This resulted in some refactoring work that extended beyond materialized views to fold them into the mix. The highlights worth calling out:

• We implemented change monitoring (on_configuration_change), which requires inspecting the materialized view at run time
• We can no longer rely on the assumption that !view == table
• Tables and views can often be swapped out for each other as they share functionality (atomic replace, rename, etc.); materialized views tend to have less functionality

This limits the following functionality:

• Backing up objects before replacing (due to renaming)
• Soft deploying objects before validating they were created successfully (due to renaming)
• Replacing objects of a different type (due to atomic replace)

At a high level, the approach that was taken establishes a relation-type agnostic operation layer that dispatches down to the relation-type specific layer, where possible. This allows for more granular control, though requires some copy paste. We determined the benefits outweigh the costs related with violating DRY principles. Let's discuss the jinja templates first.

Jinja Updates

While the file structure ultimately does not matter for jinja, we adopted the structure below to more easily navigate the files. Additionally, we're striving for a one-to-one relation between files and macros (effectively the opposite of adapters.sql).

macros
|-- materializations
|   |-- materialized_view.sql
|   |__ ...
|
|-- relations
|   |-- materialized_view
|   |   |-- alter.sql
|   |   |-- create.sql
|   |   |-- describe.sql
|   |   |-- drop.sql
|   |   |-- refresh.sql
|   |   |-- rename.sql
|   |   |__ replace.sql
|   |
|   |-- table
|   |   |-- create.sql
|   |   |-- drop.sql
|   |   |-- rename.sql
|   |   |__ replace.sql
|   |
|   |__ view
|       |-- create.sql
|       |-- drop.sql
|       |-- rename.sql
|       |__ replace.sql
|
|-- utils
|   |__ ...
|
|__ ...

These files all represent macros that need to be defined. Some of them likely already exist, like create for table and view. In that case, I would move them here for clarity. And drop may also exist, but likely in the form drop {{ relation.type }} {{ relation }}. Unfortunately that no longer works with materialized views. Instead of parsing relation.type, and to be consistent with breaking it out by type (e.g. create'), we elected to go with a macro per type. If this really irks you, you can keep one drop macro and call it within each of the drop macros by type, but we need the drop macros by type since they are called within dbt-core.

The standout file materializations/materialized_view.sql contains a macro that gets used while building the materialization in dbt-core. It determines the differences between the current and future states. Some folks (e.g. the author) just put this in materialized_view/alter.sql since there is a heavy correlation between the two; the output of one is the input of the other.

In addition to creating these macros, you'll need to update things like list_relations_without_caching, get_catalog, etc. to ensure that they include this new object type.

dbt-redshift and dbt-bigquery are good examples of how these macros were implemented. There are some differences because they are different platforms, but there are many similarities as well. This should give you a good feel for what's required and how to extend that to match your needs.

Python Updates - Jinja Config

There are a few configuration updates on your MyAdapterRelation that are needed to support the jinja updates. In particular, these are needed:

    # map relation types to python relation objects
    relation_configs = {
        RelationType.MaterializedView.value: MyAdapterMaterializedView,
    }
    # list relations that can be renamed (e.g. `RENAME my_relation TO my_new_name;`)
    renameable_relations = frozenset(
        {
            RelationType.View,
            RelationType.Table,
        }
    )
    # list relations that can be atomically replaced (e.g. `CREATE OR REPLACE my_relation..` versus `DROP` and `CREATE`)
    replaceable_relations = frozenset(
        {
            RelationType.View,
        }
    )

Python Updates - Relation Models

The approach we took for the jinja templates differs from that of tables and views. In the latter, the config object that's in the global jinja context gets parsed within the template. This embeds a lot of logic into the template. This is still a valid approach if you wish to go down this path. However, that will make change monitoring very difficult to implement. The alternative is to parse the config object into a python object. This allows for much easier comparison of current state and future state. However, if we're already parsing config for the future state, you might as well use that for the rest of the jinja macros as well. Ultimately, this is your choice, but we will talk about how we did this in the adapters dbt Labs maintains, which implements relations as a python object.

I'll start by calling out the caveat that we wanted to implement base classes slowly, to avoid getting locked in functionality that we would like to change in the future. The base classes are sparse, which aligns with our goal of minimal (hopefully none) breaking changes. With that in mind, there is no reason that you need to inherit from the base relation object in dbt-core. If you'd like to build your own object, go for it. As mentioned above, it's possible to implement all (almost, I'll point that out later) of this within jinja, we just think that would be a terrible time. In general, you can think of the materialized view as a class that marshalls config or database metadata data into a dataclass. Here's the suggested structure of a materialized view python object:

@dataclass(frozen=True, eq=True, unsafe_hash=True)
class MyAdapterMaterializedView:
    # configuration attributes that define the materialized view, e.g.
    identifier: str
    schema_name: str
    database_name: str
    something_about_refresh: str
    maybe_a_cluster_config: MyAdapterClusterConfig

    # some useful calculated properties
    @property
    def path(self) -> str:
        return ".".join([self.database_name, self.schema_name, self.identifier])

    # parsing methods
    @classmethod
    def from_dict(cls, config_dict: Dict[str, Any]) -> Self:
        # handle nested objects here, like `self.maybe_a_cluster_config`
        return cls(**config_dict)

    @classmethod
    def from_model_node(cls, model_node: ModelNode) -> Self:
        return cls.from_dict(cls.parse_model_node(model_node))

    @classmethod
    def parse_model_node(cls, model_node: ModelNode) -> Dict[str, Any]:
        # parse the global jinja context object `config.model` into the `dict` version of this class
        # handle nested objects here, like `self.maybe_a_cluster_config`
        my_config = {
            "identifier": model_node.get("identifier"),
            "schema_name": model_node.get("schema"),
            "database_name": model_node.get("database"),
            "something_about_refresh": model_node.config.extras.get("refresh"),
        }
        if cluster := model_node.config.extras.get("refresh"):
            my_config.update({"maybe_a_cluster_config": MyAdapterClusterConfig.parse_model_node(cluster)})
        return my_config

    @classmethod
    def from_relation_results(cls, relation_results: RelationResults) -> Self:
        return cls.from_dict(cls.parse_relation_results(relation_results))

    @classmethod
    def parse_relation_results(cls, relation_results: RelationResults) -> Dict[str, Any]:
        # parse the results of `relations/materialized_view/describe.sql` into the `dict` version of this class
        # handle nested objects here, like `self.maybe_a_cluster_config`
        return relation_results.to_dict()     

Again, dbt-redshift and dbt-bigquery are examples of how we implemented this object. This structure allows you to easily compare the current object (from_relation_results) to the future object (from_model_node). We also created this comparison as a python object by creating a method on MyAdapterRelation: dbt-redshift and dbt-bigquery. Then our get_materialized_view_configuration_changes macro simply passes the two versions of the relation into this macro and hands back the results.

Note

If you elect to go with a jinja implementation of get_materialized_view_configuration_changes, make sure you hand back an object with a property .has_changes == False if there are no changes (versus an empty object, or empty list). This value gets checked in the materialization to see if there are changes to apply, or if a refresh should occur.

[dataders]

using a comment to track progress across all (known) adapters

1.7 upgrade issues

• upgrade to support dbt-core v1.7.0 databricks/dbt-databricks#478
• upgrade to support dbt-core v1.7.0 dbt-athena#454
• upgrade to support dbt-core v1.7.0 starburstdata/dbt-trino#361
• upgrade to support dbt-core v1.7.0 ClickHouse/dbt-clickhouse#196
• upgrade to support dbt-core v1.7.0 duckdb/dbt-duckdb#268
• https://github.com/MaterializeInc/materialize/issues/22332
• upgrade to support dbt-core v1.7.0 oracle/dbt-oracle#108
• upgrade to support dbt-core v1.7.0 microsoft/dbt-synapse#190
• upgrade to support dbt-core v1.7.0 microsoft/dbt-fabric#77
• upgrade to support dbt-core v1.7.0 Teradata/dbt-teradata#113
• upgrade to support dbt-core v1.7.0 aws-samples/dbt-glue#243
• upgrade to support dbt-core v1.7.0 tglunde/dbt-exasol#104
• upgrade to support dbt-core v1.7.0 vertica/dbt-vertica#110
• upgrade to support dbt-core v1.7.0 dremio/dbt-dremio#209
• upgrade to support dbt-core v1.7.0 dbt-msft/dbt-sqlserver#451
• upgrade to support dbt-core v1.7.0 codeforkjeff/dbt-sqlite#50
• upgrade to support dbt-core v1.7.0 firebolt-db/dbt-firebolt#111
• upgrade to support dbt-core v1.7.0 markporoshin/dbt-greenplum#31
• upgrade to support dbt-core v1.7.0 cloudera/dbt-hive#131
• upgrade to support dbt-core v1.7.0 aurany/dbt-ibmdb2#45
• upgrade to support dbt-core v1.7.0 cloudera/dbt-impala#183
• upgrade to support dbt-core v1.7.0 iomete/dbt-iomete#8
• upgrade to support dbt-core v1.7.0 mindsdb/dbt-mindsdb#31
• upgrade to support dbt-core v1.7.0 dbeatty10/dbt-mysql#157
• upgrade to support dbt-core v1.7.0 chop-analytics/dbt-netezza#23
• upgrade to support dbt-core v1.7.0 rockset/dbt-rockset#33
• upgrade to support dbt-core v1.7.0 memsql/dbt-singlestore#22
• upgrade to support dbt-core v1.7.0 pingcap/dbt-tidb#53
• upgrade to support dbt-core v1.7.0 fal-ai/dbt-fal#909
• upgrade to support dbt-core v1.7.0 databendcloud/dbt-databend#5
• upgrade to support dbt-core v1.7.0 inferlabs/dbt-infer#9
• upgrade to support dbt-core v1.7.0 getindata/dbt-flink-adapter#67
• upgrade to support dbt-core v1.7.0 tanyshak/dbt-upsolver#5

[benc-db]

There isn't a Base version of TestIncrementalForeignKeyConstraint, so pytest is trying to run it as is when I import it (which is very strange behavior indeed).

[dataders]

ok, in consultation with @aranke, we agreed that an adapter maintainer need not concern themselves with this test, So I've struck it from the list of required additional tests

[benc-db]

Well, I made my own version of it, which was a good idea, because I found two bugs in our constraint implementation that were not directly related to the test.

[benc-db]

TestGetLastRelationModified doesn't seem to be a thing (this is the only search result in your repo).

[benc-db]

Also, should it not be TestGetRelationLastModified?

[benc-db]

Yeah, generally we need the tests for metadata freshness checks to come into core and be generalized, as its hard for me to tell what aspects are snowflake specific.

[colin-rogers-dbt]

The dbt-snowflake test was added fairly quickly to validate the functionality. We will prioritize a more generic dbt-core adapter test to make it easier to roll out to other adapters

[damian3031]

Shouldn't TestDocsGenerateOverride be moved to dbt-tests-adapter, so that it can be extended by adapters?

[benc-db]

I don't think the test as currently written is actually useful in the adapters. It validates that jinja delegation happens, but not in an adapter-specific way. We need a better test for the get_catalog_relations feature, because its significant.

[damian3031]

I think that TestCloneSameTargetAndState also doesn't test any adapter-specific logic

[jwills]

Hey @dataders the adapter-specific fixtures in #8616 are irritating to work with inside of the tests for other adapters (and as far as I can tell, there has never been adapter-specific fixture logic inside of the utils before?)

Could y'all fix that up so that it's easy for other adapter developers to specify our preferred date format and what not instead of hardcoding it to be target.type == 'postgres' etc.?

[dbeatty10]

High-level, I think we want this:

• https://github.com/dbt-labs/dbt-core/issues/8831

But the earliest that can land is in v1.8.

In the meantime, this hard-coding is def not what we want. It got inherited when we did a lift-n-shift from dbt-utils into dbt-core, but we should refactor it so that it's easy for adapter maintainers.

Just opened this issue for us to address this:

• https://github.com/dbt-labs/dbt-core/issues/8890

[nicor88]

@dataders In dbt-athena we avoid using information_schema because it's really slow.
For get_catalog we implemented an override of _get_one_catalog that under the hood uses glue APIs to fetch everything that is needed.

I'm trying to implement the same for get_catalog_relations macro, and I took a similar approach, in the end is all down to overwrite _get_one_catalog_by_relations.

Do we have a snippet somewhere that I can use as a reference to test that my macro athena__get_catalog_relations works as expected?

[nicor88]

I implemented a functional test https://github.com/dbt-athena/dbt-athena/pull/488/files#diff-0b8bfd7e8ff4632b6ea34142dffa4fa3ddc351924ae3c2353b0c6e69262ca523R11 similar to TestDocsGenerateOverride, and it's actually successful, not sure if this is enough.

[aosingh]

@dataders For dbt-oracle, database is an optional property however in 1.7 we get the following error

16:15:21  1 check failed:
16:15:21  Profile loading failed for the following reason:
Runtime Error
  Credentials in profile "dbt_test", target "dev" invalid: 'database' is a required property

as this will be a breaking change for users, I am trying to understand if there is some change expected in AdapterCredentials ?

[dataders]

@aosingh can you try and see if oracle/dbt-oracle#118 fixes things for you?

[aosingh]

Thanks @dataders for this PR. I had tried this but that resulted in the famous Dataclass error

 non-default argument follows default argument

For now, as a workaround, I treat all connection parameters as keyword arguments with default = None

[tallamohan]

@dataders, dbt-teradata also sees the same error after upgrading to dbt-core 1.7.x

01:13:57  Profile loading failed for the following reason:
Runtime Error
  Credentials in profile "jaffle_shop", target "dev" invalid: 'database' is a required property```

Please look into this and let us know if any fix will be available from the dbt's side.

[dataders]

@tallamohan @VarunSharma15 I just opened Teradata/dbt-teradata#127. Can you see if it fixes things for you?

[tallamohan]

@dataders , we already tried this solution, but failed with the below error:

non-default argument follows default argument

[dataders]

whoops. looks like my proposal failed also with @aosingh and dbt-oracle (#8307 (reply in thread)).

your choices as I see them:

1. re-order the params (perhaps as I did in Teradata/dbt-teradata@416189d for declare None as default Teradata/dbt-teradata#127)
2. do as @aosingh and make all attr have default args, e.g. None

[colin-rogers-dbt]

Unfortunately this is a rough consequence of dataclass inheritance. I'm 80% sure the only path forward will be to set all your fields to have default args (and then raise an exception if nothing is passed) like we did in dbt-spark: https://github.com/dbt-labs/dbt-spark/blob/01c9fd020ea3e92b35a1ebc9225bf9babf3d16ba/dbt/adapters/spark/connections.py#L65

[tallamohan]

@colin-rogers-dbt , this approach worked for dbt-teradata. Thanks for the suggestion.

But we are trying to understand what changes in the dbt-core causing this error.
We did not find any information regarding this in readme or changelog.

[tallamohan]

@dataders , I made the changes you proposed in Teradata/dbt-teradata@416189d.

But still shows the same error

Below is the complete backtrace:

01:14:22  Running with dbt=1.7.1
01:14:22  dbt version: 1.7.1
01:14:22  python version: 3.9.12
01:14:22  python path: C:\Users\mt255026\AppData\Local\Programs\Python\Python39\python.exe
01:14:22  os info: Windows-10-10.0.19045-SP0
01:14:22  Encountered an error:
non-default argument 'schema' follows default argument
01:14:22  Traceback (most recent call last):
  File "C:\Users\mt255026\AppData\Local\Programs\Python\Python39\lib\site-packages\dbt\cli\requires.py", line 90, in wrapper
    result, success = func(*args, **kwargs)
  File "C:\Users\mt255026\AppData\Local\Programs\Python\Python39\lib\site-packages\dbt\cli\requires.py", line 75, in wrapper
    return func(*args, **kwargs)
  File "C:\Users\mt255026\AppData\Local\Programs\Python\Python39\lib\site-packages\dbt\cli\main.py", line 444, in debug
    results = task.run()
  File "C:\Users\mt255026\AppData\Local\Programs\Python\Python39\lib\site-packages\dbt\task\debug.py", line 123, in run
    load_profile_status: SubtaskStatus = self._load_profile()
  File "C:\Users\mt255026\AppData\Local\Programs\Python\Python39\lib\site-packages\dbt\task\debug.py", line 212, in _load_profile
    profile: Profile = Profile.render(
  File "C:\Users\mt255026\AppData\Local\Programs\Python\Python39\lib\site-packages\dbt\config\profile.py", line 436, in render
    return cls.from_raw_profiles(
  File "C:\Users\mt255026\AppData\Local\Programs\Python\Python39\lib\site-packages\dbt\config\profile.py", line 401, in from_raw_profiles       
    return cls.from_raw_profile_info(
  File "C:\Users\mt255026\AppData\Local\Programs\Python\Python39\lib\site-packages\dbt\config\profile.py", line 355, in from_raw_profile_info   
    credentials: Credentials = cls._credentials_from_profile(
  File "C:\Users\mt255026\AppData\Local\Programs\Python\Python39\lib\site-packages\dbt\config\profile.py", line 165, in _credentials_from_profile
    cls = load_plugin(typename)
  File "C:\Users\mt255026\AppData\Local\Programs\Python\Python39\lib\site-packages\dbt\adapters\factory.py", line 212, in load_plugin
    return FACTORY.load_plugin(name)
  File "C:\Users\mt255026\AppData\Local\Programs\Python\Python39\lib\site-packages\dbt\adapters\factory.py", line 58, in load_plugin
    mod: Any = import_module("." + name, "dbt.adapters")
  File "C:\Users\mt255026\AppData\Local\Programs\Python\Python39\lib\importlib\__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 850, in exec_module
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "c:\dbt-teradata\dbt\adapters\teradata\__init__.py", line 1, in <module>
    from dbt.adapters.teradata.connections import TeradataConnectionManager
  File "c:\dbt-teradata\dbt\adapters\teradata\connections.py", line 17, in <module>
    class TeradataCredentials(Credentials):
  File "C:\Users\mt255026\AppData\Local\Programs\Python\Python39\lib\dataclasses.py", line 1021, in dataclass
    return wrap(cls)
  File "C:\Users\mt255026\AppData\Local\Programs\Python\Python39\lib\dataclasses.py", line 1013, in wrap
    return _process_class(cls, init, repr, eq, order, unsafe_hash, frozen)
  File "C:\Users\mt255026\AppData\Local\Programs\Python\Python39\lib\dataclasses.py", line 927, in _process_class
    _init_fn(flds,
  File "C:\Users\mt255026\AppData\Local\Programs\Python\Python39\lib\dataclasses.py", line 504, in _init_fn
    raise TypeError(f'non-default argument {f.name!r} '
TypeError: non-default argument 'schema' follows default argument```

[mikealfare]

Folks, I updated the materialized view section of this discussion. Please let me know if this is helpful, and if there's anything that I'm missing that would be useful to add. Thanks!

[jwills]

@mikealfare that would be for 1.8, right? Or is it coming earlier? If it's 1.8, how does it interact with #9171 ?

[mikealfare]

This is for 1.7. We'll cover how it interacts with 1.8 in #9171, but I'm not sure how much of an impact materialized views will have in that context.

[aosingh]

@dataders @dbeatty10

Starting with 1.7, the catalog.json generated is empty. I am debugging this problem. Any pointers? Generation of the catalog works fine with dbt-core 1.6.x

We have a bug oracle/dbt-oracle#125 reported by a user

[salimmoulouel]

fal-ai/dbt-fal#919
There's my PR i tested it with a python script supposed to move file to another gcs bucket after dbt run, it works fine