From 634dee72a1e3a37b9cd2875341e66e1051f3ea2f Mon Sep 17 00:00:00 2001 From: Jeremy Cohen Date: Thu, 22 Jun 2023 14:44:51 +0200 Subject: [PATCH 1/5] Updates to var, global macro resolution rules for package resources --- website/docs/docs/build/custom-aliases.md | 8 +++++++ website/docs/docs/build/custom-databases.md | 8 +++++++ website/docs/docs/build/custom-schemas.md | 10 +++++++- website/docs/docs/build/project-variables.md | 24 +++++++++++++++---- .../versions/01-upgrading-to-v1.6.md | 4 +++- .../reference/dbt-jinja-functions/dispatch.md | 18 ++++++++++++++ 6 files changed, 65 insertions(+), 7 deletions(-) diff --git a/website/docs/docs/build/custom-aliases.md b/website/docs/docs/build/custom-aliases.md index 9876f534f8f..589d64f8510 100644 --- a/website/docs/docs/build/custom-aliases.md +++ b/website/docs/docs/build/custom-aliases.md @@ -114,6 +114,14 @@ The default implementation of `generate_alias_name` simply uses the supplied `al + + +### Managing different behaviors across packages + +See docs on macro `dispatch`: ["Managing different global overrides across packages"](/reference/dbt-jinja-functions/dispatch) + + + ### Caveats #### Ambiguous database identifiers diff --git a/website/docs/docs/build/custom-databases.md b/website/docs/docs/build/custom-databases.md index 3df3d705837..300fd3147f1 100644 --- a/website/docs/docs/build/custom-databases.md +++ b/website/docs/docs/build/custom-databases.md @@ -87,6 +87,14 @@ The default implementation of `generate_database_name` simply uses the supplied + + +### Managing different behaviors across packages + +See docs on macro `dispatch`: ["Managing different global overrides across packages"](/reference/dbt-jinja-functions/dispatch) + + + ## Considerations ### BigQuery diff --git a/website/docs/docs/build/custom-schemas.md b/website/docs/docs/build/custom-schemas.md index 156d2f50368..2c72bec12c6 100644 --- a/website/docs/docs/build/custom-schemas.md +++ b/website/docs/docs/build/custom-schemas.md @@ -178,7 +178,7 @@ The following context methods _are_ available in the `generate_schema_name` macr | Other macros in your project | Macro | ✅ | | Other macros in your packages | Macro | ✅ | -#### Which vars are available in generate_schema_name? +### Which vars are available in generate_schema_name? @@ -190,6 +190,14 @@ for more information on these changes. Globally-scoped variables and variables defined on the command line with [--vars](/docs/build/project-variables) are accessible in the `generate_schema_name` context. + + +### Managing different behaviors across packages + +See docs on macro `dispatch`: ["Managing different global overrides across packages"](/reference/dbt-jinja-functions/dispatch) + + + ## Managing environments In the `generate_schema_name` macro examples shown above, the `target.name` context variable is used to change the schema name that dbt generates for models. If the `generate_schema_name` macro in your project uses the `target.name` context variable, you must additionally ensure that your different dbt environments are configured appropriately. While you can use any naming scheme you'd like, we typically recommend: diff --git a/website/docs/docs/build/project-variables.md b/website/docs/docs/build/project-variables.md index a84bbcb36a9..03d1cca43b2 100644 --- a/website/docs/docs/build/project-variables.md +++ b/website/docs/docs/build/project-variables.md @@ -86,18 +86,32 @@ You can find more information on defining dictionaries with YAML [here](https:// ### Variable precedence -Variables defined with the `--vars` command line argument override variables -defined in the `dbt_project.yml` file. They are globally scoped and will be -accessible to all packages included in the project. +Variables defined with the `--vars` command line argument override variables defined in the `dbt_project.yml` file. They are globally scoped and accessible to the root project and all installed packages. The order of precedence for variable declaration is as follows (highest priority first): + + 1. The variables defined on the command line with `--vars`. -3. The package-scoped variable declaration in the `dbt_project.yml` file -2. The global variable declaration in the `dbt_project.yml` file. +2. The package-scoped variable declaration in the root `dbt_project.yml` file +3. The global variable declaration in the root `dbt_project.yml` file. +4. If this node is defined in a package: variable declarations in that package's `dbt_project.yml` file +5. The variable's default argument (if one is provided). + + + + + +1. The variables defined on the command line with `--vars`. +2. The package-scoped variable declaration in the root `dbt_project.yml` file +3. The global variable declaration in the root `dbt_project.yml` file. 4. The variable's default argument (if one is provided). + + If dbt is unable to find a definition for a variable after checking these four places, then a compilation error will be raised. +**Note:** Variable scope is based on the node ultimately using that variable. Imagine the case where a model defined in the root project is calling a macro defined in an installed package. That macro, in turn, uses the value of a variable. The variable will be resolved based on the _root project's_ scope, rather than the package's scope. + diff --git a/website/docs/guides/migration/versions/01-upgrading-to-v1.6.md b/website/docs/guides/migration/versions/01-upgrading-to-v1.6.md index 4f7ddc2fd8a..865c6d3ba49 100644 --- a/website/docs/guides/migration/versions/01-upgrading-to-v1.6.md +++ b/website/docs/guides/migration/versions/01-upgrading-to-v1.6.md @@ -31,4 +31,6 @@ dbt Labs is committed to providing backward compatibility for all versions 1.x, ### Quick hits -**Coming Soon** \ No newline at end of file +More consistency and flexibility around packages! Resources defined in a package will respect variable and global macro definitions within the scope of that package. +- `vars` defined in a package's `dbt_project.yml` are now available in the resolution order when compiling nodes in that package, though CLI `--vars` and the root project's `vars` will still take precedence. See ["Variable Precedence"](/docs/build/project-variables#variable-precedence) for details. +- `generate_x_name` macros (defining custom rules for database, schema, alias naming) follow the same pattern as other "global" macros for package-scoped overrides. See [macro dispatch](/reference/dbt-jinja-functions/dispatch) for an overview of the patterns that are possible. diff --git a/website/docs/reference/dbt-jinja-functions/dispatch.md b/website/docs/reference/dbt-jinja-functions/dispatch.md index 60938361960..169971e782a 100644 --- a/website/docs/reference/dbt-jinja-functions/dispatch.md +++ b/website/docs/reference/dbt-jinja-functions/dispatch.md @@ -167,6 +167,24 @@ dispatch: +### Managing different global overrides across packages + +You can override global behaviors in different ways for each project that is installed as a package. This holds true for all global macros: `generate_schema_name`, `create_table_as`, etc. When parsing or running a resource defined in a package, the definition of the global macro within that package takes precedence over the definition in the root project, because it's more specific to those resources. + +By combining package-level overrides and `dispatch`, it is possible to achieve three different patterns: + +1. **Package always wins.** As the developer of dbt models in a project that will be deployed elsewhere as a package, I want full control over the macros used to define & materialize my models. My macros should always take precedence for my models, and there should not be any way to override them. + +_Mechanism:_ Each project/package fully overrides the macro by its name, e.g. `generate_schema_name` or `create_table_as`. Do not use dispatch. + +2. **Conditional application (root project wins).** As the maintainer of one dbt project in a mesh of multiple, my team wants conditional application of these rules. When running my project standalone (in development), I want my to apply my custom behavior; but when installed as a package and deployed alongside several other projects (in production), I want the root-level project's rules to apply. + +_Mechanism:_ Each package implements its "local" override by registering a candidate for dispatch with an adapter prefix, e.g. `default__generate_schema_name` or `default__create_table_as`. Then, the root-level project can register its own candidate for dispatch (`default__generate_schema_name`), winning the default search order, or by explicitly overriding the macro by name (`generate_schema_name`). + +3. **Same rules everywhere all the time.** As a member of the data platform team responsible for consistency across teams at my organization, I want to create a "macro package" that every team can install & use. + +_Mechanism:_ Create a standalone package of candidate macros only, e.g. `default__generate_schema_name` or `default__create_table_as`. Add a [project-level `dispatch` configuration](/reference/project-configs/dispatch-config) in every project's `dbt_project.yml`. + ## For adapter plugin maintainers Most packages were initially designed to work on the four original dbt adapters. By using the `dispatch` macro and project config, it is possible to "shim" existing packages to work on other adapters, by way of third-party compatibility packages. From 0f7335663a294010cc02c3ce4b983b0f83bca794 Mon Sep 17 00:00:00 2001 From: Matt Shaver <60105315+matthewshaver@users.noreply.github.com> Date: Tue, 27 Jun 2023 16:27:00 -0400 Subject: [PATCH 2/5] Apply suggestions from code review Keeping consistent with style guide --- website/docs/docs/build/project-variables.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/website/docs/docs/build/project-variables.md b/website/docs/docs/build/project-variables.md index 03d1cca43b2..645e557468b 100644 --- a/website/docs/docs/build/project-variables.md +++ b/website/docs/docs/build/project-variables.md @@ -94,9 +94,9 @@ The order of precedence for variable declaration is as follows (highest priority 1. The variables defined on the command line with `--vars`. 2. The package-scoped variable declaration in the root `dbt_project.yml` file -3. The global variable declaration in the root `dbt_project.yml` file. +3. The global variable declaration in the root `dbt_project.yml` file 4. If this node is defined in a package: variable declarations in that package's `dbt_project.yml` file -5. The variable's default argument (if one is provided). +5. The variable's default argument (if one is provided) From b79eb721de1174534cbabfae102e4b8b70c2793f Mon Sep 17 00:00:00 2001 From: Matt Shaver <60105315+matthewshaver@users.noreply.github.com> Date: Tue, 27 Jun 2023 16:27:47 -0400 Subject: [PATCH 3/5] Apply suggestions from code review Consistency --- website/docs/docs/build/project-variables.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/website/docs/docs/build/project-variables.md b/website/docs/docs/build/project-variables.md index 645e557468b..c5acc7fc5a7 100644 --- a/website/docs/docs/build/project-variables.md +++ b/website/docs/docs/build/project-variables.md @@ -102,9 +102,9 @@ The order of precedence for variable declaration is as follows (highest priority -1. The variables defined on the command line with `--vars`. +1. The variables defined on the command line with `--vars` 2. The package-scoped variable declaration in the root `dbt_project.yml` file -3. The global variable declaration in the root `dbt_project.yml` file. +3. The global variable declaration in the root `dbt_project.yml` file 4. The variable's default argument (if one is provided). From 0d44f223495a1cb84455701fb6a37bf4a6dec4b2 Mon Sep 17 00:00:00 2001 From: Matt Shaver <60105315+matthewshaver@users.noreply.github.com> Date: Tue, 27 Jun 2023 16:43:50 -0400 Subject: [PATCH 4/5] Apply suggestions from code review Changing perspective and formatting --- .../docs/reference/dbt-jinja-functions/dispatch.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/website/docs/reference/dbt-jinja-functions/dispatch.md b/website/docs/reference/dbt-jinja-functions/dispatch.md index 169971e782a..215761913a3 100644 --- a/website/docs/reference/dbt-jinja-functions/dispatch.md +++ b/website/docs/reference/dbt-jinja-functions/dispatch.md @@ -169,21 +169,21 @@ dispatch: ### Managing different global overrides across packages -You can override global behaviors in different ways for each project that is installed as a package. This holds true for all global macros: `generate_schema_name`, `create_table_as`, etc. When parsing or running a resource defined in a package, the definition of the global macro within that package takes precedence over the definition in the root project, because it's more specific to those resources. +You can override global behaviors in different ways for each project that is installed as a package. This holds true for all global macros: `generate_schema_name`, `create_table_as`, etc. When parsing or running a resource defined in a package, the definition of the global macro within that package takes precedence over the definition in the root project because it's more specific to those resources. By combining package-level overrides and `dispatch`, it is possible to achieve three different patterns: -1. **Package always wins.** As the developer of dbt models in a project that will be deployed elsewhere as a package, I want full control over the macros used to define & materialize my models. My macros should always take precedence for my models, and there should not be any way to override them. +1. **Package always wins** — As the developer of dbt models in a project that will be deployed elsewhere as a package, You want full control over the macros used to define & materialize my models. Your macros should always take precedence for your models, and there should not be any way to override them. -_Mechanism:_ Each project/package fully overrides the macro by its name, e.g. `generate_schema_name` or `create_table_as`. Do not use dispatch. + - _Mechanism:_ Each project/package fully overrides the macro by its name, for example, `generate_schema_name` or `create_table_as`. Do not use dispatch. -2. **Conditional application (root project wins).** As the maintainer of one dbt project in a mesh of multiple, my team wants conditional application of these rules. When running my project standalone (in development), I want my to apply my custom behavior; but when installed as a package and deployed alongside several other projects (in production), I want the root-level project's rules to apply. +2. **Conditional application (root project wins)** — As the maintainer of one dbt project in a mesh of multiple, your team wants conditional application of these rules. When running your project standalone (in development), you want to apply custom behavior; but when installed as a package and deployed alongside several other projects (in production), you want the root-level project's rules to apply. -_Mechanism:_ Each package implements its "local" override by registering a candidate for dispatch with an adapter prefix, e.g. `default__generate_schema_name` or `default__create_table_as`. Then, the root-level project can register its own candidate for dispatch (`default__generate_schema_name`), winning the default search order, or by explicitly overriding the macro by name (`generate_schema_name`). + - _Mechanism:_ Each package implements its "local" override by registering a candidate for dispatch with an adapter prefix, for example, `default__generate_schema_name` or `default__create_table_as`. The root-level project can then register its own candidate for dispatch (`default__generate_schema_name`), winning the default search order or by explicitly overriding the macro by name (`generate_schema_name`). 3. **Same rules everywhere all the time.** As a member of the data platform team responsible for consistency across teams at my organization, I want to create a "macro package" that every team can install & use. -_Mechanism:_ Create a standalone package of candidate macros only, e.g. `default__generate_schema_name` or `default__create_table_as`. Add a [project-level `dispatch` configuration](/reference/project-configs/dispatch-config) in every project's `dbt_project.yml`. + - _Mechanism:_ Create a standalone package of candidate macros only, for example, `default__generate_schema_name` or `default__create_table_as`. Add a [project-level `dispatch` configuration](/reference/project-configs/dispatch-config) in every project's `dbt_project.yml`. ## For adapter plugin maintainers From b2ffb18b8a1adabae034d7b77989e5c17eb82513 Mon Sep 17 00:00:00 2001 From: Matt Shaver <60105315+matthewshaver@users.noreply.github.com> Date: Tue, 27 Jun 2023 16:44:01 -0400 Subject: [PATCH 5/5] Apply suggestions from code review --- website/docs/reference/dbt-jinja-functions/dispatch.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/reference/dbt-jinja-functions/dispatch.md b/website/docs/reference/dbt-jinja-functions/dispatch.md index 215761913a3..d615bbdb430 100644 --- a/website/docs/reference/dbt-jinja-functions/dispatch.md +++ b/website/docs/reference/dbt-jinja-functions/dispatch.md @@ -181,7 +181,7 @@ By combining package-level overrides and `dispatch`, it is possible to achieve t - _Mechanism:_ Each package implements its "local" override by registering a candidate for dispatch with an adapter prefix, for example, `default__generate_schema_name` or `default__create_table_as`. The root-level project can then register its own candidate for dispatch (`default__generate_schema_name`), winning the default search order or by explicitly overriding the macro by name (`generate_schema_name`). -3. **Same rules everywhere all the time.** As a member of the data platform team responsible for consistency across teams at my organization, I want to create a "macro package" that every team can install & use. +3. **Same rules everywhere all the time** — As a member of the data platform team responsible for consistency across teams at your organization, you want to create a "macro package" that every team can install & use. - _Mechanism:_ Create a standalone package of candidate macros only, for example, `default__generate_schema_name` or `default__create_table_as`. Add a [project-level `dispatch` configuration](/reference/project-configs/dispatch-config) in every project's `dbt_project.yml`.