DBT Compile fails on 0.20.0 when referencing a macro from another package within a test.

[thisiscab]

Describe the bug

When running dbt compile after migrating to DBT 0.20.0 from DBT 0.19.1 we're hit with a Compilation Error.

Steps To Reproduce

We have two packages, one that "runs" the code and another one that acts as a "library" (in this example it's called custom_utils.

In our custom_utils we have the following macro:

{%- macro TRUE_ERROR() -%}
'error-shouldnt-happen'
{%- endmacro -%}

In our runner package, we have the following:

version: 2

models:
  - name: fct_trans_orders
    columns:
    - name: order_id
      tests:
        - unique
        - dbt_utils.expression_is_true:
            expression: "order_id not in ({{ custom_utils.TRUE_ERROR() }})"

packages:
 - package: fishtown-analytics/dbt_utils
   version: [">=0.7.0", "<0.8.0"]
 - git: "https://[censored]/dbt-packages/custom_utils.git"
   revision: update-dbt-to-dbt-v20

And finally, the dbt_projects of custom_utils:

name: 'custom_utils'
version: '1.0.0'
config-version: 2

macro-paths: ["macros"]

require-dbt-version: [">=0.20.0", "<0.21.0"]

Expected behavior

DBT should be able to compile the above and replace the template with expression: "order_id not in ('error-shouldnt-happen')" as it was previously doing in 0.19.1

Screenshots and log output

root@ea812dfe995d:/usr/src# dbt compile
Running with dbt=0.20.0
Encountered an error:
Compilation Error
  Invalid test config given in models/marts/core/documentations.yml:
        'custom_utils' is undefined
        @: UnparsedNodeUpdate(original_file_path='model...ne)

System information

Which database are you using dbt with?

• postgres
• redshift
• bigquery
• snowflake
• other (specify: ____________)

The output of dbt --version:

root@ea812dfe995d:/usr/src# dbt --version
installed version: 0.20.0
   latest version: 0.20.0

Up to date!

Plugins:
  - snowflake: 0.20.0

The operating system you're using:

(Debian 10.10)
root@ea812dfe995d:/usr/src# uname -amnrspv
Linux ea812dfe995d 5.10.25-linuxkit #1 SMP Tue Mar 23 09:27:39 UTC 2021 x86_64 GNU/Linux

The output of python --version:

root@ea812dfe995d:/usr/src# python --version
Python 3.8.11

Additional context

Is there a potential workaround that we could apply before waiting for a potential "official" patch?

[jtcohen6]

@thisiscab Thanks for opening. I'm still unable to reproduce this—the project compiles, runs, and tests just fine. That said, I'm using dbt v0.20.0 installed via Homebrew with python 3.9.5 on a Mac, so it's possible that a python/system discrepancy is at fault here.

The error message looks a lot like the one in #3564. Over the past few months, we've resolved a number of issues around missing macros and packages in the rendering context. Out of curiosity, how have you installed dbt?

[thisiscab]

That is very odd! I wouldn't have expected this type of issue to be caused by Python / OS! It does really look like the issue you're pointing to.

We have a Docker image:

# (Debian 10.10)
FROM python:3.8-buster

[REDACTED]

RUN apt-get -qq update \
    && apt-get -qq install -y --no-install-recommends postgresq

RUN pip install --upgrade pip setuptools wheel dbt-snowflake==0.20.0

[REDACTED]

And then within that Docker Image, we simply run the DBT command -- Nothing special!

[kwigley]

Hey @thisiscab, like @jtcohen6, I'm unable to recreate this as well ☹️ Is there any way you can provide redacted debug logs (logs/dbt.log) or the output of dbt -d compile where the issue occurs? I have a feeling that the compilation error you are seeing might be stemming from another issue. Thanks!

[thisiscab]

@kwigley Hello!

Here is the output that you've asked me. I've censored part of it!
https://gist.github.com/thisiscab/e324bd7ed686a28ffc166f109b044f28

I can't help but realize that I'm now noticing this following error:

On instance:
    {'config-version': 2,
     'dispatch': [{'macro_namespace': 'dbt_utils',
                   'search_order': ['custom_utils', 'dbt_utils']}],
     'macro-paths': ['macros'],
     'name': 'custom_utils',
     'project-root': '/tmp/dbt-downloads-t2fe_gd3/920a1245502f8ad634e7ed9535c83d57',
     'require-dbt-version': ['>=0.20.0', '<0.21.0'],
     'version': '1.0.0'}
     The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/dbt/main.py", line 125, in main
    results, succeeded = handle_and_check(args)
  File "/usr/local/lib/python3.8/site-packages/dbt/main.py", line 203, in handle_and_check
    task, res = run_from_args(parsed)
  File "/usr/local/lib/python3.8/site-packages/dbt/main.py", line 256, in run_from_args
    results = task.run()
  File "/usr/local/lib/python3.8/site-packages/dbt/task/deps.py", line 53, in run
    final_deps = resolve_packages(packages, self.config)
  File "/usr/local/lib/python3.8/site-packages/dbt/deps/resolver.py", line 137, in resolve_packages
    target = final[package].resolved().fetch_metadata(config, renderer)
  File "/usr/local/lib/python3.8/site-packages/dbt/deps/base.py", line 85, in fetch_metadata
    self._cached_metadata = self._fetch_metadata(project, renderer)
  File "/usr/local/lib/python3.8/site-packages/dbt/deps/git.py", line 82, in _fetch_metadata
    loaded = Project.from_project_root(path, renderer)
  File "/usr/local/lib/python3.8/site-packages/dbt/config/project.py", line 633, in from_project_root
    return partial.render(renderer)
  File "/usr/local/lib/python3.8/site-packages/dbt/config/project.py", line 289, in render
    return self.create_project(rendered)
  File "/usr/local/lib/python3.8/site-packages/dbt/config/project.py", line 312, in create_project
    raise DbtProjectError(validator_error_message(e)) from e
dbt.exceptions.DbtProjectError: Runtime Error
  at path []: Additional properties are not allowed ('dispatch' was unexpected)

I may be misunderstanding the documentation but what I have;

# dbt_project.yml of the custom_utils package

name: 'custom_utils'
version: '1.0.0'
config-version: 2

macro-paths: ["macros"]

require-dbt-version: [">=0.20.0", "<0.21.0"]

dispatch:
  - macro_namespace: 'dbt_utils'
    search_order: ['custom_utils', 'dbt_utils']

Seems valid per this documentation: https://docs.getdbt.com/reference/dbt-jinja-functions/dispatch#overriding-package-macros.

Thanks!

[kwigley]

@thisiscab thanks a bunch, will take a look. As one last sanity check, are you able to confirm all the versions of the dbt packages installed in the docker container? something like:

$ pip freeze | grep dbt
dbt-core==0.20.0
dbt-extractor==0.2.0
dbt-snowflake==0.20.0

[thisiscab]

@kwigley

root@edd5c11d72e0:/usr/src# pip freeze | grep dbt
dbt-core==0.20.0
dbt-extractor==0.2.0
dbt-snowflake==0.20.0

:)

[ahmedrad]

Hey just wanted to jump in and say I'm facing the exact same issue:

dbt is installed via pipenv

pipenv graph | grep dbt output:

dbt==0.20.2
  - dbt-bigquery [required: ==0.20.2, installed: 0.20.2]
    - dbt-core [required: ==0.20.2, installed: 0.20.2]
      - dbt-extractor [required: ==0.4.0, installed: 0.4.0]
  - dbt-core [required: ==0.20.2, installed: 0.20.2]
    - dbt-extractor [required: ==0.4.0, installed: 0.4.0]
  - dbt-postgres [required: ==0.20.2, installed: 0.20.2]
    - dbt-core [required: ==0.20.2, installed: 0.20.2]
      - dbt-extractor [required: ==0.4.0, installed: 0.4.0]
  - dbt-redshift [required: ==0.20.2, installed: 0.20.2]
    - dbt-core [required: ==0.20.2, installed: 0.20.2]
      - dbt-extractor [required: ==0.4.0, installed: 0.4.0]
    - dbt-postgres [required: ==0.20.2, installed: 0.20.2]
      - dbt-core [required: ==0.20.2, installed: 0.20.2]
        - dbt-extractor [required: ==0.4.0, installed: 0.4.0]
  - dbt-snowflake [required: ==0.20.2, installed: 0.20.2]
    - dbt-core [required: ==0.20.2, installed: 0.20.2]
      - dbt-extractor [required: ==0.4.0, installed: 0.4.0]

this is my schema test:

columns:
      - name: id
        description: >-
          The primary key for the segment event. Note: we can't
          necessarily assume this is a unique field for anything
          greater than a 24 hour window.
      - name: user_id
        description: >-
          The user_id of the user who just logged in
        tests:
          - not_null
          - relationships:
              to: source('auth', 'user')
              field: userid
              where: day < {{ dbt_utils.dateadd('day', -1, 'current_date') }}

Mac OS 11.6
python version 3.9.7

[jtcohen6]

@ahmedrad Thanks for the report! I was able to reproduce the issue using the example you've provided:

Encountered an error:
Compilation Error
  Invalid test config given in models/example/schema.yml:
  	'dbt_utils' is undefined
  	@: UnparsedNodeUpdate(original_file_path='model...ne)

The issue is that where is a test config, as opposed to a test argument, and the two have slightly different rendering contexts. Generally speaking, macros are not supported in yaml definitions for configs. This is a bit clearer with the new recommended syntax in v0.21, though the difference is still fairly subtle:

        tests:
          - not_null
          - relationships:
              to: source('auth', 'user')  # test argument
              field: userid  # test argument
              config:  # test config, must be rendered at parse time, macros other than var + env_var are not supported today
                where: "day < {{ dbt_utils.dateadd('day', -1, 'current_date') }}"

We augmented the rendering context for test configs in v0.20.1 to add support for var (#3646), but the config rendering context still does not have support for macros. I realize this is a bit of a functional regression, incidental to how we've reconciled configs for different resource types.

@gshank Could you weigh in on the difficulty and desirability of supporting arbitrary macros in the parse-time rendering context of configs, defined in yaml files (not dbt_project.yml)? Obviously, any macros that require execution-time inputs (e.g. introspective database queries) will not return usable results.

I do think your example is different from the original issue report, since expression is an argument (not config) for the dbt_utils.expression_is_true test. For better and for worse, in the past, we've supported macros in test arguments.

[Update: more thorough example and explanation in #4103]

[gshank]

We could statically parse for macros like we do for the test node macros calls. We could try to create a reusable context that contains all of the macros.

[github-actions]

This issue has been marked as Stale because it has been open for 180 days with no activity. If you would like the issue to remain open, please remove the stale label or comment on the issue, or it will be closed in 7 days.

[github-actions]

Although we are closing this issue as stale, it's not gone forever. Issues can be reopened if there is renewed community interest; add a comment to notify the maintainers.