Variables not picked up by tests

[Limess]

Describe the bug

Compilation fails when using variables defined in dbt_project.yaml in tests.

This was working prior to the upgrade to 0.20.0

Steps To Reproduce

Variable in dbt_project.toml

vars:
  signal_dbt:
    # the number of days we are legally allowed to retain data
    tracking_event_retention_period_days: 1460

Test:

        tests:
          - not_null:
              where: 'day >= CURRENT_DATE - interval ''{{ var("tracking_event_retention_period_days") }} days'''

Expected behavior

The variable to be used from dbt_project.toml

Screenshots and log output

Running with dbt=0.20.0
Encountered an error:
Compilation Error
  Invalid test config given in models/marts/core/core.yml:
  	Required var 'tracking_event_retention_period_days' not found in config:
  Vars supplied to <Configuration> = {}
  	@: UnparsedNodeUpdate(original_file_path='model...ne)

System information

Which database are you using dbt with?

• postgres
• redshift
• bigquery
• snowflake
• other (specify: ____________)

The output of dbt --version:

installed version: 0.20.0
   latest version: 0.20.0

Up to date!

Plugins:
  - redshift: 0.20.0
  - postgres: 0.20.0

The operating system you're using:

Mac OS big Sur 11.2.3

The output of python --version:

3.8.5

[jtcohen6]

Thanks for the quick report @Limess! I was able to reproduce this locally.

Given that this works if I pass the var from the CLI:

dbt test --vars 'tracking_event_retention_period_days: 1460'

This appears to be a bug with the rendering context used for test arguments here:
https://github.com/dbt-labs/dbt/blob/f460d275baa4df8cf8713f27e198a3577d6ea71c/core/dbt/parser/schema_test_builders.py#L231

That comes from:
https://github.com/dbt-labs/dbt/blob/f460d275baa4df8cf8713f27e198a3577d6ea71c/core/dbt/parser/schemas.py#L291-L293

It looks like we're pulling in self.root_project.cli_vars there. At the same time, self.root_project.vars.vars does contain {'signal_dbt': {'tracking_event_retention_period_days': 1460}}, so I'm not sure why it's missing in the resulting context.

I'm hopeful that this is a quick fix, and something we could easily include in a patch release.

[codigo-ergo-sum]

We are experiencing the same issue when testing an upgrade to dbt .20 as it's very common behavior for us to use vars defined in dbt_project.yml in our tests.

An example of a test that is failing for us:

- dbt_utils.expression_is_true: severity: warn tags: - known_fail_or_warn expression: "parent_classification IN ('Revenue', 'Expense')" condition: "dim_financial_account_key <> {{ var('unknown_key') }}"

Sorry, it doesn't seem to be preserving the appropriate indentation, but the indentation is correct in the .yml file.

[leahwicz]

@kwigley could you please look into this and see what might have caused the regression?

[kwigley]

Thanks for the issue report @Limess!

The where config is new in 0.20.0, and it turns out that it doesn't support using configured vars in your project. This goes for other test configs added in 0.20.0 (fail_calc, error_if, warn_if, store_failures).

I think we just need to use a context that includes project defined vars when rendering the configs. Will add a fix to address this! I confirmed that var() in test arguments are still rendered properly, so this aligns with previous behavior.

[kwigley]

We are experiencing the same issue when testing an upgrade to dbt .20 as it's very common behavior for us to use vars defined in dbt_project.yml in our tests.

An example of a test that is failing for us:

- dbt_utils.expression_is_true: severity: warn tags: - known_fail_or_warn expression: "parent_classification IN ('Revenue', 'Expense')" condition: "dim_financial_account_key <> {{ var('unknown_key') }}"

Sorry, it doesn't seem to be preserving the appropriate indentation, but the indentation is correct in the .yml file.

hey @codigo-ergo-sum, I wasn't able to reproduce this. Are you getting the same compilation error when using var() with test arguments?

Also, just a helpful tip. You can use triple back-ticks (```) instead of a single back-tick (`) to preserve code formatting and indentation! Hope that helps

```
asdf code
    indent

```

[codigo-ergo-sum]

hi @kwigley , sorry, I'm not following. Can you clarify what you mean by "test arguments"? Thank you!

[kwigley]

Ya, sure thing! For "test arguments" I mean the values you pass to the test (a.k.a. the arguments you configure to pass to the test macro). As opposed to "test configs" (docs), which are also provided as properties of the test. So for the snippet you provided:

- dbt_utils.expression_is_true:
    severity: warn   # <- test config
    tags:  # <- general config
      - known_fail_or_warn
    expression: "parent_classification IN ('Revenue', 'Expense')"  # <- test argument
    condition: "dim_financial_account_key <> {{ var('unknown_key') }}"  # <- test argument

I tried to reproduce this compilation error with var() being used for test arguments like the example you provided. I unfortunately wasn't able to reproduce the issue. Let me know if you can provide any more info to debug or if I missed anything!

[codigo-ergo-sum]

@jtcohen6 - you had originally opened this ticket and I just chimed in. looks like @kwigley wasn't able to reproduce this but I still have the issue. Just checking if you are still able to reproduce this problem?

[jtcohen6]

@codigo-ergo-sum I was able to reproduce @Limess's original issue, and when I saw you post in the slack thread, I connected it with this issue. Per @kwigley's explanation, it sounds like these are subtly different.

Using dbt v0.20.0, I haven't been able to reproduce this error using the code snippet you included above. If I define a model named my_model, and include this in models/schema.yml:

version: 2
models:
  - name: my_model
    tests:
      - dbt_utils.expression_is_true:
          severity: warn
          tags:
            - known_fail_or_warn
          expression: "parent_classification IN ('Revenue', 'Expense')"
          condition: "dim_financial_account_key <> {{ var('unknown_key') }}"

And define a default value for that variable in dbt_project.yml:

vars:
  unknown_key: whatever

I can run dbt parse and dbt test without seeing any errors. That includes no parse-time errors like Required var 'unknown_key' not found in config.

I am able to yield that error if I move the call to {{ var('unknown_key') }} into the severity config under the same test. That's the cause of this original issue—test configs being rendered with a different context from test arguments—and it's what we fixed in #3646.

[codigo-ergo-sum]

Got it thanks @jtcohen6 . Now I understand @kwigley 's previous comments and the distinction between test arguments and test configs, my fault for not reading more closely. I went back and looked some more, and it turns out that in the same .yml file where I have the snippet above regarding the - dbt_utils.expression_is_true: test, I also have this:

      - name: financial_account_id
        tests:
          - not_null:
              where: "dim_financial_account_key <> {{ var('unknown_key') }}"

By commenting this out, I was able to successfully compile the .yml file in question even with the other references to {{ var('unknown_key') }} in the dbt_utils.expression_is_true` test show further above not commented out.

So my problem was also with the test configs and not with the test arguments, so I think we are good for the fix already done in #3646 . Thank you!

[kwigley]

Whoops! Merging my PR automatically closed the issue, I meant to follow up and make sure this was resolved for you. Glad #3646 will do the trick 👍