Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add optional condition to not_null_proportion test #691

Closed
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@

## New features
- New feature to omit the `source_column_name` column on the `union_relations` macro ([#331](https://github.com/dbt-labs/dbt-utils/issues/331), [#624](https://github.com/dbt-labs/dbt-utils/pull/624))
- New feature to allow a `condition` option to `not_null_proportion` tests ([#691](https://github.com/dbt-labs/dbt-utils/pull/691))

## Fixes
- Better handling of whitespaces in the star macro ([#651](https://github.com/dbt-labs/dbt-utils/pull/651))
Expand Down
17 changes: 17 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -317,6 +317,23 @@ models:
at_least: 0.95
```

The macro accepts an optional argument `condition` that allows for asserting the proportion is not null on a subset of all records.

**Usage:**

```yaml
version: 2

models:
- name: my_model
columns:
- name: id
tests:
- dbt_utils.not_null_proportion:
at_least: 0.95
condition: "created_at > '2018-12-31'"
```

#### not_accepted_values ([source](macros/generic_tests/not_accepted_values.sql))

Asserts that there are no rows that match the given values.
Expand Down
4 changes: 4 additions & 0 deletions integration_tests/models/generic_tests/schema.yml
Original file line number Diff line number Diff line change
Expand Up @@ -146,6 +146,10 @@ seeds:
tests:
- dbt_utils.not_null_proportion:
at_least: 0.9
- dbt_utils.not_null_proportion:
at_least: 1
at_most: 1
condition: "point_9 IS NOT NULL"

models:
- name: test_recency
Expand Down
2 changes: 2 additions & 0 deletions macros/generic_tests/not_null_proportion.sql
Original file line number Diff line number Diff line change
Expand Up @@ -7,11 +7,13 @@
{% set column_name = kwargs.get('column_name', kwargs.get('arg')) %}
{% set at_least = kwargs.get('at_least', kwargs.get('arg')) %}
{% set at_most = kwargs.get('at_most', kwargs.get('arg', 1)) %}
{% set condition = kwargs.get('condition', kwargs.get('arg', '1=1')) %}

with validation as (
select
sum(case when {{ column_name }} is null then 0 else 1 end) / cast(count(*) as numeric) as not_null_proportion
from {{ model }}
where {{ condition }}
),
validation_errors as (
select
Expand Down