Skip to content

Commit

Permalink
dbt utils 0.8.3 (#534)
Browse files Browse the repository at this point in the history 
* Fix/timestamp withought timezone (#458)

* timestamp and changelog updates

* changelog fix

* Add context for why change to no timezone

Co-authored-by: Joel Labes <joel.labes@dbtlabs.com>

* also ignore dbt_packages (#463)

* also ignore dbt_packages

* Update CHANGELOG.md

* Update CHANGELOG.md

Co-authored-by: Joel Labes <joel.labes@dbtlabs.com>

* date_spine: transform comment to jinja (#462)

* Have union_relations raise exception when include parameter results in no columns (#473)

* Raise exception if no columns in column_superset

* Add relation names to compiler error message

* Add `union_relations` fix to changelog

* Added case for handling postgres foreign tables... (#476)

* Add link for fewer_rows_than schema test in docs (#465)

* Added case for handling postgres foreign tables (tables which are external to current database and are imported into current database from remote data stores by using Foreign Data Wrappers functionallity).

* Reworked getting of postges table_type.

* Added needed changes to CHANGELOG.

Co-authored-by: José Coto <jlcoto@users.noreply.github.com>
Co-authored-by: Taras Stetsiak <tstetsiak@health-union.com>

* Enhance usability of star macro by only generating column aliases when prefix and/or suffix is specified (#468)

* The star macro should only produce column aliases when there is either a prefix or suffix specified.

* Enhanced the readme for the star macro.

* Add new integration test

Co-authored-by: Nick Perrott <nperrott@roiti.com>
Co-authored-by: Josh Elston-Green
Co-authored-by: Joel Labes <joel.labes@dbtlabs.com>

* fix: extra brace typo in insert_by_period_materialization (#480)

* Support quoted column names in sequential_values test (#479)

* Add any value (#501)

* Add link for fewer_rows_than schema test in docs (#465)

* Update get_query_results_as_dict example to demonstrate accessing columnar results as dictionary values (#474)

* Update get_qu
ery_results_as_dict example to demonstrate accessing columnar results as dictionary values

* Use slugify in example

* Fix slugify example with dbt_utils. package prefix

Co-authored-by: Elize Papineau <elize.papineau@dbtlabs.com>

* Add note about not_null_where deprecation to Readme (#477)

* Add note about not_null_where deprecation to Readme

* Add docs to unique_where test

* Update pull_request_template.md to reference `main` vs `master` (#496)

* Correct coalesce -> concatenation typo (#495)

* add any_value cross-db macro

* Missing colon in test

* Update CHANGELOG.md

Co-authored-by: José Coto <jlcoto@users.noreply.github.com>
Co-authored-by: Elize Papineau <elizepapineau@gmail.com>
Co-authored-by: Elize Papineau <elize.papineau@dbtlabs.com>
Co-authored-by: Joe Ste.Marie <stemarie.joe@gmail.com>
Co-authored-by: Niall Woodward <niall@niallrees.com>

* Fix changelog

* Second take at fixing pivot to allow single quotes (#503)

* fix pivot : in pivoted column value, single quote must be escaped (on postgresql)
else ex. syntax error near : when color = 'blue's'

* patched expected

* single quote escape : added dispatched version of the macro to support bigquery & snowflake

* second backslash to escape in Jinja, change case of test file columns

Let's see if other databases allow this

* explicitly list columns to compare

* different tests for snowflake and others

* specific comparison seed

* Don't quote identifiers for apostrophe, to avoid BQ and SF problems

* Whitespace management for macros

* Update CHANGELOG.md

Co-authored-by: Marc Dutoo <marc.dutoo@gmail.com>

* Add bool or cross db (#504)

* Create bool_or cross-db func

* Forgot a comma

* Update CHANGELOG.md

* Code review tweaks

* Fix union_relations error when no include/exclude provided (#509)

* Update CHANGELOG.md

* Add _is_ephemeral test to get_column_values (#518)

* Add _is_ephemeral test

Co-authored-by: Elize Papineau <elize.papineau@dbtlabs.com>

* Add deduplication macro (#512)

* Update README.md

* Mutually excl range examples in disclosure triangle

* Fix union_relations error when no include/exclude provided

* Fix union_relations error when no include/exclude provided (#509)

* Update CHANGELOG.md

* Add dedupe macro

* Add test for dedupe macro

* Add documentation to README

* Add entry to CHANGELOG

* Implement review

* Typed materialized views as views (#525)

* Typed materialized views as views

* Update get_relations_by_pattern.sql

* Moving fix from get_tables_by_pattern_sql

reverting changes to this file to add a fix to the macro get_tables_by_pattern_sql

* removing quoting from table_type

removing quoting from table_type as this was causing an error when calling this macro within get_tables_by_pattern_sql

* calling get_table_types_sql for materialized views

calling get_table_types_sql macro to handle materialized views in sources.

* Add `alias` argument to `deduplicate` macro (#526)

* Add `alias` argument to `deduplicate

* Test `alias` argument

* Rename `alias` to `relation_alias`

* Fix/use generic test naming style instead of schema test (#521)

* Updated Rreferences to 'schema test' in README along with small improvements to test descriptions.  Updates were also carried out in folder structure and integration README

* Updated references to 'schema test' in Changelog

* updated changelog with changes to documentation and fproject file structure

* Apply suggestions from code review

Update macro descriptions to be "asserts that"

* Update CHANGELOG.md

* Update README.md

Co-authored-by: Joel Labes <joel.labes@dbtlabs.com>

* Remove extraneous whitespace (#529)

* rm whitespace from date_trunc

* datediff

* rm uncessary whitespace control

* change log

* fix CHANGELOG

* address comments

* Feature/add listagg macro (#530)

* Update README.md

* Mutually excl range examples in disclosure triangle

* Fix union_relations error when no include/exclude provided

* Fix union_relations error when no include/exclude provided (#509)

* Update CHANGELOG.md

* Add to_condition to relationships where

* very minor nit - update "an new" to "a new" (#519)

* add quoting to split_part (#528)

* add quoting to split_part

* update docs for split_part

* typo

* corrected readme syntax

* revert and update to just documentation

* add new line

* Update README.md

* Update README.md

* Update README.md

Co-authored-by: Joel Labes <joel.labes@dbtlabs.com>

* add macro to get columns (#516)

* add macro to get columns

* star macro should use get_columns

* add adapter.

* swap adapter for dbt_utils

Co-authored-by: Joel Labes <joel.labes@dbtlabs.com>

* update documentation

* add output_lower arg

* update name to get_filtered_columns_in_relation from get_columns

* add tests

* forgot args

* too much whitespace removal

    -----------
    Actual:
    -----------
    --->"field_3"as "test_field_3"<---

    -----------
    Expected:
    -----------
    --->"field_3" as "test_field_3"<---

* didnt mean to move a file that i did not create. moving things back.

* remove lowercase logic

* limit_zero

Co-authored-by: Joel Labes <joel.labes@dbtlabs.com>

* Add listagg macro and integration test

* remove type in listagg macro

* updated integration test

* Add redshift to listagg macro

* remove redshift listagg

* explicitly named group by column

* updated default values

* Updated example to use correct double vs. single quotes

* whitespace control

* Added redshift specific macro

* Remove documentation

* Update integration test so less likely to accidentally work

Co-authored-by: Joel Labes <joel.labes@dbtlabs.com>

* default everything but measure to none

* added limit functionality for other dbs

* syntax bug for postgres

* update redshift macro

* fixed block def control

* Fixed bug in redshift

* Bug fix redshift

* remove unused group_by arg

* Added additional test without order by col

* updated to regex replace

* typo

* added more integration_tests

* attempt to make redshift less complicated

* typo

* update redshift

* replace to substr

* More explicit versions with added complexity

* handle special characters

Co-authored-by: Joel Labes <joel.labes@dbtlabs.com>
Co-authored-by: Jamie Rosenberg <james.rosenberg@canva.com>
Co-authored-by: Pat Kearns <pat.kearns@fishtownanalytics.com>

* patch default behaviour in get_column_values (#533)

* Update changelog, add missing quotes around get_table_types_sql

* rm whitespace

Co-authored-by: Joe Markiewicz <74217849+fivetran-joemarkiewicz@users.noreply.github.com>
Co-authored-by: Anders <swanson.anders@gmail.com>
Co-authored-by: Mikaël Simarik <mikael.simarik@gmail.com>
Co-authored-by: Graham Wetzler <graham@wetzler.dev>
Co-authored-by: Taras <32882370+Aesthet@users.noreply.github.com>
Co-authored-by: José Coto <jlcoto@users.noreply.github.com>
Co-authored-by: Taras Stetsiak <tstetsiak@health-union.com>
Co-authored-by: nickperrott <46330920+nickperrott@users.noreply.github.com>
Co-authored-by: Nick Perrott <nperrott@roiti.com>
Co-authored-by: Ted Conbeer <tconbeer@users.noreply.github.com>
Co-authored-by: Armand Duijn <armandduijn@users.noreply.github.com>
Co-authored-by: Elize Papineau <elizepapineau@gmail.com>
Co-authored-by: Elize Papineau <elize.papineau@dbtlabs.com>
Co-authored-by: Joe Ste.Marie <stemarie.joe@gmail.com>
Co-authored-by: Niall Woodward <niall@niallrees.com>
Co-authored-by: Marc Dutoo <marc.dutoo@gmail.com>
Co-authored-by: Judah Rand <17158624+judahrand@users.noreply.github.com>
Co-authored-by: Luis Leon <98919783+luisleon90@users.noreply.github.com>
Co-authored-by: Brid Moynihan <moynihanbridgit@gmail.com>
Co-authored-by: SunriseLong <44146580+SunriseLong@users.noreply.github.com>
Co-authored-by: Grace Goheen <53586774+graciegoheen@users.noreply.github.com>
Co-authored-by: Jamie Rosenberg <james.rosenberg@canva.com>
Co-authored-by: Pat Kearns <pat.kearns@fishtownanalytics.com>
Co-authored-by: James McNeill <55981540+jpmmcneill@users.noreply.github.com>
  • Loading branch information
@joellabes @fivetran-joemarkiewicz
@dataders @Kamitenshi @grahamwetzler @Aesthet @jlcoto @nickperrott @tconbeer @armandduijn @epapineau @joestemarie @NiallRees @mdutoo @judahrand @luisleon90 @b-moynihan @SunriseLong @graciegoheen @JamieRosenberg-canva @patkearns10 @jpmmcneill
25 people authored Apr 7, 2022
1 parent b33df57 commit 37199d3
Show file tree
Hide file tree
Showing 41 changed files with 394 additions and 74 deletions.
47 changes: 36 additions & 11 deletions CHANGELOG.md
View file
Original file line number Diff line number Diff line change
@@ -1,9 +1,35 @@
# dbt-utils v0.8.3
## New features
- A macro for deduplicating data, `deduplicate()` ([#335](https://github.com/dbt-labs/dbt-utils/issues/335), [#512](https://github.com/dbt-labs/dbt-utils/pull/512))
- A cross-database implementation of `listagg()` ([#530](https://github.com/dbt-labs/dbt-utils/pull/530))
- A new macro to get the columns in a relation as a list, `get_filtered_columns_in_relation()`. This is similar to the `star()` macro, but creates a Jinja list instead of a comma-separated string. ([#516](https://github.com/dbt-labs/dbt-utils/pull/516))

## Fixes
- `get_column_values()` once more raises an error when the model doesn't exist and there is no default provided ([#531](https://github.com/dbt-labs/dbt-utils/issues/531), [#533](https://github.com/dbt-labs/dbt-utils/pull/533))
- `get_column_values()` raises an error when used with an ephemeral model, instead of getting stuck in a compilation loop ([#358](https://github.com/dbt-labs/dbt-utils/issues/358), [#518](https://github.com/dbt-labs/dbt-utils/pull/518))
- BigQuery materialized views work correctly with `get_relations_by_pattern()` ([#525](https://github.com/dbt-labs/dbt-utils/pull/525))

## Quality of life
- Updated references to 'schema test' in project file structure and documentation ([#485](https://github.com/dbt-labs/dbt-utils/issues/485), [#521](https://github.com/dbt-labs/dbt-utils/pull/521))
- `date_trunc()` and `datediff()` default macros now have whitespace control to assist with linting and readability [#529](https://github.com/dbt-labs/dbt-utils/pull/529)
- `star()` no longer raises an error during SQLFluff linting ([#506](https://github.com/dbt-labs/dbt-utils/issues/506), [#532](https://github.com/dbt-labs/dbt-utils/pull/532))

## Contributors:
- [@judahrand](https://github.com/judahrand) (#512)
- [@b-moynihan](https://github.com/b-moynihan) (#521)
- [@sunriselong](https://github.com/sunriselong) (#529)
- [@jpmmcneill](https://github.com/jpmmcneill) (#533)
- [@KamranAMalik](https://github.com/KamranAMalik) (#532)
- [@graciegoheen](https://github.com/graciegoheen) (#530)
- [@luisleon90](https://github.com/luisleon90) (#525)
- [@epapineau](https://github.com/epapineau) (#518)
- [@patkearns10](https://github.com/patkearns10) (#516)

# dbt-utils v0.8.2
## Fixes
- Fix union_relations error from [#473](https://github.com/dbt-labs/dbt-utils/pull/473) when no include/exclude parameters are provided ([#505](https://github.com/dbt-labs/dbt-utils/issues/505), [#509](https://github.com/dbt-labs/dbt-utils/pull/509))

# dbt-utils v0.8.1

## New features
- A cross-database implementation of `any_value()` ([#497](https://github.com/dbt-labs/dbt-utils/issues/497), [#501](https://github.com/dbt-labs/dbt-utils/pull/501))
- A cross-database implementation of `bool_or()` ([#504](https://github.com/dbt-labs/dbt-utils/pull/504))
Expand All @@ -29,10 +55,9 @@
- [armandduijn](https://github.com/armandduijn) (#479)
- [mdutoo](https://github.com/mdutoo) (#503)


# dbt-utils v0.8.0
## 🚨 Breaking changes
- dbt ONE POINT OH is here! This version of dbt-utils requires _any_ version (minor and patch) of v1, which means far less need for compatibility releases in the future.
- dbt ONE POINT OH is here! This version of dbt-utils requires _any_ version (minor and patch) of v1, which means far less need for compatibility releases in the future.
- The partition column in the `mutually_exclusive_ranges` test is now always called `partition_by_col`. This enables compatibility with `--store-failures` when multiple columns are concatenated together. If you have models built on top of the failures table, update them to reflect the new column name. ([#423](https://github.com/dbt-labs/dbt-utils/issues/423), [#430](https://github.com/dbt-labs/dbt-utils/pull/430))

## Contributors:
Expand Down Expand Up @@ -87,12 +112,12 @@

## Features

- Add `not_null_proportion` schema test that allows the user to specify the minimum (`at_least`) tolerated proportion (e.g., `0.95`) of non-null values ([#411](https://github.com/dbt-labs/dbt-utils/pull/411))
- Add `not_null_proportion` generic test that allows the user to specify the minimum (`at_least`) tolerated proportion (e.g., `0.95`) of non-null values ([#411](https://github.com/dbt-labs/dbt-utils/pull/411))


## Under the hood
- Allow user to provide any case type when defining the `exclude` argument in `dbt_utils.star()` ([#403](https://github.com/dbt-labs/dbt-utils/pull/403))
- Log whole row instead of just column name in 'accepted_range' schema test to allow better visibility into failures ([#413](https://github.com/dbt-labs/dbt-utils/pull/413))
- Log whole row instead of just column name in 'accepted_range' generic test to allow better visibility into failures ([#413](https://github.com/dbt-labs/dbt-utils/pull/413))
- Use column name to group in 'get_column_values ' to allow better cross db functionality ([#407](https://github.com/dbt-labs/dbt-utils/pull/407))

# dbt-utils v0.7.1
Expand Down Expand Up @@ -149,7 +174,7 @@ If you were relying on the position to match up your optional arguments, this ma
## Features
* Add new argument, `order_by`, to `get_column_values` (code originally in [#289](https://github.com/fishtown-analytics/dbt-utils/pull/289/) from [@clausherther](https://github.com/clausherther), merged via [#349](https://github.com/fishtown-analytics/dbt-utils/pull/349/))
* Add `slugify` macro, and use it in the pivot macro. :rotating_light: This macro uses the `re` module, which is only available in dbt v0.19.0+. As a result, this feature introduces a breaking change. ([#314](https://github.com/fishtown-analytics/dbt-utils/pull/314))
* Add `not_null_proportion` schema test that allows the user to specify the minimum (`at_least`) tolerated proportion (e.g., `0.95`) of non-null values
* Add `not_null_proportion` generic test that allows the user to specify the minimum (`at_least`) tolerated proportion (e.g., `0.95`) of non-null values

## Under the hood
* Update the default implementation of concat macro to use `||` operator ([#373](https://github.com/fishtown-analytics/dbt-utils/pull/314) from [@ChristopheDuong](https://github.com/ChristopheDuong)). Note this may be a breaking change for adapters that support `concat()` but not `||`, such as Apache Spark.
Expand All @@ -160,18 +185,18 @@ If you were relying on the position to match up your optional arguments, this ma

## Fixes

- make `sequential_values` schema test use `dbt_utils.type_timestamp()` to allow for compatibility with db's without timestamp data type. [#376](https://github.com/fishtown-analytics/dbt-utils/pull/376) from [@swanderz](https://github.com/swanderz)
- make `sequential_values` generic test use `dbt_utils.type_timestamp()` to allow for compatibility with db's without timestamp data type. [#376](https://github.com/fishtown-analytics/dbt-utils/pull/376) from [@swanderz](https://github.com/swanderz)

# dbt-utils v0.6.5
## Features
* Add new `accepted_range` test ([#276](https://github.com/fishtown-analytics/dbt-utils/pull/276) [@joellabes](https://github.com/joellabes))
* Make `expression_is_true` work as a column test (code originally in [#226](https://github.com/fishtown-analytics/dbt-utils/pull/226/) from [@elliottohara](https://github.com/elliottohara), merged via [#313](https://github.com/fishtown-analytics/dbt-utils/pull/313/))
* Add new schema test, `not_accepted_values` ([#284](https://github.com/fishtown-analytics/dbt-utils/pull/284) [@JavierMonton](https://github.com/JavierMonton))
* Add new generic test, `not_accepted_values` ([#284](https://github.com/fishtown-analytics/dbt-utils/pull/284) [@JavierMonton](https://github.com/JavierMonton))
* Support a new argument, `zero_length_range_allowed` in the `mutually_exclusive_ranges` test ([#307](https://github.com/fishtown-analytics/dbt-utils/pull/307) [@zemekeneng](https://github.com/zemekeneng))
* Add new schema test, `sequential_values` ([#318](https://github.com/fishtown-analytics/dbt-utils/pull/318), inspired by [@hundredwatt](https://github.com/hundredwatt))
* Add new generic test, `sequential_values` ([#318](https://github.com/fishtown-analytics/dbt-utils/pull/318), inspired by [@hundredwatt](https://github.com/hundredwatt))
* Support `quarter` in the `postgres__last_day` macro ([#333](https://github.com/fishtown-analytics/dbt-utils/pull/333/files) [@seunghanhong](https://github.com/seunghanhong))
* Add new argument, `unit`, to `haversine_distance` ([#340](https://github.com/fishtown-analytics/dbt-utils/pull/340) [@bastienboutonnet](https://github.com/bastienboutonnet))
* Add new schema test, `fewer_rows_than` (code originally in [#221](https://github.com/fishtown-analytics/dbt-utils/pull/230/) from [@dmarts](https://github.com/dmarts), merged via [#343](https://github.com/fishtown-analytics/dbt-utils/pull/343/))
* Add new generic test, `fewer_rows_than` (code originally in [#221](https://github.com/fishtown-analytics/dbt-utils/pull/230/) from [@dmarts](https://github.com/dmarts), merged via [#343](https://github.com/fishtown-analytics/dbt-utils/pull/343/))

## Fixes
* Handle booleans gracefully in the unpivot macro ([#305](https://github.com/fishtown-analytics/dbt-utils/pull/305) [@avishalom](https://github.com/avishalom))
Expand Down Expand Up @@ -245,7 +270,7 @@ enabling users of community-supported database plugins to add or override macro
specific to their database ([#267](https://github.com/fishtown-analytics/dbt-utils/pull/267))
* Use `add_ephemeral_prefix` instead of hard-coding a string literal, to support
database adapters that use different prefixes ([#267](https://github.com/fishtown-analytics/dbt-utils/pull/267))
* Implement a quote_columns argument in the unique_combination_of_columns schema test ([#270](https://github.com/fishtown-analytics/dbt-utils/pull/270) [@JoshuaHuntley](https://github.com/JoshuaHuntley))
* Implement a quote_columns argument in the unique_combination_of_columns generic test ([#270](https://github.com/fishtown-analytics/dbt-utils/pull/270) [@JoshuaHuntley](https://github.com/JoshuaHuntley))

## Quality of life
* Remove deprecated macros `get_tables_by_prefix` and `union_tables` ([#268](https://github.com/fishtown-analytics/dbt-utils/pull/268))
Expand Down
Loading

0 comments on commit 37199d3

Please sign in to comment.