Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BQ] Fix rendering for int64-type partition #3063

Closed
1 of 5 tasks
jtcohen6 opened this issue Feb 9, 2021 · 0 comments · Fixed by #3098
Closed
1 of 5 tasks

[BQ] Fix rendering for int64-type partition #3063

jtcohen6 opened this issue Feb 9, 2021 · 0 comments · Fixed by #3098
Labels
bigquery bug Something isn't working regression

Comments

@jtcohen6
Copy link
Contributor

jtcohen6 commented Feb 9, 2021

partition_by.render() was updated in v0.19.0 to handle granularity-based partitioning (week, month, year). This resulted in an regression for int64-type columns with incremental models using the insert_overwrite strategy.

https://github.com/fishtown-analytics/dbt/blob/6c6649f9129d5d108aa3b0526f634cd8f3a9d1ed/plugins/bigquery/dbt/adapters/bigquery/impl.py#L61-L65

Lines 61-62 should become:

 if self.data_type.lower() == 'int64' or (
         self.data_type.lower() == 'date' and
         self.granularity.lower() == 'day'
 ): 

Let's fix these lines and add:

  • Unit tests for partition_by.render(). We added test_parse_partition_by in v0.16.0 for parsing old partition_by configs as new dict-style configs.
  • An integration test case (022_bigquery_test): int64 + insert_overwrite

Steps to reproduce

{{ config(
    materialized = 'incremental',
    incremental_strategy = 'insert_overwrite',
    partition_by = {
        'field': 'id',
        'data_type': 'int64',
        'range': {
            'start': 0,
            'end': 10,
            'interval': 1
        }
    }
) }}

select 1 as id
$ dbt run
$ dbt run
...

Database Error in model int_table (models/int_table.sql)
  Query error: Function not found: int64_trunc at [33:34]
  compiled SQL at target/run/root_project/models/int_table.sql

v0.19.0:

when not matched by source
         and int64_trunc(DBT_INTERNAL_DEST.id, day) in unnest(dbt_partitions_for_replacement) 
        then delete

v0.18.1:

when not matched by source
         and DBT_INTERNAL_DEST.id in unnest(dbt_partitions_for_replacement) 
        then delete

Screenshots and log output

If applicable, add screenshots or log output to help explain your problem.

System information

Which database are you using dbt with?

  • postgres
  • redshift
  • bigquery
  • snowflake
  • other (specify: ____________)

The output of dbt --version:

$ dbt --version
installed version: 0.19.0
   latest version: 0.19.0

Up to date!

Plugins:
  - bigquery: 0.19.0
  - snowflake: 0.19.0
  - redshift: 0.19.0
  - postgres: 0.19.0
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bigquery bug Something isn't working regression
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant