Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Claires suggestions #2

Open
wants to merge 4 commits into
base: master
Choose a base branch
from
Open

Claires suggestions #2

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
b973071
Add a gitignore
clrcrl Feb 14, 2020
0b3ddc6
Specify a project (database) for sources
clrcrl Feb 14, 2020
5535e15
Fix errant SQL
clrcrl Feb 14, 2020
10a7dd9
Use simpler incremental logic
clrcrl Feb 14, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 2 additions & 8 deletions reddit_aita/models/aita_comments.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,12 +6,6 @@ FROM {{ source('reddit_comments', '20*') }}
WHERE subreddit = 'AmItheAsshole'
AND _table_suffix > '19_'

{%- if is_incremental() -%}
{%- if execute -%}
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So I'm using this weird complex logic because one of these queries is much better for BigQuery:

  • SELECT * FROM * WHERE _table_suffix < '2019_01'
  • SELECT * FROM * WHERE _table_suffix < (SELECT date FROM ...)

One is a constant and will prune how much data is read. The other one is variable, and BigQuery doesn't optimize it as well.

I can get that as a constant by using run_query(). I have another option, but it doesn't work now (dbt-labs/dbt-core#2136)

{%- set last_stamp_sql -%}SELECT FORMAT_TIMESTAMP('%y_%mX', MAX(ts)) maxsuffix FROM {{this}}{%- endset -%}
{%- set last_stamp_result = run_query(last_stamp_sql) -%}
AND _table_suffix > "{{last_stamp_result.rows[0].get('maxsuffix')[:-1]}}"
{# Somewhere '19_01' gets transformed to '1901', unless I add an 'X' to FORMAT_TIMESTAMP() and [:-1] later. #}
{% if is_incremental() -%}
AND _table_suffix > (SELECT FORMAT_TIMESTAMP('%y_%m', MAX(ts)) from {{ this }})
{%- endif -%}
{%- endif -%}

10 changes: 2 additions & 8 deletions reddit_aita/models/aita_posts.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,12 +10,6 @@ FROM {{ source('reddit_posts', '20*') }}
WHERE subreddit = 'AmItheAsshole'
AND _table_suffix > '19_'

{%- if is_incremental() -%}
{%- if execute -%}
{%- set last_stamp_sql -%}SELECT FORMAT_TIMESTAMP('%y_%mX', MAX(ts)) maxsuffix FROM {{this}}{%- endset -%}
{%- set last_stamp_result = run_query(last_stamp_sql) -%}
AND _table_suffix > "{{last_stamp_result.rows[0].get('maxsuffix')[:-1]}}"
{# Somewhere '19_01' gets transformed to '1901', unless I add an 'X' to FORMAT_TIMESTAMP() and [:-1] later. #}
{% if is_incremental() -%}
AND _table_suffix > (SELECT FORMAT_TIMESTAMP('%y_%m', MAX(ts)) from {{ this }})
{%- endif -%}
{%- endif -%}