Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

BigQuery Labels #1964

Merged
merged 6 commits into from
Dec 13, 2019
Merged

BigQuery Labels #1964

merged 6 commits into from
Dec 13, 2019

Conversation

kconvey
Copy link
Contributor

@kconvey kconvey commented Nov 27, 2019

Implement BigQuery labels as a configurable option for the BQ adapter.

Also cleaned up Jinja whitespace a bit around 'create or replace' and 'OPTIONS', as there was a pretty huge block of empty lines in the query ultimately submitted to BQ.

#1942

@cla-bot cla-bot bot added the cla:yes label Nov 27, 2019
Copy link
Contributor

@drewbanin drewbanin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Couple of quick comments here!

Are you also able to update a BQ integration test to set these labels? It might be tricky to confirm that the labels are being set correctly in the resulting table/view; I'm mostly just interested in adding a smoke test that makes sure dbt doesn't fail really hard if labels are present. Maybe you could add the config somewhere around here? https://github.com/fishtown-analytics/dbt/blob/e51c942e91a94936f68f2965963d3b46f1257658/test/integration/022_bigquery_test/models/table_model.sql#L4

@kconvey
Copy link
Contributor Author

kconvey commented Dec 2, 2019

Added an integration test and incorporated your suggestions @drewbanin

@drewbanin
Copy link
Contributor

thanks @kconvey! I kicked off the tests here. Will follow up when they finish running :)

@kconvey
Copy link
Contributor Author

kconvey commented Dec 3, 2019

@drewbanin May need you to kick the integration tests off again. Made some tweaks to (hopefully) fix the integration test I added and add label support for views.

@drewbanin
Copy link
Contributor

ok - tests should be running again now :)

Unsure if you saw this comment but it looks like labels can also be supplied without values as "tags". Sounds like that pattern will work just fine with this implementation.

I need to do some smoke testing here, but largely, this is looking very good to me :D

@kconvey
Copy link
Contributor Author

kconvey commented Dec 4, 2019

@drewbanin Did see that comment and I think the way that works in this implementation is that, since some labels have values, and yml dicts require that all keys have values, you have to set the empty string as the value for a tag when configuring in dbt.

ex.

models:
  fishtown:
    materialized: table
    labels: {test_tag: '', test_label: 'test_value'}

This is consistent with the bq ddl syntax for tags:

ALTER TABLE mydataset.mytable
 SET OPTIONS (
   labels=[("tag1", ""), ("tag2", "")]
 )

(Taken from https://cloud.google.com/bigquery/docs/adding-labels#adding_a_tag)

Copy link
Contributor

@drewbanin drewbanin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I kicked off the tests here - this PR will be good to merge when they're all passing. Thanks for your contribution @kconvey!!

@drewbanin
Copy link
Contributor

merging this - thanks again :)

@drewbanin drewbanin merged commit fea06c6 into dbt-labs:dev/0.15.1 Dec 13, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants