Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Check if a snowflake column exists before altering its comment #3149

Merged
merged 9 commits into from
May 13, 2021

Conversation

elikastelein
Copy link
Contributor

@elikastelein elikastelein commented Mar 5, 2021

Resolves #3039

Description

  • This PR modifies the "alter column comment" snowflake macro by breaking each column comment into a separate DDL statement, so that we can check that the column exists before attempting to comment on it
  • This helps avoid the confusing scenario of getting a database error for a column that doesn't appear in your model's .sql file
  • I couldn't find an equivalent to comment if exists in BQ, redshift, or postgres so I did this PR for snowflake only
  • As mentioned in Improve Error Message for Persisted Comments on Columns #3039, the downside of this PR is that breaking things up into multiple DDL statements causes dbt run to take longer. I hope this is okay since persist_docs defaults to off and users who are ultra-concerned with performance would most likely have it off anyways.

Checklist

  • I have signed the CLA
  • I have run this code in development and it appears to resolve the stated issue
  • This PR includes tests, or tests are not required/relevant for this PR
  • I have updated the CHANGELOG.md and added information about my change to the "dbt next" section.

@cla-bot
Copy link

cla-bot bot commented Mar 5, 2021

Thanks for your pull request, and welcome to our community! We require contributors to sign our Contributor License Agreement and we don't seem to have your signature on file. Check out this article for more information on why we have a CLA.

In order for us to review and merge your code, please submit the Individual Contributor License Agreement form attached above above. If you have questions about the CLA, or if you believe you've received this message in error, don't hesitate to ping @drewbanin.

CLA has not been signed by users: @elikastelein

@elikastelein
Copy link
Contributor Author

I've just signed the CLA now

@kwigley
Copy link
Contributor

kwigley commented Mar 5, 2021

@cla-bot check

@cla-bot cla-bot bot added the cla:yes label Mar 5, 2021
@cla-bot
Copy link

cla-bot bot commented Mar 5, 2021

The cla-bot has been summoned, and re-checked this pull request!

Copy link
Contributor

@jtcohen6 jtcohen6 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice one @elikastelein, this works for me locally!

Could you add a new test case to test/integration/060_persist_docs_test for this functionality? This would look like a new models folder, models-column-missing, containing model that has a description defined on a column that does not actually exist. Then, a new test script (TestPersistDocsColumnMissing) that uses that model, runs dbt, and ensures it passes as expected. I believe BigQuery and (now) Snowflake will be able to handle this and pass, whereas Postgres and Redshift would fail. Let me know if you want any help, or more pointers to get started!

Could you also add a Changelog entry under v0.20.0 - Fixes, and add yourself to the list of contributors?

@jtcohen6
Copy link
Contributor

Hey @elikastelein, hope you're well! Just wanted to check in and see if you'd be able to add a test case for this change? More than happy to help with any pointers :)

@elikastelein
Copy link
Contributor Author

Hey @jtcohen6, thanks for the ping. Sorry about leaving this hanging. I'll get started on adding tests soon and let you know if I need any pointers

Copy link
Contributor

@jtcohen6 jtcohen6 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@elikastelein Nice work adding the test—I confirmed it fails without the change to snowflake__alter_column_comment, and succeeds with it.

This LGTM, thank you for the contribution (:

@jtcohen6 jtcohen6 merged commit fffcd3b into dbt-labs:develop May 13, 2021
@elikastelein elikastelein deleted the fix/column-comment-if-exists branch May 13, 2021 01:37
@elikastelein
Copy link
Contributor Author

@jtcohen6 thanks so much for the help, your comments were very helpful

kwigley pushed a commit that referenced this pull request May 19, 2021
* Check if column exists when altering column comments in snowflake

* Add new test class for persist docs models with missing columns

* Parallel run all integration tests after unit (#3328)

* don't clobber default args

* update changelog

* Update changelog for PR #3149

* Pull in upstream changes

Co-authored-by: Jeremy Cohen <jeremy@fishtownanalytics.com>
Co-authored-by: Kyle Wigley <kyle@fishtownanalytics.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects