-
-
Notifications
You must be signed in to change notification settings - Fork 20
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat: bootstrap task #233
feat: bootstrap task #233
Conversation
Codecov Report
@@ Coverage Diff @@
## dev/0.1.0 #233 +/- ##
=============================================
+ Coverage 82.83% 82.86% +0.02%
=============================================
Files 18 19 +1
Lines 1101 1167 +66
=============================================
+ Hits 912 967 +55
- Misses 189 200 +11
Flags with carried forward coverage won't be shown. Click here to find out more.
Continue to review full report at Codecov.
|
83ba209
to
45669d4
Compare
dbt_sugar/core/connectors/base.py
Outdated
target_table: str, | ||
target_schema: str, | ||
) -> Optional[List[Tuple[Any]]]: | ||
self, target_table: str, target_schema: str, **kwargs |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
what is this argument **kwargs
is missing in the documentation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It's for the snowflake connector which may need to get the argument for use_describe
. How would you document it?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
add all models placeholders
* 'Refactored by Sourcery' * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci Co-authored-by: Sourcery AI <> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
3723978
to
e591a6f
Compare
Sourcery Code Quality Report❌ Merging this PR will decrease code quality in the affected files by 0.55%.
Here are some functions in these files that still need a tune-up:
Legend and ExplanationThe emojis denote the absolute quality of the code:
The 👍 and 👎 indicate whether the quality has improved or gotten worse with this pull request. Please see our documentation here for details on how these metrics are calculated. We are actively working on this report - lots more documentation and extra metrics to come! Let us know what you think of it by mentioning @sourcery-ai in a comment. |
* setup arg parser and skeleton of bootstrap task * implement barebone all dbt models dict building * iterate through models and get col list and schema path * combine getting columns and finding descriptor * update tests for bootstrap_task * update tests for audit_task * add model placeholders in bootstrap_task add all models placeholders * refactor dbt models info in a dataclass for cleaner type checking * feat: bootstrap task (Sourcery refactored) (#234) * 'Refactored by Sourcery' * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci Co-authored-by: Sourcery AI <> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * fix typo in new fragment Co-authored-by: sourcery-ai[bot] <58596630+sourcery-ai[bot]@users.noreply.github.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* feat: add describe table logic in snowflake connector (#212) * implement `describe table` logic in SnowflakeConnector * add flag for experimental feature in config * update test expectations to have use_describe_snowflake False * add use_describe_snowflake CLI arg parsing * update cli consumption tests * 'Refactored by Sourcery' (#213) Co-authored-by: Sourcery AI <> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * add changelog entry * lowercase column names Co-authored-by: sourcery-ai[bot] <58596630+sourcery-ai[bot]@users.noreply.github.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * feat: automatic insertion of primary key tests (#203) * feat: automatic insertion of primary key tests. * feat: adding tests feat: adding tests * feat: cleaning documentation * feat: change the way of getting the SQL file * Add new tests * 'Refactored by Sourcery' (#216) Co-authored-by: Sourcery AI <> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * feat: adding changelog defintion primary key tests * feat: adding changelog defintion primary key tests * rename and rephrase news fragment (#219) Co-authored-by: sourcery-ai[bot] <58596630+sourcery-ai[bot]@users.noreply.github.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Bastien Boutonnet <bastien.b1@gmail.com> * refactor: add use_describe to base (#236) * add `use_describe` to the base `get_columns_from_table` def * 'Refactored by Sourcery' (#237) Co-authored-by: Sourcery AI <> Co-authored-by: sourcery-ai[bot] <58596630+sourcery-ai[bot]@users.noreply.github.com> * feat: bootstrap task (#233) * setup arg parser and skeleton of bootstrap task * implement barebone all dbt models dict building * iterate through models and get col list and schema path * combine getting columns and finding descriptor * update tests for bootstrap_task * update tests for audit_task * add model placeholders in bootstrap_task add all models placeholders * refactor dbt models info in a dataclass for cleaner type checking * feat: bootstrap task (Sourcery refactored) (#234) * 'Refactored by Sourcery' * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci Co-authored-by: Sourcery AI <> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * fix typo in new fragment Co-authored-by: sourcery-ai[bot] <58596630+sourcery-ai[bot]@users.noreply.github.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * feat: allow audit task to run bootstrap (#239) * add bootstrap option to `audit` and call it if passed * add news fragement * feat: collect rich user input for tests with subprocess (#238) * feat: collect rich user input for tests with subprocess * fix: order of the combine list Make nicer the logs Secure variable * add: feature.md * [pre-commit.ci] pre-commit autoupdate (#242) updates: - [github.com/psf/black: 21.4b0 → 21.4b2](psf/black@21.4b0...21.4b2) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * add dbt sugar logo * chore(deps): Bump sqlalchemy from 1.4.12 to 1.4.13 (#243) Bumps [sqlalchemy](https://github.com/sqlalchemy/sqlalchemy) from 1.4.12 to 1.4.13. - [Release notes](https://github.com/sqlalchemy/sqlalchemy/releases) - [Changelog](https://github.com/sqlalchemy/sqlalchemy/blob/master/CHANGES) - [Commits](https://github.com/sqlalchemy/sqlalchemy/commits) Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * chore(deps-dev): Bump pytest from 6.2.3 to 6.2.4 (#244) * feat: add redshift connector (#229) * add RedshiftConnector engine creation module * add news fragment * move pytest-dictsdiff to dev requirements * make read_profile use PostgresDbtProfilesModel pydantic validation class * 'Refactored by Sourcery' (#231) Co-authored-by: Sourcery AI <> * Bump version: 0.0.0 → 0.1.0-a.0 Co-authored-by: sourcery-ai[bot] <58596630+sourcery-ai[bot]@users.noreply.github.com> Co-authored-by: sourcery-ai[bot] <58596630+sourcery-ai[bot]@users.noreply.github.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: virvirlopez <virbyte@gmail.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Description
Adds a new command called
bootstrap
which allows dbt users to generate placeholders for all models as well as their columns in a dbt project.This command can be ran before doing an
audit
to generate placeholders for all models so that the results of theaudit
task (which relies solely on content present inschema.yml
files) are exhaustive and accurate.How was the change tested
Issue Information
Partially addresses #210
Checklist