Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: bootstrap task #233

Merged
merged 10 commits into from
May 2, 2021
Merged

feat: bootstrap task #233

merged 10 commits into from
May 2, 2021

Conversation

bastienboutonnet
Copy link
Member

@bastienboutonnet bastienboutonnet commented May 2, 2021

Description

Adds a new command called bootstrap which allows dbt users to generate placeholders for all models as well as their columns in a dbt project.

This command can be ran before doing an audit to generate placeholders for all models so that the results of the audit task (which relies solely on content present in schema.yml files) are exhaustive and accurate.

How was the change tested

Issue Information

Partially addresses #210

Checklist

  • I formatted my PR name according to CONTRIBUTING.md
  • I added a news fragment to help populating the changelog as encouraged in CONTRIBUTING.md
  • I added "Closes #<issue_number>" in the "Issue Information" section (if no issue, feel free to tick thick the box anyway).

@bastienboutonnet bastienboutonnet changed the base branch from main to dev/0.1.0 May 2, 2021 09:29
@bastienboutonnet bastienboutonnet marked this pull request as draft May 2, 2021 09:30
@codecov
Copy link

codecov bot commented May 2, 2021

Codecov Report

Merging #233 (e591a6f) into dev/0.1.0 (eb73c4d) will increase coverage by 0.02%.
The diff coverage is 88.15%.

Impacted file tree graph

@@              Coverage Diff              @@
##           dev/0.1.0     #233      +/-   ##
=============================================
+ Coverage      82.83%   82.86%   +0.02%     
=============================================
  Files             18       19       +1     
  Lines           1101     1167      +66     
=============================================
+ Hits             912      967      +55     
- Misses           189      200      +11     
Flag Coverage Δ
unittests 82.86% <88.15%> (+0.02%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
dbt_sugar/core/main.py 73.11% <57.14%> (-1.60%) ⬇️
dbt_sugar/core/task/bootstrap.py 89.13% <89.13%> (ø)
dbt_sugar/core/task/base.py 82.72% <92.30%> (-2.28%) ⬇️
dbt_sugar/core/connectors/base.py 95.00% <100.00%> (ø)
dbt_sugar/core/connectors/snowflake_connector.py 52.63% <100.00%> (ø)
dbt_sugar/core/task/audit.py 69.23% <100.00%> (+0.23%) ⬆️
dbt_sugar/core/task/doc.py 66.85% <100.00%> (+2.08%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update eb73c4d...e591a6f. Read the comment docs.

@bastienboutonnet bastienboutonnet marked this pull request as ready for review May 2, 2021 11:14
@bastienboutonnet bastienboutonnet added this to the Toxe - Honey Island milestone May 2, 2021
target_table: str,
target_schema: str,
) -> Optional[List[Tuple[Any]]]:
self, target_table: str, target_schema: str, **kwargs
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what is this argument **kwargs
is missing in the documentation

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's for the snowflake connector which may need to get the argument for use_describe. How would you document it?

virvirlopez
virvirlopez previously approved these changes May 2, 2021
Copy link
Contributor

@virvirlopez virvirlopez left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@sourcery-ai
Copy link
Contributor

sourcery-ai bot commented May 2, 2021

Sourcery Code Quality Report

❌  Merging this PR will decrease code quality in the affected files by 0.55%.

Quality metrics Before After Change
Complexity 4.09 ⭐ 4.06 ⭐ -0.03 👍
Method Length 68.04 🙂 69.54 🙂 1.50 👎
Working memory 10.89 😞 11.10 😞 0.21 👎
Quality 67.99% 🙂 67.44% 🙂 -0.55% 👎
Other metrics Before After Change
Lines 2793 2882 89
Changed files Quality Before Quality After Quality Change
dbt_sugar/core/main.py 62.91% 🙂 60.46% 🙂 -2.45% 👎
dbt_sugar/core/connectors/base.py 83.67% ⭐ 83.67% ⭐ 0.00%
dbt_sugar/core/connectors/snowflake_connector.py 73.25% 🙂 73.25% 🙂 0.00%
dbt_sugar/core/task/audit.py 74.73% 🙂 74.55% 🙂 -0.18% 👎
dbt_sugar/core/task/base.py 67.22% 🙂 67.73% 🙂 0.51% 👍
dbt_sugar/core/task/doc.py 66.44% 🙂 66.31% 🙂 -0.13% 👎
tests/audit_task_test.py 63.65% 🙂 60.84% 🙂 -2.81% 👎
tests/base_task_test.py 80.38% ⭐ 73.69% 🙂 -6.69% 👎
tests/doc_task_test.py 64.96% 🙂 64.93% 🙂 -0.03% 👎

Here are some functions in these files that still need a tune-up:

File Function Complexity Length Working Memory Quality Recommendation
dbt_sugar/core/task/base.py BaseTask.find_model_schema_file 25 😞 138 😞 16 ⛔ 34.07% 😞 Refactor to reduce nesting. Try splitting into smaller methods. Extract out complex expressions
tests/doc_task_test.py test_order_schema_yml 0 ⭐ 417 ⛔ 34 ⛔ 34.92% 😞 Try splitting into smaller methods. Extract out complex expressions
dbt_sugar/core/task/base.py BaseTask.update_model_description_test_tags 25 😞 116 🙂 12 😞 42.09% 😞 Refactor to reduce nesting. Extract out complex expressions
tests/audit_task_test.py test_get_model_column_description_coverage 0 ⭐ 130 😞 36 ⛔ 49.10% 😞 Try splitting into smaller methods. Extract out complex expressions
tests/doc_task_test.py test_column_has_primary_key_tests 0 ⭐ 140 😞 26 ⛔ 49.25% 😞 Try splitting into smaller methods. Extract out complex expressions

Legend and Explanation

The emojis denote the absolute quality of the code:

  • ⭐ excellent
  • 🙂 good
  • 😞 poor
  • ⛔ very poor

The 👍 and 👎 indicate whether the quality has improved or gotten worse with this pull request.


Please see our documentation here for details on how these metrics are calculated.

We are actively working on this report - lots more documentation and extra metrics to come!

Let us know what you think of it by mentioning @sourcery-ai in a comment.

@bastienboutonnet bastienboutonnet merged commit b1ecc6b into dev/0.1.0 May 2, 2021
@bastienboutonnet bastienboutonnet deleted the feat/bootstrap_task branch May 2, 2021 14:11
bastienboutonnet added a commit that referenced this pull request May 3, 2021
* setup arg parser and skeleton of bootstrap task

* implement barebone all dbt models dict building

* iterate through models and get col list and schema path

* combine getting columns and finding descriptor

* update tests for bootstrap_task

* update tests for audit_task

* add model placeholders in bootstrap_task

add all models placeholders

* refactor dbt models info in a dataclass for cleaner type checking

* feat: bootstrap task (Sourcery refactored) (#234)

* 'Refactored by Sourcery'

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

Co-authored-by: Sourcery AI <>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* fix typo in new fragment

Co-authored-by: sourcery-ai[bot] <58596630+sourcery-ai[bot]@users.noreply.github.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
bastienboutonnet added a commit that referenced this pull request May 5, 2021
* feat: add describe table logic in snowflake connector (#212)

* implement `describe table` logic in SnowflakeConnector

* add flag for experimental feature in config

* update test expectations to have use_describe_snowflake False

* add use_describe_snowflake CLI arg parsing

* update cli consumption tests

* 'Refactored by Sourcery' (#213)

Co-authored-by: Sourcery AI <>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* add changelog entry

* lowercase column names

Co-authored-by: sourcery-ai[bot] <58596630+sourcery-ai[bot]@users.noreply.github.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* feat: automatic insertion of primary key tests (#203)

* feat: automatic insertion of primary key tests.

* feat: adding tests

feat: adding tests

* feat: cleaning documentation

* feat: change the way of getting the SQL file

* Add new tests

* 'Refactored by Sourcery' (#216)

Co-authored-by: Sourcery AI <>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* feat: adding changelog defintion primary key tests

* feat: adding changelog defintion primary key tests

* rename and rephrase news fragment (#219)

Co-authored-by: sourcery-ai[bot] <58596630+sourcery-ai[bot]@users.noreply.github.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Bastien Boutonnet <bastien.b1@gmail.com>

* refactor: add use_describe to base (#236)

* add `use_describe` to the base `get_columns_from_table` def

* 'Refactored by Sourcery' (#237)

Co-authored-by: Sourcery AI <>

Co-authored-by: sourcery-ai[bot] <58596630+sourcery-ai[bot]@users.noreply.github.com>

* feat: bootstrap task (#233)

* setup arg parser and skeleton of bootstrap task

* implement barebone all dbt models dict building

* iterate through models and get col list and schema path

* combine getting columns and finding descriptor

* update tests for bootstrap_task

* update tests for audit_task

* add model placeholders in bootstrap_task

add all models placeholders

* refactor dbt models info in a dataclass for cleaner type checking

* feat: bootstrap task (Sourcery refactored) (#234)

* 'Refactored by Sourcery'

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

Co-authored-by: Sourcery AI <>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* fix typo in new fragment

Co-authored-by: sourcery-ai[bot] <58596630+sourcery-ai[bot]@users.noreply.github.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* feat: allow audit task to run bootstrap (#239)

* add bootstrap option to `audit` and call it if passed

* add news fragement

* feat: collect rich user input for tests with subprocess (#238)

* feat: collect rich user input for tests with subprocess

* fix: order of the combine list

Make nicer the logs

Secure variable

* add: feature.md

* [pre-commit.ci] pre-commit autoupdate (#242)

updates:
- [github.com/psf/black: 21.4b0 → 21.4b2](psf/black@21.4b0...21.4b2)

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* add dbt sugar logo

* chore(deps): Bump sqlalchemy from 1.4.12 to 1.4.13 (#243)

Bumps [sqlalchemy](https://github.com/sqlalchemy/sqlalchemy) from 1.4.12 to 1.4.13.
- [Release notes](https://github.com/sqlalchemy/sqlalchemy/releases)
- [Changelog](https://github.com/sqlalchemy/sqlalchemy/blob/master/CHANGES)
- [Commits](https://github.com/sqlalchemy/sqlalchemy/commits)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* chore(deps-dev): Bump pytest from 6.2.3 to 6.2.4 (#244)

* feat: add redshift connector (#229)

* add RedshiftConnector engine creation module

* add news fragment

* move pytest-dictsdiff to dev requirements

* make read_profile use PostgresDbtProfilesModel pydantic validation class

* 'Refactored by Sourcery' (#231)

Co-authored-by: Sourcery AI <>

* Bump version: 0.0.0 → 0.1.0-a.0

Co-authored-by: sourcery-ai[bot] <58596630+sourcery-ai[bot]@users.noreply.github.com>

Co-authored-by: sourcery-ai[bot] <58596630+sourcery-ai[bot]@users.noreply.github.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: virvirlopez <virbyte@gmail.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants