process: add mypy-valid type annotations to samples #1081

plamut · 2021-12-08T15:34:50Z

Towards #1074.

This PR adds type annotations to samples and fixes quite a few issues discovered along. It also adds a new nox session to run type checks with mypy on samples.

The mypy.ini files in samples/* subdirectories are added for the upcoming type-check sessions that will be added to the noxfiles in these directories (through the code generator).

PR checklist:

Make sure to open an issue as a bug/issue before writing your code! That way we can discuss the change, evaluate designs, and agree on the general idea
Ensure the tests and linter pass
Code coverage does not decrease (if any source code was changed)
Appropriate docs were updated (if necessary)

* feat: allow cell magic body to be a $variable * Fix missing indefinitive article in error msg * Adjust test assertion to error message change * Refactor logic for extracting query variable * Explicitly warn about missing query variable name * Thest the query "variable" is not identifier case

…eapis#1073) * feat: promote `to_arrow_iterable` to public method * use correct version number * Update google/cloud/bigquery/table.py Co-authored-by: Tim Swast <swast@google.com>

* fix: apply timeout to all resumable upload requests * Fix stub in test case * Improve timeout type and other type annotations * Annnotate return type of _do_resumable_upload()

Co-authored-by: release-please[bot] <55107282+release-please[bot]@users.noreply.github.com>

…taFrame (googleapis#1078) Thank you for opening a Pull Request! Before submitting your PR, there are a few things you can do to make sure it goes smoothly: - [ ] Make sure to open an issue as a [bug/issue](https://github.com/googleapis/python-bigquery/issues/new/choose) before writing your code! That way we can discuss the change, evaluate designs, and agree on the general idea - [ ] Ensure the tests and linter pass - [ ] Code coverage does not decrease (if any source code was changed) - [ ] Appropriate docs were updated (if necessary) Towards #1076 🦕 (edit: moved to googleapis/python-db-dtypes-pandas#45 )

* revoke dataset access setup * basic template for sample * sample + test * revoke dataset access sample * 🦉 Updates from OwlBot See https://github.com/googleapis/repo-automation-bots/blob/master/packages/owl-bot/README.md * docs: add sample for revoking dataset access - update year and string formatting * docs: add sample for revoking dataset access - move to snippets and change parameter pattern for readibility * moving update_dataset to /snippets and adjusting imports on both revoke_access and update_access * Update samples/snippets/revoke_dataset_access.py removed nested START/END tags Co-authored-by: Tim Swast <swast@google.com> * Update samples/snippets/revoke_dataset_access.py update readability in API request Co-authored-by: Tim Swast <swast@google.com> * 🦉 Updates from OwlBot See https://github.com/googleapis/repo-automation-bots/blob/master/packages/owl-bot/README.md * updated test * 🦉 Updates from OwlBot See https://github.com/googleapis/repo-automation-bots/blob/master/packages/owl-bot/README.md * change after running test * resolving linting failure, rewriting test * removed relative import errors * remove relative mport from update_dataset_access * adding fixture to conftest.py * updated sample * updating sample to match new update_access sample * fixing region tags * consolidated tests into one file for both methods * updating test to full_dataset format * updated revoke sample * updating test * refactored sample * Update samples/snippets/conftest.py * Update samples/snippets/revoke_dataset_access.py Co-authored-by: Tim Swast <swast@google.com> * Update samples/snippets/update_dataset_access.py Co-authored-by: Tim Swast <swast@google.com> * Update samples/snippets/revoke_dataset_access.py Co-authored-by: Tim Swast <swast@google.com> * Update samples/snippets/revoke_dataset_access.py Co-authored-by: Tim Swast <swast@google.com> * refactoring entry * added comment for entry access * Update samples/snippets/README.rst Co-authored-by: Tim Swast <swast@google.com> * Update samples/snippets/dataset_access_test.py Co-authored-by: Tim Swast <swast@google.com> * Update samples/snippets/dataset_access_test.py Co-authored-by: Tim Swast <swast@google.com> * added develper TODO in sample * add comments to samples Co-authored-by: Owl Bot <gcf-owl-bot[bot]@users.noreply.github.com> Co-authored-by: Tim Swast <swast@google.com> Co-authored-by: Peter Lamut <plamut@users.noreply.github.com> Co-authored-by: Anthonios Partheniou <partheniou@google.com> Co-authored-by: meredithslota <meredithslota@google.com>

Samples use the code from the core library, but mypy complains about calling an untyped function in a typed context, thus some additional type annotations need to be added.

plamut · 2021-12-08T15:53:48Z

ImportError: cannot import name 'TypedDict'

This was only added to typing in Python 3.8. I actually fixed this, but not at all places, and I was only running the checks with Python 3.8, missing this. 🙃

tswast

So cool!

A few nits. Plus I'd to see some improvements to the ExternalConfig class to see if we can avoid casts in those samples.

samples/geography/insert_geojson.py

samples/geography/insert_wkt.py

samples/query_external_gcs_temporary_table.py

samples/query_external_sheets_permanent_table.py

samples/query_external_sheets_temporary_table.py

samples/snippets/update_with_dml.py

snippet-bot · 2021-12-14T15:26:30Z

Here is the summary of changes.

You are about to add 4 region tags.

samples/snippets/revoke_dataset_access.py:20, tag bigquery_revoke_dataset_access
samples/snippets/revoke_dataset_access.py:30, tag bigquery_revoke_dataset_access
samples/snippets/update_dataset_access.py:20, tag bigquery_update_dataset_access
samples/snippets/update_dataset_access.py:49, tag bigquery_update_dataset_access

This comment is generated by snippet-bot.
If you find problems with this result, please file an issue at:
https://github.com/googleapis/repo-automation-bots/issues.
To update this comment, add snippet-bot:force-run label or use the checkbox below:

Refresh this comment

plamut · 2021-12-14T15:29:56Z

@tswast Since there was a new sample added to main, I merged it into this branch. Had to resolve some conflicts, thus that commit should also be checked.

The rest of the comments have been addressed, and I got rid of several cast() calls by using explicit is None assertions instead - let me know how that looks to you.

(and I didn't commit suggestions through the web UI, as that has lead to the CLA bot complaining without merit)

tswast

Awesome!

It's a bit funny to have assertions in the code samples, but not too out of place. It looks more natural than the type casts.

plamut · 2021-12-14T15:38:41Z

It's a bit funny to have assertions in the code samples, but not too out of place. It looks more natural than the type casts.

True that, but I agree that's much better than those rather ugly casts.

deps!: BigQuery Storage and pyarrow are required dependencies (#776) fix!: use nullable `Int64` and `boolean` dtypes in `to_dataframe` (#786) feat!: destination tables are no-longer removed by `create_job` (#891) feat!: In `to_dataframe`, use `dbdate` and `dbtime` dtypes from db-dtypes package for BigQuery DATE and TIME columns (#972) fix!: automatically convert out-of-bounds dates in `to_dataframe`, remove `date_as_object` argument (#972) feat!: mark the package as type-checked (#1058) feat!: default to DATETIME type when loading timezone-naive datetimes from Pandas (#1061) feat: add `api_method` parameter to `Client.query` to select `INSERT` or `QUERY` API (#967) fix: improve type annotations for mypy validation (#1081) feat: use `StandardSqlField` class for `Model.feature_columns` and `Model.label_columns` (#1117) docs: Add migration guide from version 2.x to 3.x (#1027) Release-As: 3.0.0

deps!: BigQuery Storage and pyarrow are required dependencies (googleapis#776) fix!: use nullable `Int64` and `boolean` dtypes in `to_dataframe` (googleapis#786) feat!: destination tables are no-longer removed by `create_job` (googleapis#891) feat!: In `to_dataframe`, use `dbdate` and `dbtime` dtypes from db-dtypes package for BigQuery DATE and TIME columns (googleapis#972) fix!: automatically convert out-of-bounds dates in `to_dataframe`, remove `date_as_object` argument (googleapis#972) feat!: mark the package as type-checked (googleapis#1058) feat!: default to DATETIME type when loading timezone-naive datetimes from Pandas (googleapis#1061) feat: add `api_method` parameter to `Client.query` to select `INSERT` or `QUERY` API (googleapis#967) fix: improve type annotations for mypy validation (googleapis#1081) feat: use `StandardSqlField` class for `Model.feature_columns` and `Model.label_columns` (googleapis#1117) docs: Add migration guide from version 2.x to 3.x (googleapis#1027) Release-As: 3.0.0

plamut and others added 16 commits November 16, 2021 22:57

cleanup: silence non-relevant system test warnings (googleapis#1068)

1b5dc5c

feat: promote RowIterator.to_arrow_iterable to public method (googl…

21cd710

…eapis#1073) * feat: promote `to_arrow_iterable` to public method * use correct version number * Update google/cloud/bigquery/table.py Co-authored-by: Tim Swast <swast@google.com>

fix: apply timeout to all resumable upload requests (googleapis#1070)

3314dfb

* fix: apply timeout to all resumable upload requests * Fix stub in test case * Improve timeout type and other type annotations * Annnotate return type of _do_resumable_upload()

chore: release 2.31.0 (googleapis#1066)

a135956

Co-authored-by: release-please[bot] <55107282+release-please[bot]@users.noreply.github.com>

Add type hints to magics samples

d1a9902

Add type hints to geography samples

8141548

Add type hints to snippets

0b58d9d

Add type hints to top level samples (part 1)

452f15e

Add type hints to top level samples (part 2)

ee1179a

Add type hints to top level samples (part 3)

d77d70e

Add type hints to core libraries for samples' needs

8195894

Samples use the code from the core library, but mypy complains about calling an untyped function in a typed context, thus some additional type annotations need to be added.

Add top level mypy config and samples session

6a3713b

Add mypy.ini files to samples subdirectories

c4d0ab5

plamut requested review from tswast and a team December 8, 2021 15:34

plamut requested review from a team as code owners December 8, 2021 15:34

plamut requested a review from busunkim96 December 8, 2021 15:34

product-auto-label bot added the api: bigquery Issues related to the googleapis/python-bigquery API. label Dec 8, 2021

plamut changed the title ~~Iss 1074~~ Add mypy-valid type annotations to samples Dec 8, 2021

plamut changed the title ~~Add mypy-valid type annotations to samples~~ process: add mypy-valid type annotations to samples Dec 8, 2021

Enforce late evaluation of some annotations

c0ef9a0

Install typing.TypedDict backport

86dd389

plamut force-pushed the iss-1074 branch from c00fa0b to be35bc4 Compare December 8, 2021 18:32

Make sure typing.TypedDict is available in samples

278dac7

plamut force-pushed the iss-1074 branch from be35bc4 to 278dac7 Compare December 8, 2021 18:34

plamut added 2 commits December 8, 2021 22:35

Use import fallback for TypedDict

2c78c72

Fix service account fixture in snippets

12e118d

product-auto-label bot added the samples Issues that are directly related to samples. label Dec 9, 2021

chore: update python-docs-samples link to main branch (googleapis#1082)

44221f4

tswast requested changes Dec 10, 2021

View reviewed changes

plamut added 5 commits December 14, 2021 12:02

Merge branch 'main' into iss-1074

eaaffb7

Add missing annotations to new samples

040a21c

Move the override_values default check out

fdabea3

Get rid of type cast of an optional int property

484a6ee

Simplify external config type annotations

8c40ec7

tswast approved these changes Dec 14, 2021

View reviewed changes

plamut added the automerge Merge the pull request once unit tests and other checks pass. label Dec 14, 2021

tswast merged commit 7e3721e into googleapis:v3 Dec 14, 2021

gcf-merge-on-green bot removed the automerge Merge the pull request once unit tests and other checks pass. label Dec 14, 2021

plamut deleted the iss-1074 branch December 14, 2021 16:07

tswast mentioned this pull request Dec 15, 2021

chore: sync v3 with main branch #1088

Merged

tswast mentioned this pull request Mar 29, 2022

fix!: remove out-of-date BigQuery ML protocol buffers #1178

Merged

4 tasks

release-please bot mentioned this pull request Mar 29, 2022

chore(main): release 3.0.0 #1179

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

process: add mypy-valid type annotations to samples #1081

process: add mypy-valid type annotations to samples #1081

plamut commented Dec 8, 2021

plamut commented Dec 8, 2021

tswast left a comment

snippet-bot bot commented Dec 14, 2021

plamut commented Dec 14, 2021 •

edited

Loading

tswast left a comment

plamut commented Dec 14, 2021

process: add mypy-valid type annotations to samples #1081

process: add mypy-valid type annotations to samples #1081

Conversation

plamut commented Dec 8, 2021

plamut commented Dec 8, 2021

tswast left a comment

Choose a reason for hiding this comment

snippet-bot bot commented Dec 14, 2021

plamut commented Dec 14, 2021 • edited Loading

tswast left a comment

Choose a reason for hiding this comment

plamut commented Dec 14, 2021

plamut commented Dec 14, 2021 •

edited

Loading