feat: Support passing struct data to the DB API #718

jimfulton · 2021-06-23T16:17:42Z

Thank you for opening a Pull Request! Before submitting your PR, there are a few things you can do to make sure it goes smoothly:

Make sure to open an issue as a bug/issue before writing your code! That way we can discuss the change, evaluate designs, and agree on the general idea
Ensure the tests and linter pass
Code coverage does not decrease (if any source code was changed)
[ x] Appropriate docs were updated (if necessary)

Fixes #717 🦕

(In fact, we can't bind with parametrized types.)

…ct-types

See https://github.com/googleapis/repo-automation-bots/blob/master/packages/owl-bot/README.md

This reverts commit 462f2eb.

…ct-types

jimfulton · 2021-06-24T12:42:22Z

This also, as a side effect makes the pandas system tests run faster by changing the scope of the dataset_id fixture to session.

jimfulton · 2021-06-24T12:44:48Z

A non-trivial part of this is allowing type parameters, as in string(10), which it then strips off.

This allows users to specify type parameters matching the table definitions. But we have to strip them off because they aren't allowed past table creation. I'm not sure the benefit of this is worth the extra complexity.

plamut · 2021-06-24T14:24:48Z

Quick comment - it would be helpful to reason about the code if helpers had an example or two in their docstrings, explaining the format of a typical (sub)string they process. You know, just to give readers some context. :)

plamut

Just two quick comments for a start, will have another look next time with a fresh head. :)

(some examples in helpers' docstrings would also be appreciated)

google/cloud/bigquery/dbapi/_helpers.py

See https://github.com/googleapis/repo-automation-bots/blob/master/packages/owl-bot/README.md

jimfulton · 2021-06-24T17:10:44Z

Quick comment - it would be helpful to reason about the code if helpers had an example or two in their docstrings, explaining the format of a typical (sub)string they process. You know, just to give readers some context. :)

Good idea! Done.

…ct-types

…ython-bigquery into riversnake-dbi-struct-types

plamut

Spotted a few minor things, but otherwise this looks good!

See if we could reduce code duplication with a reasonable effort, but we can do it separately if it proves to be too complex for now.

google/cloud/bigquery/dbapi/_helpers.py

docs/dbapi.rst

google/cloud/bigquery/dbapi/cursor.py

…icit struct type and also factored some repeated code

plamut · 2021-06-28T09:00:56Z

google/cloud/bigquery/dbapi/_helpers.py

+        raise NotImplementedError(
+            f"STRUCT-like parameter values are not supported"
+            f"{' (parameter ' + name + ')' if name else ''},"
+            f" unless an explicit type is give in the parameter placeholder"
+            f" (e.g. '%({name if name else ''}:struct<...>)s')."
+            )


Much more informative, great!

google/cloud/bigquery/dbapi/_helpers.py

plamut

LGTM, thanks for the refactoring.

I'll wait with merging a bit, just in case @shollyman has something to add.

google/cloud/bigquery/dbapi/_helpers.py

tswast · 2021-06-29T17:53:24Z

google/cloud/bigquery/dbapi/_helpers.py

+        raise exceptions.ProgrammingError(f"Invalid parameter type, {type_}")
+    tname, sub = m.group(1, 2)
+    if tname.upper() == "ARRAY":
+        sub_type = complex_query_parameter_type(None, sub, base)


This allows for arrays of scalars, right? Looks like it from looking at the code, but then I'm a bit confused why we need the scalar logic in lines 183-197.

We need the code on line 183-197 because we support arrays and structs of scalars. Basically, we allow arrays of anything (other than arrays). To handle that, we recurse to handle whatever's inside the <>s, which might be a simple declaration like INT64.

Co-authored-by: Tim Swast <swast@google.com>

plamut · 2021-07-01T11:00:13Z

Last call - anybody else wants to add anything here? If not, I'll merge it soon. 🙂

jimfulton added 9 commits June 22, 2021 18:08

Added support for dbi struct parameters with explicit types

586a022

Make the dataset_id fixture a session fixture

13f9107

Make the dataset_id fixture a session fixture

aea7bae

system test of the struct machinery

6f26130

Verify that we can bind non-parameterized types

c386448

(In fact, we can't bind with parametrized types.)

Parse and remove type parameters from explcit types.

71c8614

Document passing struct data.

6f5b345

Merge remote-tracking branch 'origin/master' into riversnake-dbi-stru…

e08f6f6

…ct-types

blacken

411c336

product-auto-label bot added the api: bigquery Issues related to the googleapis/python-bigquery API. label Jun 23, 2021

google-cla bot added the cla: yes This human has signed the Contributor License Agreement. label Jun 23, 2021

jimfulton and others added 7 commits June 23, 2021 15:27

using match.groups() throws off pytypes, also fix some type hints.

ef2b323

🦉 Updates from OwlBot

654b108

See https://github.com/googleapis/repo-automation-bots/blob/master/packages/owl-bot/README.md

blacken

525b8fd

remove type hints -- maybe they broke docs?

462f2eb

merge upstream

904d2ce

Revert "remove type hints -- maybe they broke docs?"

a6393e6

This reverts commit 462f2eb.

pin gcp-sphinx-docfx-yaml==0.2.0 so docfx doesn't fail.

e63e8b7

jimfulton marked this pull request as ready for review June 23, 2021 21:19

jimfulton requested a review from a team June 23, 2021 21:19

jimfulton requested a review from a team as a code owner June 23, 2021 21:19

jimfulton requested review from shollyman and removed request for a team June 23, 2021 21:19

Merge remote-tracking branch 'origin/master' into riversnake-dbi-stru…

2f2bdcd

…ct-types

plamut reviewed Jun 24, 2021

View reviewed changes

google/cloud/bigquery/dbapi/_helpers.py Outdated Show resolved Hide resolved

google/cloud/bigquery/dbapi/_helpers.py Outdated Show resolved Hide resolved

jimfulton and others added 2 commits June 24, 2021 13:06

Review comments: examples, and guard against large number of fields

91b0028

🦉 Updates from OwlBot

35555aa

See https://github.com/googleapis/repo-automation-bots/blob/master/packages/owl-bot/README.md

jimfulton added 2 commits June 24, 2021 16:11

Merge remote-tracking branch 'origin/master' into riversnake-dbi-stru…

0d81b80

…ct-types

Merge branch 'riversnake-dbi-struct-types' of github.com:googleapis/p…

d3f959c

…ython-bigquery into riversnake-dbi-struct-types

plamut reviewed Jun 25, 2021

View reviewed changes

jimfulton added 4 commits June 25, 2021 09:57

Factored some repeated code in handling complex parameters

5554301

Improved the error for dict (structish) parameter values without expl…

8830113

…icit struct type and also factored some repeated code

blacken

a72cafb

removed repeated word

cba9697

plamut reviewed Jun 28, 2021

View reviewed changes

plamut approved these changes Jun 28, 2021

View reviewed changes

tswast self-requested a review June 29, 2021 16:33

tswast reviewed Jun 29, 2021

View reviewed changes

Update google/cloud/bigquery/dbapi/_helpers.py

12bd941

Co-authored-by: Tim Swast <swast@google.com>

jimfulton added kokoro:force-run Add this label to force Kokoro to re-run the tests. kokoro:run Add this label to force Kokoro to re-run the tests. labels Jun 29, 2021

yoshi-kokoro removed kokoro:run Add this label to force Kokoro to re-run the tests. kokoro:force-run Add this label to force Kokoro to re-run the tests. labels Jun 29, 2021

tswast approved these changes Jul 1, 2021

View reviewed changes

jimfulton merged commit 38b3ef9 into master Jul 1, 2021

jimfulton deleted the riversnake-dbi-struct-types branch July 1, 2021 14:49

release-please bot mentioned this pull request Jan 4, 2022

chore(main): release python-bigquery 1.27.1 #1097

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Support passing struct data to the DB API #718

feat: Support passing struct data to the DB API #718

jimfulton commented Jun 23, 2021

jimfulton commented Jun 24, 2021

jimfulton commented Jun 24, 2021

plamut commented Jun 24, 2021

plamut left a comment •

edited

Loading

jimfulton commented Jun 24, 2021

plamut left a comment

plamut Jun 28, 2021

plamut left a comment •

edited

Loading

tswast Jun 29, 2021

jimfulton Jun 29, 2021

plamut commented Jul 1, 2021 •

edited

Loading

feat: Support passing struct data to the DB API #718

feat: Support passing struct data to the DB API #718

Conversation

jimfulton commented Jun 23, 2021

jimfulton commented Jun 24, 2021

jimfulton commented Jun 24, 2021

plamut commented Jun 24, 2021

plamut left a comment • edited Loading

Choose a reason for hiding this comment

jimfulton commented Jun 24, 2021

plamut left a comment

Choose a reason for hiding this comment

plamut Jun 28, 2021

Choose a reason for hiding this comment

plamut left a comment • edited Loading

Choose a reason for hiding this comment

tswast Jun 29, 2021

Choose a reason for hiding this comment

jimfulton Jun 29, 2021

Choose a reason for hiding this comment

plamut commented Jul 1, 2021 • edited Loading

plamut left a comment •

edited

Loading

plamut left a comment •

edited

Loading

plamut commented Jul 1, 2021 •

edited

Loading