Fallback to BQ API when there are problems reading from BQ Storage. #7633
Conversation
The tabledata.list API works for more kinds of tables, including small anonymous query results tables. By falling back to this API, we enable a developer to always specify a `bqstorage_client` even when they aren't writing their query results to a destination table and don't know how large their query results will be.
tswast
added
type: feature request
googlebot
added
the
cla: yes
| return self._to_dataframe_bqstorage(bqstorage_client, dtypes) | ||
| else: | ||
| return self._to_dataframe_tabledata_list(dtypes, progress_bar=progress_bar) | ||
| try: |
There was a problem hiding this comment.
Two questions:
-
Is the difference in progress_bar intentional?
-
Should we try to be more specific about what errors trigger fallback? I think the current exception is probably the right call, but do cases like non-enabled storage api fall under this as well?
There was a problem hiding this comment.
Is the difference in progress_bar intentional?
I haven't implemented the progress bar for BQ Storage API, yet. It'll need some additional features on the BQ Storage client to do well (expose to_dataframe() per block)
Should we try to be more specific about what errors trigger fallback?
I didn't want to be too specific, because the API currently returns "InternalError" when trying to read from small anonymous tables, which seemed like there's the potential to change in the future.
You're right that this does cach non-enabled storage API, which throws
E google.api_core.exceptions.PermissionDenied: 403 BigQuery Storage API has not been used in project before or it is disabled.
GoogleAPICallError is a superclass of PermissionDenied.
There was a problem hiding this comment.
Now that I've started work on the tutorial, I've had a change of heart regarding catching PermissionDenied and other authn/authz errors. #7661 We should rethrow those, but let the other (expected) errors get caught.
The tabledata.list API works for more kinds of tables, including small
anonymous query results tables. By falling back to this API, we enable
a developer to always specify a
bqstorage_clienteven when theyaren't writing their query results to a destination table and don't
know how large their query results will be.