Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fetch a batch of rows from bigquery #5632

Merged
merged 3 commits into from
Aug 15, 2018
Merged

Fetch a batch of rows from bigquery #5632

Changes from 1 commit
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
5f66741
Fetch a batch of rows from bigquery
Aug 14, 2018
ae7afb5
unused const
Aug 15, 2018
18c48ff
review comments
Aug 15, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 10 additions & 0 deletions superset/db_engine_specs.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1367,6 +1367,15 @@ class BQEngineSpec(BaseEngineSpec):
As contributed by @mxmzdlv on issue #945"""
engine = 'bigquery'

"""
https://www.python.org/dev/peps/pep-0249/#arraysize
raw_connections bypass the pybigquery query execution context and deal with
raw dbapi connection directly.
If this value is not set, the default value is set to 1, as described here,
https://googlecloudplatform.github.io/google-cloud-python/latest/_modules/google/cloud/bigquery/dbapi/cursor.html#Cursor
"""
arraysize = 5000
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like the idea of setting the arraysize as a class attr, that would be the expected way to set it in derived classes. Maybe add those two lines in BaseEngineSpec.fetch_data:

if cls.arraysize:
    cursor.arraysize = cls.arraysize

and set arraysize = None in the base class

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

also let' s add a short comment on the arraysize = 5000 line that says where you got that number from (the default value in pybigquery)


time_grain_functions = {
None: '{col}',
'PT1S': 'TIMESTAMP_TRUNC({col}, SECOND)',
Expand All @@ -1388,6 +1397,7 @@ def convert_dttm(cls, target_type, dttm):

@classmethod
def fetch_data(cls, cursor, limit):
cursor.arraysize = 5000
data = super(BQEngineSpec, cls).fetch_data(cursor, limit)
if len(data) != 0 and type(data[0]).__name__ == 'Row':
data = [r.values() for r in data]
Expand Down