Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bigquery: Add page_size parameter to QueryJob.result. #8206

Merged
merged 4 commits into from
Jun 5, 2019

Conversation

AzemaBaptiste
Copy link
Contributor

Makes the user control page size for a bigquery job.

@AzemaBaptiste AzemaBaptiste requested a review from a team June 2, 2019 17:04
@googlebot
Copy link

We found a Contributor License Agreement for you (the sender of this pull request), but were unable to find agreements for all the commit author(s) or Co-authors. If you authored these, maybe you used a different email address in the git commits than was used to sign the CLA (login here to double check)? If these were authored by someone else, then they will need to sign a CLA as well, and confirm that they're okay with these being contributed to Google.
In order to pass this check, please resolve this problem and have the pull request author add another comment and the bot will run again. If the bot doesn't comment, it means it doesn't think anything has changed.

ℹ️ Googlers: Go here for more info.

@googlebot googlebot added the cla: no This human has *not* signed the Contributor License Agreement. label Jun 2, 2019
@AzemaBaptiste
Copy link
Contributor Author

I signed it!

@googlebot
Copy link

CLAs look good, thanks!

ℹ️ Googlers: Go here for more info.

@googlebot googlebot added cla: yes This human has signed the Contributor License Agreement. and removed cla: no This human has *not* signed the Contributor License Agreement. labels Jun 2, 2019
@tseaver tseaver added api: bigquery Issues related to the BigQuery API. kokoro:force-run Add this label to force Kokoro to re-run the tests. labels Jun 3, 2019
@yoshi-kokoro yoshi-kokoro removed the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Jun 3, 2019
@tswast
Copy link
Contributor

tswast commented Jun 3, 2019

Coverage failure

tests/unit/test_magics.py                     327      0      4      1    99%   242->exit

is unrelated to this change and was fixed in #8179


result = job.result(page_size=3)

self.assertEqual(result.total_rows, 4)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's assert that tabledata.list is called with the right page size (maxResults) as a parameter.

See:

def test_list_rows(self):

Specifically, I'd like to see something like:

tabledata_path = "/projects/%s/datasets/%s/tables/%s/data" % (
    self.PROJECT,
    self.DS_ID,
    self.TABLE_ID,
)
conn.api_request.assert_called_once_with(
    method="GET", path=tabledata_path, query_params={"maxResults": 3}
)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hello Tim,
Thanks for the feedback but I think I need a hand.
Do you mean that the maxResults should be directly linked to the page_size ? The api documentation seems to say it link but the code don't

row_iterator = RowIterator(
client=self,
api_request=functools.partial(self._call_api, retry),
path="%s/data" % (table.path,),
schema=schema,
page_token=page_token,
max_results=max_results,
page_size=page_size,
extra_params=params,
table=table,
# Pass in selected_fields separately from schema so that full
# tables can be fetched without a column filter.
selected_fields=selected_fields,
)
return row_iterator

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Confusingly, page_size in Python is used to set maxResults in the API request, since that's what BigQuery calls its page size parameter.

I've updated your PR to test for this.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Okay I see...
Many thanks for the help on the unit test !

@tswast
Copy link
Contributor

tswast commented Jun 4, 2019

Thanks for the contribution @AzemaBaptiste. This is a great addition.

I like the system test. I'd just like to also verify in a unit test that the parameter is actually sent.

@tswast tswast added the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Jun 4, 2019
@yoshi-kokoro yoshi-kokoro removed kokoro:force-run Add this label to force Kokoro to re-run the tests. labels Jun 4, 2019
@tswast tswast added the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Jun 4, 2019
@yoshi-kokoro yoshi-kokoro removed the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Jun 4, 2019
@tseaver tseaver changed the title Bigquery: adds page_size parameter for result Bigquery: Add page_size parameter to QueryJob.result. Jun 5, 2019
@tseaver tseaver merged commit 6db6b4e into googleapis:master Jun 5, 2019
@bokub
Copy link

bokub commented Jun 6, 2019

Great job, thanks 🎉

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
api: bigquery Issues related to the BigQuery API. cla: yes This human has signed the Contributor License Agreement.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants