Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: support BI Engine statistics in query job #1144

Merged
merged 4 commits into from
Feb 18, 2022

Conversation

anmolsahoo25
Copy link
Contributor

@anmolsahoo25 anmolsahoo25 commented Feb 18, 2022

The REST API return BiEngineStatistics which denotes if the query
was accelerated by BI Engine or not. This commit adds the necessary
function to access this information for executed queries.

Thank you for opening a Pull Request! Before submitting your PR, there are a few things you can do to make sure it goes smoothly:

  • Make sure to open an issue as a bug/issue before writing your code! That way we can discuss the change, evaluate designs, and agree on the general idea
  • Ensure the tests and linter pass
  • Code coverage does not decrease (if any source code was changed)
  • Appropriate docs were updated (if necessary)

Fixes #1054

@anmolsahoo25 anmolsahoo25 requested a review from a team February 18, 2022 07:45
@anmolsahoo25 anmolsahoo25 requested a review from a team as a code owner February 18, 2022 07:45
@product-auto-label product-auto-label bot added the api: bigquery Issues related to the googleapis/python-bigquery API. label Feb 18, 2022
The REST API returns BiEngineStatistics for a query which denotes if the
query was accelerated by BI Engine or not. This commit adds the
necessary function to access this information for executed queries.
@tswast tswast changed the title Add support for accessing BI Engine statistics of query feat: support BI Engine statistics in query job Feb 18, 2022
google/cloud/bigquery/enums.py Outdated Show resolved Hide resolved
google/cloud/bigquery/enums.py Outdated Show resolved Hide resolved
google/cloud/bigquery/job/query.py Show resolved Hide resolved
@tswast tswast added the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Feb 18, 2022
@yoshi-kokoro yoshi-kokoro removed the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Feb 18, 2022
@anmolsahoo25
Copy link
Contributor Author

Thanks for the review! Appreciate it. I'll get to the changes. Since I mostly write OCaml, I am used to writing enums for everything!

Also, just needed some help on the system tests? I'd like to add system tests for the BI Engine statistics, but does the CI have BI Engine reservation? I can reliably trigger BI Engine messages on my local machine, but was wondering what the recommended practice was?

@tswast
Copy link
Contributor

tswast commented Feb 18, 2022

does the CI have BI Engine reservation?

We do not currently, and I'm not certain that we'd want to due to cost reasons (plus we try to avoid long-lived resources in our CI infrastructure)

I'm okay with just unit tests for this. I'll make sure to give a careful look comparing the implementation with the REST docs.

@anmolsahoo25
Copy link
Contributor Author

Yup, that makes sense. Thanks! Do let me know if anything else is required.

Copy link
Contributor

@tswast tswast left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking good. Thanks so much for the contribution. Just a little bit more to get this ready for merge.

Comment on lines 1240 to 1243
if stats is not None:
return None
else:
return BiEngineStats.from_api_repr(stats)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we've got the logic flipped here. (Possibly an indication we need some unit test coverage for this?)

Suggested change
if stats is not None:
return None
else:
return BiEngineStats.from_api_repr(stats)
if stats is None:
return None
else:
return BiEngineStats.from_api_repr(stats)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's add a test much like this one:

def test_ddl_target_routine(self):

That checks that it can be None if the stats aren't present, otherwise return the expected object.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My bad! Added a test case as well.

Comment on lines 156 to 161
reasons = stats.get("biEngineReasons")

if reasons is None:
reasons = []
else:
reasons = [BiEngineReason.from_api_repr(r) for r in reasons]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can simplify this by using the "default" argument for get

Suggested change
reasons = stats.get("biEngineReasons")
if reasons is None:
reasons = []
else:
reasons = [BiEngineReason.from_api_repr(r) for r in reasons]
reasons = [BiEngineReason.from_api_repr(r) for r in stats.get("biEngineReasons", []]

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done! Thank you for the tip. Learnt something new today.

@tswast tswast added the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Feb 18, 2022
@yoshi-kokoro yoshi-kokoro removed the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Feb 18, 2022
@anmolsahoo25
Copy link
Contributor Author

Thanks for your time. :)

google/cloud/bigquery/job/query.py Outdated Show resolved Hide resolved
google/cloud/bigquery/job/query.py Outdated Show resolved Hide resolved
google/cloud/bigquery/job/query.py Outdated Show resolved Hide resolved
@tswast tswast added the owlbot:run Add this label to trigger the Owlbot post processor. label Feb 18, 2022
@gcf-owl-bot gcf-owl-bot bot removed the owlbot:run Add this label to trigger the Owlbot post processor. label Feb 18, 2022
@tswast tswast added kokoro:force-run Add this label to force Kokoro to re-run the tests. owlbot:run Add this label to trigger the Owlbot post processor. labels Feb 18, 2022
@gcf-owl-bot gcf-owl-bot bot removed the owlbot:run Add this label to trigger the Owlbot post processor. label Feb 18, 2022
@yoshi-kokoro yoshi-kokoro removed the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Feb 18, 2022
@tswast tswast merged commit 7482549 into googleapis:main Feb 18, 2022
abdelmegahedgoogle pushed a commit to abdelmegahedgoogle/python-bigquery that referenced this pull request Apr 17, 2023
* chore: Add support for accessing BI Engine statistics

The REST API returns BiEngineStatistics for a query which denotes if the
query was accelerated by BI Engine or not. This commit adds the
necessary function to access this information for executed queries.

* fix: Removed enums and replaced with string constants

* fix: Fixed logic for creating BIEngineStats and added test case

* Attempt at mypy fix

Co-authored-by: Tim Swast <swast@google.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
api: bigquery Issues related to the googleapis/python-bigquery API.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add BiEngineStatistics to query job statistics
3 participants