The magics API do not support setting client options

[boazsade]

Environment details

• OS type and version: jupyter notebook
• Python version: Python 3.7.3
• pip version: pip 20.1.1
• google-cloud-bigquery version: 1.25.0

Steps to reproduce

This is a missing option that do exists in google bigquery client - "https://googleapis.dev/python/bigquery/latest/generated/google.cloud.bigquery.client.Client.html#google.cloud.bigquery.client.Client"
But is missing from "https://github.com/googleapis/python-bigquery/blob/master/google/cloud/bigquery/magics.py"

Code example

client = bigquery.Client(
        project=project,
        credentials=context.credentials,
        default_query_job_config=context.default_query_job_config,
        client_info=client_info.ClientInfo(user_agent=IPYTHON_USER_AGENT),
    )

Note the the client_option is missing in creating the client in the code above

The patch to apply to fix this flow:

diff --git a/google/cloud/bigquery/magics.py b/google/cloud/bigquery/magics.py
index 7128e32..00e26d0 100644
--- a/google/cloud/bigquery/magics.py
+++ b/google/cloud/bigquery/magics.py
@@ -140,6 +140,10 @@
         Out[7]:     num
            ...: -------
            ...: 0    17
+        Executing query with job ID: bf633912-af2c-4780-b568-5d868058632b
+        Query executing: 2.61s
+        Query complete after 2.92s
+
 """
 
 from __future__ import print_function
@@ -183,6 +187,7 @@ class Context(object):
         self._project = None
         self._connection = None
         self._default_query_job_config = bigquery.QueryJobConfig()
+        self._client_options = google.api_core.client_options.ClientOptions()
 
     @property
     def credentials(self):
@@ -244,6 +249,30 @@ class Context(object):
     def project(self, value):
         self._project = value
 
+    @property
+    def client_options(self):
+        """ google.api_core.client_options.ClientOptions: client options to be used through IPython
+        Magics
+
+        Note::
+            The client options does not need to be explicitly defined if you have an
+            no special network conenctions are required. Normally you would
+            be using https://www.googleapis.com/ end point
+
+        Example:
+            Manually setting the endpoint:
+
+            >>> from google.cloud.bigquery import magics
+            >>> client_option = {}
+            >>> client_option['api_endpoint'] = "https://some.special.url"
+            >>> magics.context.client_options = client_option
+        """
+        return self._client_options
+
+    @client_options.setter
+    def client_options(self, value):
+        self._client_options = value
+
     @property
     def default_query_job_config(self):
         """google.cloud.bigquery.job.QueryJobConfig: Default job
@@ -459,6 +488,18 @@ def _create_dataset_if_necessary(client, dataset_id):
         "name (ex. $my_dict_var)."
     ),
 )
+@magic_arguments.argument(
+    "--options",
+    nargs="+",
+    default=None,
+    help=(
+        "Parameters to format the query string. If present, the --options "
+        "flag should be followed by a string representation of a dictionary "
+        "in the format (Union[google.api_core.client_options.ClientOptions, Dict])"
+        "or a reference to a dictionary in the same format. The dictionary "
+        "key/value matches does at google.api_core.client_options.ClientOptions"
+    ),
+)
 def _cell_magic(line, query):
     """Underlying function for bigquery cell magic
 
@@ -501,6 +542,7 @@ def _cell_magic(line, query):
         credentials=context.credentials,
         default_query_job_config=context.default_query_job_config,
         client_info=client_info.ClientInfo(user_agent=IPYTHON_USER_AGENT),
+        client_options=context.client_options
     )
     if context._connection:
         client._connection = context._connection

[plamut]

@boazsade Thanks for the suggestion and the patch!

I'll just check if Product has any objections (cc: @shollyman), and if not, we'll add this.

P.S.: If confirmed, would you mind opening an actual PR with added tests, or are you fine with us taking the patch as a base and create a PR ourselves?
(if significant portions of the patch are used, it would probably still require signing the CLA, though)

[boazsade]

You can take this pach, this is something that we really need to be part of the code base, thanks for the reply

[boazsade]

@plamut is there any progress with this issue? I was unable to create pull request, what about applying the patch? was that approved? thanks

[plamut]

@boazsade Not at this point, unfortunately, but I can re-check at this week's meeting. Lately one of the primary things in focus are preparations to dropping Python 2 support and moving them to the new code generator (for those who have some auto-generated parts).

[yoavcloud]

Hi @plamut. I wanted to check in and see if there's an update regarding including this patch in an upcoming release. One of our customers is in the process of migrating their data environment to BigQuery and this has become a priority. Thanks in advance.

[cguardia]

@yoavcloud Hi, I'm working on a PR to add your patch. Should be ready for review in a couple of days.

[yoavcloud]

Thank you so much @cguardia!

[cguardia]

@yoavcloud Well, I've been looking at this and have a couple of questions. While api_endpoint does seem like a useful option to have here, I don't think the rest of the client options are practical for magics, with the possible exception of quota_project_id. Passing in callables in the command line is just not possible, plus getting the lexer to parse the dictionary format in the first place would not be trivial. Have you tried the patch? I don't believe it can work as it is. We could probably add the api endpoint as an option, though. Would that cover your use case?

[boaz-work]

The patch file was generated by me (@boazsade), this patch was generate after we tested this to work. This was the reason we wanted it - after verify it. What we really need is the option to set the end point value (so that the traffic can pass not directly to google but through other authenticated host. As for the other parameters (like quota_project_id) I didn't test that, the reason I wrote it the way I did was to keep it with the same as the internal calls, but this can be dropped - again we really need the option to set the api_endpoint, other parameters as less important for us.

[tswast]

I don't think the rest of the client options are practical for magics

We don't necessarily need command-line options for all of these (though it wouldn't hurt to add). The most important thing is to make sure it is settable from google.cloud.bigquery.magics.context, as this object sets default values for all calls to the magics.

[cguardia]

@boazsade sorry about my confusion with the patch's authorship. Setting the client options directly on the context worked, but the command line options did not, so I changed the patch to add an --api_endpoint parameter instead. This is under review now.

[boaz-work]

good to here it, looking forward for this

[yoavcloud]

Thank you everyone!