BigQuery Client should allow a Connection object to be passed in

[vimota]

We (at Kaggle) support a BigQuery Proxy which allows users of our managed Notebooks (Kernels) to query BigQuery without any setup and for free. We do this by allowing them to query public datasets using our own credentials, injected in by a proxy we run that receives requests from the client and forwards it to BigQuery. To support this in the BigQuery client the users use, we have to monkeypatch the client and update the internal instance variable _connection to our own subclass of it (https://github.com/Kaggle/docker-python/blob/2653b37c15556996c964fc6cd33197030c4a4228/patches/kaggle_gcp.py#L48).

It would be less hacky if we could pass the Connection object in the client constructor like we do the Credentials object, or be able to set it as a class variable when we subclass Client.

[tseaver]

@tswast Can you comment?

[tswast]

@vimota Is the main purpose of the connection object to pass in extra HTTP headers for your proxy? Is this the sort of thing necessary for other kinds of proxies?

CC @alixhami

[vimota]

@tswast that, and also overriding the URL of the Connection: https://github.com/Kaggle/docker-python/blob/2653b37c15556996c964fc6cd33197030c4a4228/patches/kaggle_gcp.py#L25. Thanks!

[tswast]

In general we suggest customers set the HTTPS_PROXY environment variable per http://docs.python-requests.org/en/master/user/advanced/#proxies , but I see that may not be sufficient in your case due to the extra header and the fact this isn't a traditional HTTP proxy.

I'd like for there to be some way to reconcile this with how the gRPC-based clients work rather than make _connection a parameter to the constructor. For example, the transport argument may be a good candidate.

google-cloud-python/bigquery_datatransfer/google/cloud/bigquery_datatransfer_v1/gapic/data_transfer_service_client.py

Lines 133 to 140 in 4a0131b

	transport (Union[~.DataTransferServiceGrpcTransport,
	Callable[[~.Credentials, type], ~.DataTransferServiceGrpcTransport]): A transport
	instance, responsible for actually making the API calls.
	The default transport uses the gRPC protocol.
	This argument may also be a callable which returns a
	transport instance. Callables will be sent the credentials
	as the first argument and the default transport class as
	the second argument.

Transport even takes a credentials object and an address, similar to what you'd like.

google-cloud-python/bigquery_datatransfer/google/cloud/bigquery_datatransfer_v1/gapic/transports/data_transfer_service_grpc_transport.py

Lines 47 to 52 in 4a0131b

	credentials (google.auth.credentials.Credentials): The
	authorization credentials to attach to requests. These
	credentials identify this application to the service. If none
	are specified, the client will attempt to ascertain the
	credentials from the environment.
	address (str): The address where the service is hosted.

Since the Connection object for our hand-written HTTP-based clients is similar to the XTransport object in our gRPC-based clients, maybe we add a "transport" argument that accepts a Connection. Unfortunately they are different in one material way: in gRPC clients it is an error to supply both a transport and credentials object (since transport contains the credentials object). @crwilcox @tseaver WDYT?

[vimota]

I filed a related issue but regarding Magics: #8024.