Vision: Google Cloud Storage Client project=None bootstrapping problem

[safi-manar]

Hello,

In using the Google Cloud API's, I noticed a small discrepancy in the documentation's expected behavior, and the actual behavior.
The documentation and Google cloud examples code typically assumes that credentials should be created using a JSON key store saved on persistent storage. However, this isn't always an option, as with deployments using ephemeral storage. Luckily, a Credentials object can be created directly from a dictionary, which houses the same mapping that the JSON keystore would hold. This is the procedure that I use in my project (the specifics of which, is shown below).

The Storage.Client documentation suggests that the project parameter is optional, and thus may be None. We found, however, that when it is not specified, an error is thrown, as a result of a sort of bootstrapping problem in which the Google Cloud APIs check to see if the project is None, and if so, attempt to create it (which is impossible, without first having the keys).
To elaborate:

1. When we make a call to Storage.Client(), we step into storage/google/cloud/storage/client.py.
2. This class calls super(Client, self).__init__(project=project, credentials=credentials, _http=_http), which takes us to core/google/cloud/client.py .
3. This calls _ClientProjectMixin.__init__(self, project=project), where we see the following call:

        project = self._determine_default(project)

4. Following this call, we see:

    if project is None:
        _, project = google.auth.default()

So, there is an attempt to infer the project parameter based on the environment credentials. However, as discussed above, we are using a dictionary-created Credentials object as our credentials, so no such credentials exist in the environment. This is a problem because the documentation claims that project is optional, when, in fact, it is not if the project is not stored in the environment.

Note, however, that this seems to be a problem specific with Storage.Client(), and not other API clients. Trying this procedure (see the code below) for google.cloud.vision.ImageAnnotatorClient seems to work perfectly fine with just a credentials object, without the project argument.

---

1. OS: Ubuntu 16.04

2. Python Version: Python 2.7.10

3. Google APIs:
   google-auth==1.1.1
   google-cloud-core==0.27.1
   google-cloud-storage==1.4.0
   google-cloud-vision==0.27.0
   google-gax==0.15.15
   google-resumable-media==0.2.3
   googleapis-common-protos==1.5.3
   oauth2client==4.1.2

4. Stacktrace:

Traceback (most recent call last):
  File "storage_client.py", line 23, in <module>
    client = _get_client()
  File "storage_client.py", line 18, in _get_client
    client = storage.Client(credentials=creds)
  File "/Users/admin/testproject/venv/lib/python2.7/site-packages/google/cloud/storage/client.py", line 59, in __init__
    _http=_http)
  File "/Users/admin/testproject/venv/lib/python2.7/site-packages/google/cloud/client.py", line 211, in __init__
    _ClientProjectMixin.__init__(self, project=project)
  File "/Users/admin/testproject/venv/lib/python2.7/site-packages/google/cloud/client.py", line 165, in __init__
    project = self._determine_default(project)
  File "/Users/admin/testproject/venv/lib/python2.7/site-packages/google/cloud/client.py", line 178, in _determine_default
    return _determine_default_project(project)
  File "/Users/admin/testproject/venv/lib/python2.7/site-packages/google/cloud/_helpers.py", line 179, in _determine_default_project
    _, project = google.auth.default()
  File "/Users/admin/testproject/venv/lib/python2.7/site-packages/google/auth/_default.py", line 286, in default
    raise exceptions.DefaultCredentialsError(_HELP_MESSAGE)
google.auth.exceptions.DefaultCredentialsError: Could not automatically determine credentials. Please set GOOGLE_APPLICATION_CREDENTIALS or
explicitly create credential and re-run the application. For more
information, please see
https://developers.google.com/accounts/docs/application-default-credentials.

5. Steps to reproduce:

• So, create an OAuth2 credentials object from a python dictionary (let's call it ketfile_dict using google.oauth2.service_account.Credentials.from_service_account_info(keyfile_dict).

• Create an instance of a storage client using google.cloud.storage.Client(credentials=creds)

• Although the documentation states that the "project" argument is optional, the API will in fact throw an error when no project argument is presented.

6. Code example

Assume that in the same directory, you create a file, constants.py, with the following fields defined from the appropriate fields of the service account key JSON.

constants.GCP_SCOPE
constants.GCP_TYPE
constants.GCP_CLIENT_EMAIL
constants.GCP_PRIVATE_KEY
constants.GCP_PRIVATE_KEY_ID
constants.GCP_CLIENT_ID
constants.GCP_TOKEN_URI
constants.GCP_PROJECT_ID

from google.cloud import storage
from google.oauth2 import service_account
import constants


# Return an Oauth2 credentials instance
def get_credentials(with_keyfile=False):
    keyfile_dict = _get_keyfile_dict()
    scope = [constants.GCP_SCOPE]
    creds = service_account.Credentials.from_service_account_info(keyfile_dict).with_scopes(scope)

    if not with_keyfile:
        return creds
    else:
        # Return creds and the keyfile.
        return creds, keyfile_dict


# Construct a dictionary of the required key:value pairs.
def _get_keyfile_dict():
    keyfile_dict = {}
    keyfile_dict['type'] = constants.GCP_TYPE
    keyfile_dict['client_email'] = constants.GCP_CLIENT_EMAIL
    keyfile_dict['private_key'] = constants.GCP_PRIVATE_KEY
    keyfile_dict['private_key_id'] = constants.GCP_PRIVATE_KEY_ID
    keyfile_dict['client_id'] = constants.GCP_CLIENT_ID
    keyfile_dict['token_uri'] = constants.GCP_TOKEN_URI
    keyfile_dict['project_id'] = constants.GCP_PROJECT_ID
    return keyfile_dict


def _get_client():
    creds, keyfile_dict = get_credentials(with_keyfile=True)
    project = keyfile_dict['project_id']
    #client = storage.Client(project=project, credentials=creds) # This WILL work because a project is passed in.
    #client = vision.ImageAnnotatorClient(credentials=creds) #Note that the Vision client will work without a project argument
    client = storage.Client(credentials=creds) # This will throw an error because project=None.
    return client


# Instantiates a client
client = _get_client()

Let me know if you have any other questions. :)

[tseaver]

@jonparrott Is this just a documentation issue, or should the client be willing to infer the project from the passed credentials?

[theacodes]

This is interesting, because storage effectively doesn't need a project to operate. Buckets are global resources.

[tseaver]

Places which touch client.project:

• list_buckets is the only client method which uses the project: I guess we could add an optional project argument, falling back to the client's project. Likely we should raise an exception if neither is set: project is a required parameter for 'buckets.list'.
• Bucket.create is the only bucket method which uses it: it is likewise required, so we could allow an optional project argument and fall back / fail in the same way. We would also need to plumb it through to Client.create_bucket.
• BucketNotification.__init__ takes and an optional topic_project parameter, and falls back to the client's project. We would need to have it raise if neither was set.

To implement this, we could have the default for the project argument be a marker, and then treat None specially (pass through a dummy to the super() version and then assign None.

[theacodes]

Sounds reasonable, @tseaver.