AttributeError: 'NoneType' object has no attribute '_http' when using Blob.upload_from_string

[Ark-kun]

Environment details

• OS type and version: Linux
• Python version: 3.9.2
• pip version: 21.2.4
• google-cloud-storage version: 1.42.0

Code example

from google.cloud import storage
storage.Blob.from_string(uri="gs://avolkov/mlflow/Model1/model_info.json").upload_from_string("foo")

Stack trace

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/local/google/home/avolkov/.local/lib/python3.9/site-packages/google/cloud/storage/blob.py", line 2853, in upload_from_string
    self.upload_from_file(
  File "/usr/local/google/home/avolkov/.local/lib/python3.9/site-packages/google/cloud/storage/blob.py", line 2580, in upload_from_file
    created_json = self._do_upload(
  File "/usr/local/google/home/avolkov/.local/lib/python3.9/site-packages/google/cloud/storage/blob.py", line 2382, in _do_upload
    response = self._do_multipart_upload(
  File "/usr/local/google/home/avolkov/.local/lib/python3.9/site-packages/google/cloud/storage/blob.py", line 1865, in _do_multipart_upload
    transport = self._get_transport(client)
  File "/usr/local/google/home/avolkov/.local/lib/python3.9/site-packages/google/cloud/storage/blob.py", line 809, in _get_transport
    return client._http
AttributeError: 'NoneType' object has no attribute '_http'

[Ark-kun]

I'm encountering this error almost everywhere. Even in cases where I initially start with storage.Client():

from google.cloud import storage
client = storage.Client()
bucket = storage.Bucket.from_string("gs://avolkov")
blobs = client.list_blobs(
    bucket_or_name=bucket,
    prefix="mlflow/Model4/",
)
for blob in blobs:
    blob.bucket.rename_blob(blob, blob.name + ".bak")

   blob.bucket.rename_blob(blob, new_path)
  File "/usr/local/google/home/avolkov/.local/lib/python3.9/site-packages/google/cloud/storage/bucket.py", line 1988, in rename_blob
    new_blob = self.copy_blob(
  File "/usr/local/google/home/avolkov/.local/lib/python3.9/site-packages/google/cloud/storage/bucket.py", line 1868, in copy_blob
    copy_result = client._post_resource(
AttributeError: 'NoneType' object has no attribute '_post_resource'

I think this bug needs to be fixed ASAP.

[shaffeeullah]

@Ark-kun To resolve the initial issue, please specify the client in upload_from_string. Try:

from google.cloud import storage
client = storage.Client()
storage.Blob.from_string(uri="gs://avolkov/mlflow/Model1/model_info.json").upload_from_string("foo", client=client)

Does that work for you?

I am unable to reproduce the second issue that you listed. Is there any more information that you can provide?

[Ark-kun]

To resolve the initial issue, please specify the client in upload_from_string.

Thank you for the workaround. I'll use it until there is a fix on the library side.

Do we have consensus that this is a bug in the library?

I am unable to reproduce the second issue that you listed. Is there any more information that you can provide?

I've updated the repro. It looks like the bug only occurs when the you pass a Bucket to list_blobs, not a bucket name.

from google.cloud import storage
client = storage.Client()
bucket = storage.Bucket.from_string("gs://avolkov")
blobs = client.list_blobs(
    bucket_or_name=bucket,
    prefix="mlflow/Model4/",
)
for blob in blobs:
    blob.bucket.rename_blob(blob, blob.name + ".bak")

[tseaver]

Allowing client to be optional on these two methods was discussed here. Basically, client=None exists only to allow passing a URI string to a client method which would otherwise require a Blob or Bucket instance. If you are constructing one yourself using these methods, you should always pass the client argument.

PR #540 updates the docstring examples for Blob.from_string and Bucket.from_string to show passing an explicit client argument.

[Ark-kun]

I'm working on reducing Google Cloud onboarding friction and this looks pretty confusing from the user perspective. Especially the second case where I actually use client.
I've also noticed that the errors are not human-intended and not actionable.

I think we can pretty easily improve the user experience here by fixing the _require_client function.

Let me make a PR.

[tseaver]

@Ark-kun FWIW, the more idiomatic way to write your first examples above would be:

from google.cloud import storage

client = storage.Client()
blob = storage.Blob.from_string(uri="gs://avolkov/mlflow/Model1/model_info.json", client=client)
blob.upload_from_text("foo")  # note 'Blob.upload_from_string' is deprecated

or even more idiomatic:

from google.cloud import storage

client = storage.Client()
bucket = client.bucket("avolkov")
blob = bucket.blob("mlflow/Model1/model_info.json")
blob.upload_from_text("foo")

For the second example:

from google.cloud import storage

client = storage.Client()
bucket = storage.Bucket.from_string("gs://avolkov", client=client)
blobs = bucket.list_blobs(
    prefix="mlflow/Model4/",
)
for blob in blobs:
    bucket.rename_blob(blob, blob.name + ".bak")

or, even more idiomatic:

from google.cloud import storage

client = storage.Client()
bucket = client.bucket("avolkov")
for blob in bucket.list_blobs(
    prefix="mlflow/Model4/",
):
    bucket.rename_blob(blob, blob.name + ".bak")

Basically, never, ever call Blob.from_string or Bucket.from_string without passing an explicit client.

[cojenco]

Based on #545 and previous discussion here, closing this issue with fixes in #545. If there's something that I missed please reopen.