Add support for batch requests in gcloud.storage

[jgeewax]

Or wait for the new primitive client that simplifies this.

[tseaver]

Is there a server-side API feature that this issue refers to?

[silvolu]

https://cloud.google.com/storage/docs/json_api/v1/how-tos/batch

Not sure if apitools helps with this.

[tseaver]

Cool, I had missed that doc.

[silvolu]

It's kinda easy to get lost in the Cloud Encyclopedia :D

[tseaver]

I think we would end up implementing this by subclassing Connection, overriding api_request such that it stored params for non-idempotent HTTP requests (anything other than GET) (similar to the Transaction's mutation in datastore).

I don't know how it should handle interleaving reads: write-after-read can lead to inconsistent state without some help from the server to isolate other changes.

Another approach would be to use a pattern like #352, where the context manager hooks api_request to defer the writes.

[tseaver]

OK, here's a plan:

• Move datastore.batch._Batches (which is just a thread-local stack) up to gcloud._localstack. Move the 'thread-local stack' implementation up from datastore. #622
• Add storage.batch.Batch, paralleling datastore.batch.Batch, and exposing the same methods as storage.connection.Connection. This class will "store up" the mutations and generate the multipart/mixed payload defined in Sending Batch Requests. As with the datastore variant, this Batch should be usable as a context manager, as well as in "imperative" mode (where it will need to be passed into the buckets as their connection?). #15: Add storage.batch.Batch #654
• Update storage.api to look first for the active batch, falling back to the implied default connection.
• Update Bucket.__init__() to look first for the active batch, but fall back (per Should Blob and Bucket constructors fall back to _implicit_environ defaults? #621).

[dhermes]

SGTM.

[tseaver]

@dhermes, I'll hold off on the third point until your work on #632 clarifies (I think we would be stepping all over each other there).

[dhermes]

OK cool. I sent the first PR of the oncoming flood (I haven't written the flood yet, I just know it's coming).

[dhermes]

@tseaver I checked and verified that GET requests can be sent as part of batch requests.

In order to accommodate this, we'd need to create a container object (maybe called Future, as in ndb) that we could hold onto and then populate when the batch returns.

For example:

>>> future_bucket = batch.get_bucket('BOGUS.NAME')
>>> future_bucket
<FutureBucket>
>>> future_bucket.get()
Traceback (most recent call last):
...
ValueError: Batch not finished yet.
>>>
>>> batch.finish()
>>> bucket = future_bucket.get()
>>> bucket
<Bucket: BOGUS.NAME>