asyncio support

[graingert]

I want to use google-cloud-python in a Sanic web service, however I'm having to use run_in_executor() to use this library without stalling the eventloop.

This is much less efficient than an Asyncio native client.

[graingert]

@jonparrott it would be possible to support python 2.x and python 3.x with various event loops from the same source by using a custom deferred, future or IOAction implementation.

[theacodes]

This might be possible once we exorcise httplib2 from this project. @lukesneeringer

[lukesneeringer]

That sounds right to me (and that has moved up in priority slightly).

Tagging #1998 for reference.

[graingert]

the core code would use Action composition:

return get_url(xys).map(process_request).flat_map(make_another_request)

Then there could be various, pluggable http transport implementations, that would take an Action instance and return a Future from various event systems (asyncio, twisted), or even use blocking IO.

[graingert]

import asyncio
import functools


class AsyncIOAction(object):
    def __init__(self, fn):
        this.exec = fn

    def map(self, fn):
        @functools.wraps(fn)
        async def exec():
            value = await self.exec()
            return fn(value)

        return AsyncIOAction(exec)

    def flat_map(self, fn):
        @functools.wraps(fn)
        async def exec():
            value = await self.exec()
            return await fn(value).exec()

        return AsyncIOAction(exec)

    @classmethod
    def all(items):
        async def exec():
            return await asyncio.wait([x.exec() for x in items])

        return AsyncIOAction(exec)

def aiohttp_action(request):
    async def exec():
        return await aiohttp.request(**request)

    return AsyncIOAction(exec)
        
# blocking.py
import functools

class BlockingIOAction(object):
    def __init__(self, fn):
        this.exec = fn

    def map(self, fn):
        @functools.wraps(self, fn)
        def exec():
            value = self.exec()
            return fn(value)
        return BlockingIOAction(exec)

    def flat_map(self, fn):
        @functools.wraps(fn)
        def exec():
            value = self.exec()
            return fn(value).exec()
        
        return BlockingIOAction(exec)

    @classmethod
    def all(items):
        def exec():
            return [x.exec() for x in items]

        return BlockingIOAction(exec)

def requests_action(request): # or httplib2 etc.
    def exec():
        requests.request(**request)

    return BlockingIOAction(exec)

# twisted.py

class TwistedIOAction(object):
    ...

Should be pretty easy with twisted deferred too. Idea being, calling code would choose an IO type, configure the google cloud platform library appropriately, call a method on some API endpoint and then be able to call .exec() on the returned action. The calling code would then be able to receive the value, await and receive the value, or yield it in twisted inline callbacks.

[graingert]

Here's how you might to inline callbacks: https://gist.github.com/graingert/ca6cdd5d9ae2e18ca917b4594ac8a633#file-asyncio_and_blocking_io-py-L141-L148

[graingert]

inside google code platform:

def some_api_method(IOAction, ham, spam):
    @IOAction.inline_callbacks
    def _some_api_method():
        v = yield some_other_method(IOAction, ham)
        v2 = yield another_method(IOAction, spam, v)
        Return(v + v2)
        
return _some_api_method()

In some calling code:

async def async_caller():
    await some_api_method(AsyncIOAction, ham="ham", spam="spam").exec()

def blocking_caller():
    some_api_method(BlockingIOAction, ham="ham", spam="spam").exec()

[graingert]

of course https://pypi.python.org/pypi/txaio might be a better way of implementing it.

[theacodes]

@graingert oh that's super interesting. However, this seems to re-enforce the fact that we can't really support this without breaking our public interface and without seriously re-thinking all of our internals. We'd have to start from scratch with this in mind. :/

[graingert]

@jonparrott with crossbario/txaio#113 and crossbario/txaio#110

you probably won't have to change that much.

And you'd be able to keep the same public api for people using requests.

[theacodes]

inlinecallbacks would help, but this is an enormous undertaking and a lot of our code that deals with IO is non-trivial (resumable-media).

I'm open to doing this, but from my perspective it seems like the tooling isn't quite ready and it would take an enormous amount of engineering effort to get all of our plumbing ready- plus, we have a consider how gRPC plays into all this.

[dhermes]

FWIW, resumable-media would be the easiest thing of all to add support for because the actual I/O is "isolated" (I took the sans-I/O approach when writing it)

[graingert]

@jonparrott @dhermes yeah a sans-io approach for this whole library would be the best option!

[theacodes]

then it might be a good place to proof-of-concept this?

[dhermes]

@jonparrott I'm happy to review PRs but I don't have cycles.

As an example, take a look at the very small amount of code in ResumableUpload.transmit_next_chunk:

        method, url, payload, headers = self._prepare_request()
        result = _helpers.http_request(
            transport, method, url, data=payload, headers=headers,
            retry_strategy=self._retry_strategy)
        self._process_response(result, len(payload))
        return result

Every public method is split into "prepare" and "process" stages in the same way.

[lukesneeringer]

I am going to move this to resumable-media.

[lukesneeringer]

This issue was moved to googleapis/google-resumable-media-python#28

[ncouture]

#4513 (Firebase/Firestore listen/watch/server-sent events) brought me here as it seems like a good candidate for using asyncio;

@lukesneeringer should there be medium or long-term plans for using asyncio in google-cloud-python?

[theacodes]

There really isn't a feasible way for us to support asyncio until we can drop Python 2 support.

[graingert]

You can support twisted and that works with asyncio

…

On Tue, 15 May 2018, 07:01 Thea Flowers, ***@***.***> wrote: There really isn't a feasible way for us to support asyncio until we can drop Python 2 support. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#3103 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAZQTGp4O6WL-oNGedf0QVrOGHuynEwUks5tym8ngaJpZM4MUZWv> .

[theacodes]

Supporting twisted is essentially removing our entire underlying transport and replacing it, not to mention replacing all of the layers above that with async layers. That doesn't even get into grpc. It's just not feasible while we still have to support Python 2.

I would love to be pushing the edge of the boundaries here, but it's really not commercially viable for us to drop Python 2, re-architect our GA libraries, or force our users to use a large dependency like Twisted (we already have enough on our hands with grpc).

[systocrat]

How feasible would it be for a third party to implement functionality of certain services with asyncio or Twisted on their own? Is the documentation there for it?

[vladyslav2]

@theacodes Thea, why don't you want to create a new repo with support asyncio and without python2?

In that case, your clients will be able to choose which way they are want to move.
And by the way - some of us too commercial clients and already been using asyncio for a few years!
I'm sad to see google does not have any plans to support it

[theacodes]

Simply but we don't have the resources to maintain two divergent forks. Once we're able to drop 2.7 support (2019/2020) we can start adopting stuff like asyncio.

…

On Mon, Jul 30, 2018, 5:23 PM Vladysav Tarasenko ***@***.***> wrote: @theacodes <https://github.com/theacodes> Thea, why don't you want to create a new repo with support asyncio and without python2? In that case, your clients will be able to choose which way they are want to move. And by the way - some of us too commercial clients and already been using asyncio for a few years! I'm sad to see google does not have any plans to support it — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#3103 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAPUc9L7zkB77ZvniF8TSKFhe3hyElHfks5uL6OQgaJpZM4MUZWv> .

[vladyslav2]

@theacodes why do you believe supporting python 3 will be so significant big task?
Are you willing to accept help from a community? I just check iot-core (stuff that i'm interested) and most of calls based on the paho.mqtt.python library which does support asyncio.
So, i'm expecting I can port google-cloud-python/iot in few days since you have tests coverage

[theacodes]

It's not that supporting python 3 is a significant task (we *do* support Python 3 after all), it's that maintaining 2 separate, divergent codebases for 40+ libraries isn't feasible. It's much more realistic for us to wait until we can remove 2.7 support. One of the reasons these libraries are open so is precisely to empower users to fork and extend as much as they want. So you're welcome to do so, just note that any support for that will be best-effort.

…

On Mon, Jul 30, 2018, 5:55 PM Vladysav Tarasenko ***@***.***> wrote: @theacodes <https://github.com/theacodes> why do you believe supporting python 3 will be so significant big task? Are you willing to accept help from a community? I just check iot-core (stuff that i'm interested) and most of calls based on the paho.mqtt.python library which does support asyncio. So, i'm expecting I can port google-cloud-python/iot in few days since you have tests coverage — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#3103 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAPUc6gh7LxtKsCRLLiYaydryDgdpuNHks5uL6r6gaJpZM4MUZWv> .

[alexpirine]

Please drop support for 2.7 now and add asyncio support :)
Blocking networking calls is such a slow, has-been thing…

[rubik]

What is the status of this issue? Will it be fixed in the near future?

[tseaver]

@rubik Dropping 2.7 support is scheduled for Q1 2020. Adding asyncio support will be a major effort, but the relative priority for it isn't set, AFAIK.

[kamyar]

Is there any development on this front?
Also can you please reopen this issue as it is mentioned as a blocker for firebase/firebase-admin-python#104 ?
We have been using firestore and lack of async is a very big downside of firestore.
If async supported is not planned for google services, It would be nice if we know beforehand so we can plan our migration to services that support it. Lack of this feature is causing us both money and product quality tbh.

[software-dov]

API core is adding asyncio support right now (see here for details), but it's tough to say how long before client libraries providing asynchronous interfaces will be available.

[ghost]

So whats going on with this? Did it make happen after half a year or not?

[ntoshev]

This is sorely needed, @lukesneeringer. An update will be greatly appreciated.

I'm trying to work it around using https://github.com/talkiq/gcloud-aio

[NoamNol]

What do you think about Aiogoogle?

[adriangb]

FWIW one workaround I've been using is generating presigned URLs for myself and then using them with httpx. But that compounds with other issues, namely #922. Python 2 has been EOL for 2 years now.

[zagfai]

We make our lib which support asyncio for spanner open source: https://pypi.org/project/aspanner/