Generator with await not considered an AsyncGenerator

[audoh-tickitto]

Environment data

• Language Server version: 2021.5.3
• OS and version: Ubuntu 20.04.2
• Python version (& distribution if applicable, e.g. Anaconda): 3.9.2, acquired via pyenv

Expected behaviour

• Inferred type should be AsyncGenerator
• Await should be allowed within a non-async function when within a generator

Actual behaviour

• Inferred type is Generator.
• 'await' only allowed within async function

Logs

N/A (should be reproducible from code snippet)

Code Snippet / Additional information

import asyncio
from typing import AsyncGenerator

async def get_value(v: int) -> int:
    await asyncio.sleep(1)
    return v + 1

def get_generator() -> AsyncGenerator[int, None]:
    return (await get_value(v) for v in [1, 2, 3])

async def test() -> None:
    print(type(get_generator()))
    async for x in get_generator():
        print(x)

asyncio.get_event_loop().run_until_complete(test())

Python result:

<class 'async_generator'>
2
3
4

Pylance error:

Note: the exact same issues seem to be present in mypy:
python/mypy#10534

[erictraut]

I'm surprised that this code runs without generating a runtime exception.

PEP 492, which introduced the async and await keywords to the Python language explicitly states:

It is a SyntaxError to use await outside of an async def function (like it is a SyntaxError to use yield outside of def function).

Likewise, PEP 530, which introduced the use of await in comprehensions, says:

We propose to allow the use of await expressions in both asynchronous and synchronous comprehensions. This is only valid in async def function body.

Perhaps you can help me understand why the Python interpreter appears to allow the use of await in this example even though it's used outside of an async def function body?

[audoh-tickitto]

That aspect surprised me as well to be honest and is more likely just lenience in the Python interpreter allowing it to run technically invalid code as if it were valid. Originally the sample code did use an async function, but I removed it once I realised it still held up.

Edit: actually, on second thought, it makes sense that await would be allowed within a generator expression in a non-async function - otherwise what you actually have is a coroutine function which resolves to an async generator (and has to be called with 'await' as in the amended example, rather than directly returning the asynchronous generator). I agree though it seems to contradict the PEP. Perhaps it's documented elsewhere.

However, with regards to comprehensions - I believe that would be referring specifically to list comprehensions, rather than generators - I don't believe there's any way a synchronous generator could involve an await, regardless of the function which called it.

Therefore, the issue of asynchronous generators still holds if you make the function async and ignore the weirdness of having an await in a non-async function:

import asyncio
from typing import AsyncGenerator


async def get_value(v: int) -> int:
    await asyncio.sleep(1)
    return v + 1


async def get_generator() -> AsyncGenerator[int, None]:
    return (await get_value(v) for v in [1, 2, 3])


async def test() -> None:
    print(type(await get_generator()))
    async for x in await get_generator():
        print(x)


asyncio.get_event_loop().run_until_complete(test())

I'll revise the original issue accordingly.

[brettcannon]

I think the code was technically valid for the def spam(): return (await x for x in range(3)) because the generator expression isn't executed upon creation, so your synchronous function is simply returning an async generator that isn't run (yet), so to Python you're just returning an object which was created in isolation and will be call elsewhere. And thanks to async generators not being a plain generator, you can't execute it without using it in something like async for which will then trigger a check in Python for using an awaitable outside of an async function and cause the TypeError.

[audoh-tickitto]

Yes, actually I think that PEP, when it refers to comprehension, means specifically list/dict/set comprehension (as the examples in that section would suggest), and not generators. Async comprehension is only legal in an async function of course because the function must await the result to be able to have a list/dict/set, but not the case with async generators.

[erictraut]

I've updated the type checking logic to allow these cases without generating false positive errors.

[jakebailey]

This issue has been fixed in version 2021.6.0, which we've just released. You can find the changelog here: https://github.com/microsoft/pylance-release/blob/main/CHANGELOG.md#202160-2-june-2021