Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Failure to account for None-returning generators in StopIteration.args #18073

Closed
dvarrazzo opened this issue Oct 30, 2024 · 4 comments
Closed
Labels
bug mypy got something wrong

Comments

@dvarrazzo
Copy link

dvarrazzo commented Oct 30, 2024

In psycopg 3 we have generators returning a final result, which sometimes is None, sometimes a different object. We also have some consumers of these generators, which wait for I/O completion according to the values yielded by the generators and finally return the final generator result.

Something similar can be represented with the following code:

from typing import Any, Generator, TypeAlias, TypeVar

RV = TypeVar("RV")

PQGen: TypeAlias = Generator[Any, Any, RV]


def str_gen() -> PQGen[str]:
    yield 10
    return "abc"


def none_gen() -> PQGen[None]:
    yield 10
    return None


def wait(gen: PQGen[RV]) -> RV:
    try:
        while True:
            next(gen)
    except StopIteration as ex:
        rv: RV = ex.args[0] if ex.args else None
        return rv


print(f"{wait(str_gen())=}")
print(f"{wait(none_gen())=}")

If a generator returns None, the StopIteration.args tuple will empty, which is accounted for in the wait()'s return statement (removing the check will cause an IndexError on wait(none_gen())). However, since Mypy 1.12, the return statement in the wait returns an error:

bug_mypy.py:24: error: Incompatible types in assignment (expression has type "Any | None", variable has type "RV")  [assignment]

This wasn't reported as error in Mypy 1.11

@dvarrazzo dvarrazzo added the bug mypy got something wrong label Oct 30, 2024
dvarrazzo added a commit to psycopg/psycopg that referenced this issue Oct 30, 2024
If a generator returns None, the StopIteration.args tuple is empty. The
fact we explicitly checks for it and in that case we return none looks
now an error for Mypy.

I don't see an obvious way around it.

Maybe related to python/mypy#1933 which is an
historical issue (opened in 2016).

Reported in python/mypy#18073
@JelleZijlstra
Copy link
Member

This seems related to #17427, and the error looks correct to me. The function is declared as returning RV, but it may return None, as mypy correctly points out.

@brianschubert
Copy link
Collaborator

brianschubert commented Oct 30, 2024

What Jelle said. Note that there’s no connection in the type system between RV and the type of ex. Mypy has no way of knowing that "ex.args is empty at runtime" means "RV must be None", which is what that assignment assumes. Usually, in order to inject that sort of runtime knowledge into the type system, you’d need to use a cast or a narrowing construct (asserts, type guards, etc.).

Also, you may find it easier to use StopIteration.value (which is meant for this purpose) instead of trying to parse BaseException.args. value has a type of Any, so for example writing rv: RV = ex.value will type check.

dvarrazzo added a commit to psycopg/psycopg that referenced this issue Oct 30, 2024
…ting

Work around the issue reported in python/mypy#18073

Thank you very much, @brianschubert, for pointing it out!
@dvarrazzo
Copy link
Author

dvarrazzo commented Oct 30, 2024

Yes, the primer in #17427 indeed shows the changes in psycopg check.

Observation: If expr is Any, isn't "expr if cond else None" an Any too?

However, thank you very much for pointing out the use of StopIteration.value, @brianschubert! I missed it, and it seems we can use it instead of args[0] to work around the problem. I think this issue can be closed.

@JelleZijlstra
Copy link
Member

Observation: If expr is Any, isn't "expr if cond else None" an Any too?

No, it is Any | None, which is a different type. For example, accessing arbitrary attributes is OK on a value of type Any, but not on a value of type Any | None.

dvarrazzo added a commit to psycopg/psycopg that referenced this issue Nov 14, 2024
If a generator returns None, the StopIteration.args tuple is empty. The
fact we explicitly checks for it and in that case we return none looks
now an error for Mypy.

I don't see an obvious way around it.

Maybe related to python/mypy#1933 which is an
historical issue (opened in 2016).

Reported in python/mypy#18073
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug mypy got something wrong
Projects
None yet
Development

No branches or pull requests

3 participants