Callables and __init__ methods

[Viicos]

Bug Report

I am currently working on encode/starlette#1987 where I'm defining a type hint for an argument that can take either a type of a class having a __call__ method, or a callable returning another callable. That is, both definitions work:

class ASGIMiddleware:
    def __init__(app: ASGIApp, p0: str, p1: int) -> None:
        ...
    async def __call__(self, scope, receive, send) -> None:
        ...
# or
def asgi_middleware(app, p0: str, p1: int) -> ASGIApp:
    async def _asgi(scope, receive, send) -> None:
        ...
    return _asgi

I was told that the following protocol would support both:

class MiddlewareType(Protocol):
    def __call__(self, app: ASGIApp, **options: Any) -> ASGIApp:
        ...

With the class definition, the __init__ method is checked against the __call__ method of MiddlewareType, and with the callable definition, well it is checked against the __call__ method naturally. I was surprised to see that mypy supported the first case, as it was unclear to me what was the correct definition of a callable (in fact, definitions differ in docs):

• https://docs.python.org/3/library/functions.html#callable

Return True if the object argument appears callable, False if not. If this returns True, it is still possible that a call fails, but if it is False, calling object will never succeed. Note that classes are callable (calling a class returns a new instance); instances are callable if their class has a __call__() method.

• https://docs.python.org/3/library/typing.html#typing.Callable

Callable type; Callable[[int], str] is a function of (int) -> str.

• https://docs.python.org/3/library/collections.abc.html#collections.abc.Callable

ABC for classes that provide the __call__() method.

Now if this behaviour is supported by mypy, maybe this could be clarified in this part of the docs?

I've made this issue a bug report as I'm still struggling to have the MiddlewareType type working, as **options: typing.Any doesn't seem to catch the args/kwargs following app: ASGIApp.

See this mypy-play example:

https://mypy-play.net/?mypy=0.991&python=3.11&gist=49f042e0c90bdaf202e519bd1467ce3a (works on 0.991)
https://mypy-play.net/?mypy=1.1.1&python=3.11&gist=6c7629c60546af1490ba092821564d63 (doesn't work since 1.0.0)

The last example is also returning a misleading error message:

main.py:24: note: Following member(s) of "ExampleMiddleware" have conflicts:
main.py:24: note:     Expected:
main.py:24: note:         def __call__(app: int, **options: Any) -> Callable[[str], None]
main.py:24: note:     Got:
main.py:24: note:         def __call__(self: ExampleMiddleware, el: str) -> None  # should be: def __init__(self, app: int, other_arg: Any) -> None:

Your Environment

• Mypy version used: 0.991 / > 1.0.0
• Mypy command-line flags: None
• Mypy configuration options from mypy.ini (and other config files):
• Python version used: 3.9 / 3.10 / 3.11

[hauntsaninja]

Thanks for the report!

When comparing a type object to a callback protocol, we should be comparing the type's __init__ to the protocol's __call__.

Note that ExampleMiddleware isn't a MiddleWareType because it doesn't accept arbitrary kwargs.

class MiddlewareType(Protocol):
    def __call__(self, app: int, **options: Any) -> ASGIApp:
        ...

class ExampleMiddleware:
    def __init__(self, app: int, other_arg: Any) -> None:
        pass

If you change to ExampleMiddleware.__init__ to (self, app: int, **options: Any) -> None:, your example type checks.

The change between 0.991 and 1.0 bisects back to #14121, which is not a PR I really understand. There's definitely a bug here, but it might only be a bug in the error messaging

[hauntsaninja]

Okay yeah, I think just a buggy error message. #15042 should fix

[Viicos]

Thanks for the fix! So is there a way to define the __call__ signature of the MiddlewareType protocol so that it could match a signature with app: int as the first argument, and any arbitrary kwargs following (e.g. the __init__ signature of ExampleMiddleware)? I've tried adding *args as well but with no success:

class MiddlewareType(Protocol):
    def __call__(self, app: int, *args: Any, **kwargs: Any) -> ASGIApp:
        ...

Also if you agree, I'd happy to open a PR to explicitly state in docs that checking against __init__ for callback protocols is supported 😉

[hauntsaninja]

Thanks, PR welcome! :-)

If there's a way to spell it (without adding additional nesting), I'm blanking. Maybe the best thing to do is just not put **options: Any in your protocol?

(It's sort of a similar use case to #5876 but only for a partial signature)

[Viicos]

Maybe the best thing to do is just not put **options: Any in your protocol?

Then it would fail to type check as ExampleMiddleware has other arguments.
I think the best way for me (until partial signatures may be fully supported) is to use def __call__(self, *args: Any, **kwargs: Any), or to have #14656 (comment) supported if it is indeed a bug (working on pyright)

[hauntsaninja]

Ah of course, that only works if there are defaults. I should probably go to sleep.

Great link, yeah, #14656 looks like a good way to spell it. I think it actually type checks the way you want with mypy today, other than the "unexpected ..." error at the type alias definition

[Viicos]

The only downside is that you can't enforce the arg name to be app, but I don't even know if this is possible in any way. Supporting typing with callbacks and protocols is such a hard task as there're so many different use cases! Thanks again for the quick error message fix