Is it possible to define a protocol for a method?

[tobinus]

To avoid repeating myself too much, I sometimes like to create helper functions that return a new function which is intended to be used as a method. The "method creating function" would typically take in some arguments. It would then create a new function that receives self as its first argument along with some other arguments, while making use of the arguments that were given to the method creating function. This new function would be returned and used as any other method.

What I'm looking for, is a way to:

• Define a "method creating function" that takes some arguments and returns a new function
• …with the new function being passed the instance as its first argument (self)
• …and being properly typechecked, so that any users of the class can have their method calls checked
• …and using overloads when defining the new function's signature

Using Callable

Typing this using Callable works properly (please don't mind the examples here not justifying the use of a method creating function):

from collections.abc import Callable
from typing import Any


def create_int_converter() -> Callable[[Any, str], int]:
    return lambda self, s: int(s)


class ClassWithDynamicallyCreatedMethod:
    convert_to_int = create_int_converter()


o = ClassWithDynamicallyCreatedMethod()
converted: int = o.convert_to_int("123")
print(repr(converted))

Mypy: ✅
Python: ✅

Using callback protocol without extra self argument

However, I'd like to define overloads for the generated method. To do this, I thought I could build on the concept of "Callback protocols" as mentioned in PEP 544:

from typing import Any, Protocol


class ConverterProtocol(Protocol):
    def __call__(self, s: str) -> int: ...


def create_int_converter() -> ConverterProtocol:
    def converter(self: Any, s: str) -> int:
        return int(s)

    return converter


class ClassWithDynamicallyCreatedMethod:
    greeting = "Hi there!"
    convert_to_int = create_int_converter()


o = ClassWithDynamicallyCreatedMethod()
converted: int = o.convert_to_int("123")
print(repr(converted))

Mypy: ❌
Python: ✅

Mypy error:

test_dynamic_method_mypy.py:13: error: Incompatible return value type (got "Callable[[Any, str], int]", expected "ConverterProtocol")
test_dynamic_method_mypy.py:13: note: "ConverterProtocol.__call__" has type "Callable[[Arg(str, 's')], int]"
Found 1 error in 1 file (checked 1 source file)

Using callback protocol with extra self argument

I tried experimenting with adding an explicit self argument, in addition to the one that is needed because __call__ is a method itself:

from typing import Any, Protocol


class ConverterProtocol(Protocol):
    def __call__(self_, self: Any, s: str) -> int: ...


def create_int_converter() -> ConverterProtocol:
    def converter(self: Any, s: str) -> int:
        return int(s)

    return converter


class ClassWithDynamicallyCreatedMethod:
    greeting = "Hi there!"
    convert_to_int = create_int_converter()


o = ClassWithDynamicallyCreatedMethod()
converted: int = o.convert_to_int("123")
print(repr(converted))

Mypy: ❌
Python: ✅ (since we only changed the annotated Protocol)

Mypy error:

test_dynamic_method_mypy.py:21: error: Missing positional argument "s" in call to "__call__" of "ConverterProtocol"
Found 1 error in 1 file (checked 1 source file)

Thoughts

I first thought this would be a bug in Mypy. And maybe it is? But I'm also thinking there is a nuanced difference between how a function and a callable user object are treated with respect to the first self argument? Because if I make it so that create_int_converter() returns a proper instance of a class that implements ConverterProtocol, I do in fact not receive o as any argument, thus Python fails with the error that Mypy reported. Meanwhile, the converter() function receives o as its first argument.

I'm starting to think that the "callback protocol" approach is not usable for this usecase, because an instance of a class with a __call__ method is not functionally equivalent to a function when assigned as a class attribute. And the "callback protocol" approach wrongfully assumes those two are functionally equivalent. Is there any other way of creating a method creating function whose return value has an overloaded signature?

The following two Mypy issues may be related: python/mypy#708 and python/mypy#5485 but I don't think they address the inconsistencies the latter two examples demonstrate? I'll consider opening issues on that later.

[Akuli]

I'm starting to think that the "callback protocol" approach is not usable for this usecase, because an instance of a class with a __call__ method is not functionally equivalent to a function when assigned as a class attribute.

That's correct. When Python looks up a method from an instance, it calls __get__ to get a method object, which it then __call__()s. If there is no __get__, the object is called as is, without passing self.

Consider this, for example:

>>> class Foo:
...     def method(self):
...             print("hi")
... 
>>> f = Foo()
>>> f.method
<bound method Foo.method of <__main__.Foo object at 0x7f58edb5f310>>
>>> Foo.method
<function Foo.method at 0x7f58edb30790>
>>> Foo.method.__get__(f)
<bound method Foo.method of <__main__.Foo object at 0x7f58edb5f310>>

For function objects, the __get__ method returns a method object, which ensures that self is passed: f.method, aka Foo.method.__get__(f), behaves a lot like functools.partial(Foo.method, f). But the return value of __get__ can be anything, and __get__() can do whatever it wants instead of even calling your method. This is how @property and @staticmethod work, for example.

To tell mypy that your object isn't just a callable, but a method, you need to explain to mypy that __get__ behaves just like shown above. You can do this by adding a __get__ method to your protocol. I believe it works, but I haven't tried it. Let me know if you have trouble getting this to work, and I'll try to write some example code :)

[Akuli]

Also, I forgot to mention that types.MethodType is a joke. At runtime, it's same as types.FunctionType. In typeshed, it isn't generic, and uses a lot of Any. If you want to improve the definition in typeshed, feel free to make a pull request :)

[tobinus]

Hm, okay, thank you! Just so I understand you correctly: In the solution for the first problem, the protocols are generalized to work for any function and do not contain any information about the function signature, other than how you no longer need to provide the first argument when accessing the method as an instance member? So the function's signature is inferred from how it looks inside create_int_converter, where the @overload decorator could have been used (for a more complex example)?

Regarding pyright: Thanks for the pointer, I may try to use it. Mypy just happened to be the first static type checking tool I was introduced to in Python.

Regarding types.MethodType: I'm not sure if it is possible to make it generic in typeshed, considering subscription of types.MethodType would likely fail at runtime? So the standard library would need to implement types.MethodType.__class_getitem__ if it is not implemented already. Anyways, I don't think I can prioritize improving this in typeshed, unfortunately.

[Akuli]

the protocols are generalized to work for any function and do not contain any information about the function signature, other than how you no longer need to provide the first argument when accessing the method as an instance member? So the function's signature is inferred from how it looks inside create_int_converter

Yes, the ParamSpec is inferred from the function's signature. Apart from inferring from an existing function, I don't think there is currently a way to tell a ParamSpec to represent specific parameters. I believe ParamSpec works with @overload, but I haven't checked.

So the standard library would need to implement types.MethodType.__class_getitem__ if it is not implemented already.

CPython has previously accepted pull requests like this, but it's not super important that it exists at runtime. It's often possible to use from __future__ import annotations or if TYPE_CHECKING: to work around missing __class_getitem__ methods.

[tobinus]

Okay, thanks!

[tobinus]

I had overlooked that the __get__ method is called with its second argument being None when accessed on the class object, and not an instance. The protocols should reflect this, so I've updated the example I posted (in case anyone else finds this useful):

from __future__ import annotations

from typing import Protocol, Type, cast, overload


class WithDefaultInt(Protocol):
    default_int: int


class BoundConverter(Protocol):
    def __call__(self, s: str) -> int:
        ...


class Converter(Protocol):
    def __call__(self_, self: WithDefaultInt, s: str) -> int:
        ...

    @overload
    def __get__(
        self, obj: WithDefaultInt, objtype: Type[WithDefaultInt] | None = None
    ) -> BoundConverter:
        ...

    @overload
    def __get__(
        self, obj: None, objtype: Type[WithDefaultInt] | None = None
    ) -> Converter:
        ...

    def __get__(
        self, obj: WithDefaultInt | None, objtype: Type[WithDefaultInt] | None = None
    ) -> BoundConverter | Converter:
        ...


def create_int_converter(accept_errors: bool = False) -> Converter:
    def converter(self: WithDefaultInt, s: str) -> int:
        try:
            return int(s)
        except ValueError:
            if accept_errors:
                return self.default_int
            else:
                raise

    return cast(Converter, converter)


class ConverterCollection:
    default_int = 0
    convert_to_int = create_int_converter()
    convert_to_int_with_default = create_int_converter(True)


# Happy case:
converters = ConverterCollection()
result: int = converters.convert_to_int("123")
another_result: int = converters.convert_to_int_with_default("text")
print(repr(result), repr(another_result))  # prints 123 0


# This does not typecheck, ensuring us that our method signature is used:
str_result: str = converters.convert_to_int("456")


# Case which should fail because we are leaving out default_int:
class ConverterCollectionWithoutDefault:
    convert_to_int = create_int_converter(True)


converters_wo_default = ConverterCollectionWithoutDefault()
# This fails as it should, ensuring the method is not used with an incompatible class:
third_result: int = converters_wo_default.convert_to_int("testing")

[zoranbosnjak]

@tobinus @Akuli
I have read many issues about this problem and this discussion is the closest I could find. I am trying to use this approach in the following example:

from typing import *

class Test1:
    @overload
    def f(self, key: Literal['A']) -> int: ...
    @overload
    def f(self, key: Literal['B']) -> str: ...
    def f(self, key: Any) -> Any:
        if key == 'A': return 1
        elif key == 'B': return 'x'
        assert_never(key)

class Test2:
    def __init__(self, arg: Test1):
        self._arg = arg

    def get_arg(self) -> Test1:
        return self._arg

    # not sure how to deduce the type signature from Test1.f
    def f(self, key):
        return self.get_arg().f(key)

test2 = Test2(Test1())

# expecting the same type
reveal_type(test2.get_arg().f)
reveal_type(test2.f)

... but I could not make it work with mypy.

With this approach, I get Invalid self argument "Test2" to attribute function "f"..., which kind of makes sense.

T = TypeVar('T')

def same_as(f: T) -> Callable[[Callable[..., Any]], T]:
    def wrap(_g): # ignore wrapped function, use 'f'
        def wrapped_f(*args, **kwargs):
            return f(*args, **kwargs)
        return wrapped_f
    return wrap

class Test2:
    ...
    @same_as(Test1.f)
    def f(self, key):
        ...

But If I try with a protocol, like this:

from typing import *

P = ParamSpec("P")
R = TypeVar("R", covariant=True)

class Method(Protocol, Generic[P, R]):
    def __get__(self, instance: Any, owner: type | None = None) -> Callable[P, R]:
        ...

    def __call__(self_, self: Any, *args: P.args, **kwargs: P.kwargs) -> R:
        ...

def same_as(f: Method[P, R]) -> Callable[[Callable[..., Any]], Method[P, R]]:
    raise NotImplementedError

class Test1:
    @overload
    def f(self, key: Literal['A']) -> int: ...
    @overload
    def f(self, key: Literal['B']) -> str: ...
    def f(self, key: Any) -> Any:
        if key == 'A': return 1
        elif key == 'B': return 'x'
        assert_never(key)

class Test2:
    def __init__(self, arg: Test1):
        self._arg = arg

    def get_arg(self) -> Test1:
        return self._arg

    @same_as(Test1.f)
    def f(self, key: Any) -> Any:
        ...

test2 = Test2(Test1())

# expecting the same type
reveal_type(test2.get_arg().f)
reveal_type(test2.f)

... I get the wrong type for test2.f, like if the @overload is ignored.

test1.py:40: note: Revealed type is "Overload(def (key: Literal['A']) -> builtins.int, def (key: Literal['B']) -> builtins.str)"
test1.py:41: note: Revealed type is "def (key: Literal['A']) -> Any"

I would appreciate any suggestion how to resolve the problem, that is: test2.f to be the same as test2.get_arg().f