Introducing CallableParametersVariable

[mrkmndz]

At previous typing meetups and at the summit we have discussed plans for typing *args and **kwargs using variadic type variables.
I still think that that is a worthwhile project, but have encountered a limitation of that approach.

If we were to try to type a decorator that transforms a function's return type while leaving the parameters alone, it would be reasonable to try to define a callback protocol that could capture the *args and **kwargs like so.

Treturn = typing.TypeVar(“Treturn”)
Tpositionals = ....
Tkeywords = ...
class BetterCallable(typing.Protocol[Tpositionals, Tkeywords, Treturn]):
  def __call__(*args: Tpositionals, **kwargs: Tkeywords) -> Treturn: …

However there are some problems with trying to come up with a consistent solution for those type variables for a given callable. This problem comes up with even the simplest of callables:

def simple(x: int) -> None: ...
simple <: BetterCallable[[int], [], None]
simple <: BetterCallable[[], {“x”: int}, None]
BetterCallable[[int], [], None] </: BetterCallable[[], {“x”: int}, None]

Any time where a type can implement a protocol in more than one way that aren’t mutually compatible, we can run into situations where we lose information. If we were to make a decorator using this protocol, we have to pick one calling convention to prefer.

def decorator(
  f: BetterCallable[[Ts], [Tmap], int],
) -> BetterCallable[[Ts], [Tmap], str]:
    def decorated(*args: Ts, **kwargs: Tmap) -> str:
       x = f(*args, **kwargs) 
       return int_to_str(x)
    return decorated
@decorator
def foo(x: int) -> int:
    return x
reveal_type(foo) # Option A: BetterCallable[[int], {}, str]
                 # Option B: BetterCallable[[], {x: int}, str]
foo(7)   # fails under option B
foo(x=7) # fails under option A

The core problem here is that by default, parameters in Python can either be passed in positionally or as a keyword parameter. This means we really have three categories (positional-only, positional-or-keyword, keyword-only) we’re trying to jam into two categories.

This strongly suggests the need for a higher-level primitive for capturing all three classes of parameters. I propose this syntax:

from typing import Callable
from typing_extensions import CallableParametersVariable
Tparams = CallableParametersVariable(“Tparams”)
def decorator(f: Callable[Tparams, int]) -> Callable[Tparams, str]: …
@decorator
def foo(x: int) -> int:
    return x
reveal_type(foo) # Callable[[Named(x, int)], str]
foo(7)   # succeeds!
foo(x=7) # also succeeds!

This syntax prioritizes the experience of the consumers of decorators as it directly supports transporting all of the parameter information. However, this comes at the cost of type checking the body of a decorator function:

def decorator(f: Callable[Tparams, int]) -> Callable[Tparams, str]:
    def decorated(*args: object, **kwargs: object) -> str:
       x = f(*args, **kwargs) # error: expected Tparams, got objects
       return int_to_str(x)
    return decorated # error: expected Callable[Tparams, str]

Without separate variables for the positional and keyword arguments, there is no safe way to call a function with CallableParametersVariable parameters. However, this is a small surface of errors/unsoundness as compared to the gains we are getting from the caller-side. This also meshes with the proposal at the meetup to not require body-checking of functions involving variadics at all.

However, with some additional extensions we could type check those calls if we would like to go down that road. We would need to define operators on these variables that would look something like this:

def decorator(f: Callable[Tparams, int]) -> Callable[Tparams, str]:
    def decorated(*args: Positionals[TParams], **kwargs: Keywords[TParams]) -> str:
       x = f(*args, **kwargs) # special case on calling Tparams functions with these special types
       return int_to_str(x)
    return decorated # special case functions defined like this to be subtypes of Callable[Tparams, str]

If you were to want to type a decorator that was doing some kind of mutation on the parameter set, you would likely need to accept the restrictions imposed by the other kinds of proposed variadics, or we would need to define some sort of operation on these CallableParameterVariables.

I have already implemented this in Pyre and would appreciate it's inclusion in typing_extensions.

[mrkmndz]

Paging @ambv for potential interest :)

[JukkaL]

Something like this has been proposed previously in python/mypy#3157. I think it would be better bikeshed the feature some more before committing to a design, since the design space is pretty big.

I'd suggest creating an issue with a short spec and examples (from real code) where this would work. I'd leave out the original example with separate type variables for positional and keyword arguments (or move it to later in a section for rejected alternatives) since, as far as I know, it hasn't been seriously suggested and it confused me a bit.

Some other things have been considered previously that could be worth discussing:

• Is it important to represent decorators that add/remove arguments?
• Since the body of a function that uses CallableParametersVariable won't be type checked, should this be made explicit in the function declaration somehow?
• Previously various different names have been proposed for CallableParametersVariable. This name doesn't quite feel consistent with existing naming conventions.

Let's have the design discussion somewhere else? I guess typing-sig@ is also an alternative.

[mrkmndz]

Thanks for the pointer to python/mypy#3157! Good to know that there is already interest in this topic.

If I understand correctly (re: #435) this feature doesn't yet meet the bar for typing_extensions since there will still likely be interface level changes for this.

With that being the case, I think it would make sense for me to start drafting a pre-PEP of the form you described (better code examples, discussion of parameter mutating decorators etc.) and in the meantime make this externally usable through a new pyre_extensions PyPI package.
Do you think that that is the best course of action, or should I just go straight for a PEP?

[JukkaL]

Do you think that that is the best course of action, or should I just go straight for a PEP?

That sounds like a good plan! It's probably better to write a PEP a bit later after some discussion, though it's up to you.

[ilevkivskyi]

I am with @JukkaL here, I think this is a bit premature and needs some design discussions. I will be glad to participate (I am also going to dig through our internal code to collect more real-world examples).

[sobolevn]

Should be: f: Callable[Tparams, List[Treturn]],

[sobolevn]

Should be f: Callable[Tparams, List[Treturn]],

[gvanrossum]

@mrkmndz is this PR still relevant?

[mrkmndz]

nope!