Use ParamSpec for classmethod and staticmethod

[AlexWaygood]

Fixes #7689. Fixes #7134 (as much as we're ever going to fix, anyway). Cc. @NeilGirdhar.

[AlexWaygood]

Links to the primer hits:

• pydantic:
  • https://github.com/pydantic/pydantic/blob/c2cc3a62eb4000032604bc0e46b52c9c5eddd6f3/pydantic/_internal/_decorators.py#L195-L202
  • https://github.com/pydantic/pydantic/blob/c2cc3a62eb4000032604bc0e46b52c9c5eddd6f3/pydantic/decorators.py#L26
  • https://github.com/pydantic/pydantic/blob/c2cc3a62eb4000032604bc0e46b52c9c5eddd6f3/pydantic/main.py#L32
• sympy: https://github.com/sympy/sympy/blob/c6cb7c5602fa48034ab1bd43c2347a7e8488f12e/sympy/polys/solvers.py#L34

These are all somewhat atypical use cases. pydantic is using classmethod directly as an annotation, so will just have to change the number of parameters it has in its classmethod annotations. The sympy error is sort-of a false positive, but can also be worked around.

Pytype crashes on this PR because they don't use typeshed's builtins stub, but I don't think it's worth bugging the pytype maintainers about unless we agree that this is worth doing.

[asvetlov]

lgtm

[AlexWaygood]

@rchen152, how difficult would the pytype crash be to fix? :)

(There's no particular rush here on this one!)

[rchen152]

I believe all we need to do is copy the changes to builtins.pyi over to pytype's copy of the stub. I'll give it a try and let you know how it goes.

[rchen152]

The builtins.pyi changes from this PR have been copied to pytype and released in version 2023.03.13.

[AlexWaygood]

The builtins.pyi changes from this PR have been copied to pytype and released in version 2023.03.13.

That seems to have done the trick — thank you very much!

[github-actions]

Diff from mypy_primer, showing the effect of this PR on open source code:

pydantic (https://github.com/samuelcolvin/pydantic)
+ pydantic/_internal/_decorators.py:195: error: "classmethod" expects 3 type arguments, but 1 given  [type-arg]
+ pydantic/_internal/_decorators.py:202: error: Need type annotation for "f_cls"  [var-annotated]
+ pydantic/decorators.py:26: error: "classmethod" expects 3 type arguments, but 1 given  [type-arg]
+ pydantic/decorators.py:48: error: "classmethod" expects 3 type arguments, but 1 given  [type-arg]
+ pydantic/decorators.py:64: error: "classmethod" expects 3 type arguments, but 1 given  [type-arg]
+ pydantic/decorators.py:73: error: "classmethod" expects 3 type arguments, but 1 given  [type-arg]
+ pydantic/decorators.py:82: error: "classmethod" expects 3 type arguments, but 1 given  [type-arg]
+ pydantic/decorators.py:92: error: "classmethod" expects 3 type arguments, but 1 given  [type-arg]
+ pydantic/decorators.py:107: error: "classmethod" expects 3 type arguments, but 1 given  [type-arg]
+ pydantic/decorators.py:131: error: "classmethod" expects 3 type arguments, but 1 given  [type-arg]
+ pydantic/main.py:44: error: "classmethod" expects 3 type arguments, but 1 given  [type-arg]

sympy (https://github.com/sympy/sympy)
+ sympy/polys/solvers.py:34: error: Cannot infer type of lambda  [misc]
+ sympy/polys/solvers.py:34: error: Argument 1 to "staticmethod" has incompatible type "Callable[[Any], Any]"; expected "Callable[[Any, Any, Any, Any, Any, Any], Any]"  [arg-type]

discord.py (https://github.com/Rapptz/discord.py)
- discord/ext/commands/cog.py:500: error: Incompatible types in assignment (expression has type "Callable[..., Any]", variable has type "FuncT")  [assignment]
+ discord/ext/commands/cog.py:500: error: Incompatible types in assignment (expression has type "Callable[[VarArg(Any), KwArg(Any)], Any]", variable has type "FuncT")  [assignment]

[AlexWaygood]

@samuelcolvin, just to give you a heads-up about this — this changes the stub for classmethod so that it is now generic over two TypeVars and a ParamSpec, whereas previously it was only generic over a single TypeVar. This makes the stub much more precise, but it means you'll probably have to update a few annotations in pydantic with the next mypy release (see the mypy_primer diff just above for details).

[samuelcolvin]

Thanks @AlexWaygood for pinging me.

What's the time frame for this being released?

Also, what's your recommendation for supporting this while keeping support for older versions? I'm a bit confused about where typeshed fits in to mypy, cpython, pyright etc.

cc @dmontagu and @adriangb

[AlexWaygood]

Most of CPython has no type annotations. Even for the parts of CPython that do have type annotations, type checkers ignore the type annotations in CPython, and only look to typeshed's stubs when it comes to type hints in the stdlib.

Typeshed's stdlib stubs are vendored by mypy; mypy updates its vendored stdlib stubs to sync with typeshed's main branch about once every two weeks. As such, you won't see any change in mypy's behaviour until the next mypy release (mypy 1.2). There's no tracking issue for planning mypy 1.2 yet over at the mypy repo, so I can't tell you when mypy 1.2 will be released. I would guess it won't be released for a few weeks at least.

All other major type checkers also vendor typeshed's stdlib stubs in a very similar way. You may see pyright start to emit errors sooner than mypy, due to the fact that pyright has a much more frequent release schedule than mypy. (Again, I have no control over the mypy release process, I'm afraid! I'm not a mypy core dev.)

When mypy 1.2 comes out, supporting the latest mypy version while maintaining support for older mypy versions might, admittedly, be a little tricky here. If that's something you want to do, I'd probably advise not parameterising classmethod at all in your annotations, e.g.:

- foo: classmethod[Any]`
+ foo: classmethod

If you have --disallow-any-generics enabled in your mypy config (this setting is enabled by default if you use mypy --strict), you may need to add a type: ignore or two to silence some mypy complaints, so that mypy is happy with you using classmethod as an annotation without parameterising it.

I hope that's helpful, and sorry for the disruption caused!

[samuelcolvin]

That's really helpful, it was the vendoring that was confusing me, since I couldn't remember seeing typeshed in installed dependencies.

[srittau]

@samuelcolvin Also, as far as I can see, pydantic already pins mypy to a specific version. This is good. When you decide to update mypy, you will surely run it and fix all type annotations at that point. So, this should not break CI at unexpected times.

[samuelcolvin]

Sadly not. We pin it in our tests, but we can't stop users from installing newer versions - hence lots of problems at the moment with mypy v1.1.1.

There's no wayin packaging to say "X isn't required, but if is installed, it must be Y version".

In v2 we might add an optional dependency group just to constrain the mypy version, e.g. pydantic[mypy].

[AlexWaygood]

What's the time frame for this being released?

The tracking issue for the mypy 1.2 release has just been opened:

• Release 1.2 planning mypy#14916