Use a TypeGuard for dataclasses.is_dataclass(); refine asdict(), astuple(), fields(), replace()

[AlexWaygood]

Fixes #9345. Refs python/mypy#14215

[AlexWaygood]

The pyright tests are failing due to microsoft/pyright#4339.

Links to the mypy_primer hits:

• porcupine is using dataclasses.asdict on a class that isn't actually a dataclass, but is used as the base class for many dataclasses: https://github.com/Akuli/porcupine/blob/2845a5d537d85ddd7a7b7c8cbcb3e3a4d8d924ca/porcupine/utils.py#L369-L408
• xarray-dataclasses:
  • The code where the error is being emitted is https://github.com/astropenguin/xarray-dataclasses/blob/1f9816f19dd2da69b50432add4a256ac380a1134/xarray_dataclasses/v2/specs.py#L110-L119
  • The root cause, I think, is that they're using a slightly more complex protocol than the one I've got here, which is generic around a ParamSpec: https://github.com/astropenguin/xarray-dataclasses/blob/1f9816f19dd2da69b50432add4a256ac380a1134/xarray_dataclasses/v2/typing.py#L49-L55.
• homeassistant:
  • The first error is a true positive:
    • The error is emitted here: https://github.com/home-assistant/core/blob/3d7a3afca70c0e760ffb2dbf3c72ddec24fa0c0e/homeassistant/components/p1_monitor/diagnostics.py#L45
    • coordinator.data[SERVICE_WATERMETER] is defined as having an optional type here: https://github.com/home-assistant/core/blob/3d7a3afca70c0e760ffb2dbf3c72ddec24fa0c0e/homeassistant/components/p1_monitor/__init__.py#L67
  • Haven't looked at the second one yet

[github-actions]

Diff from mypy_primer, showing the effect of this PR on open source code:

porcupine (https://github.com/Akuli/porcupine)
+ porcupine/utils.py:408: error: No overload variant of "asdict" matches argument type "EventDataclass"  [call-overload]
+ porcupine/utils.py:408: note: Possible overload variants:
+ porcupine/utils.py:408: note:     def asdict(obj: _DataclassInstance) -> Dict[str, Any]
+ porcupine/utils.py:408: note:     def [_T] asdict(obj: _DataclassInstance, *, dict_factory: Callable[[List[Tuple[str, Any]]], _T]) -> _T

hydra-zen (https://github.com/mit-ll-responsible-ai/hydra-zen)
- src/hydra_zen/structured_configs/_implementations.py:1666: error: Incompatible types in assignment (expression has type "Union[Type[Any], Callable[P, R], Callable[..., Any]]", variable has type overloaded function)  [assignment]
+ src/hydra_zen/structured_configs/_implementations.py:1666: error: Incompatible types in assignment (expression has type "Union[Type[_DataclassInstance], Type[Any], Callable[P, R], Callable[..., Any], DataClass_]", variable has type overloaded function)  [assignment]

xarray-dataclasses (https://github.com/astropenguin/xarray-dataclasses)
+ xarray_dataclasses/v2/specs.py:115: error: Argument 1 to "fields" has incompatible type "Type[DataClass[P]]"; expected "Union[_DataclassInstance, Type[_DataclassInstance]]"  [arg-type]
+ xarray_dataclasses/v2/specs.py:115: note: Only class variables allowed for class object access on protocols, __dataclass_fields__ is an instance variable of "DataClass"
+ xarray_dataclasses/v2/specs.py:115: note: ClassVar protocol member _DataclassInstance.__dataclass_fields__ can never be matched by a class object
+ xarray_dataclasses/v2/specs.py:158: error: Argument 1 to "fields" has incompatible type "Type[DataClass[P]]"; expected "Union[_DataclassInstance, Type[_DataclassInstance]]"  [arg-type]
+ xarray_dataclasses/v2/specs.py:158: note: Only class variables allowed for class object access on protocols, __dataclass_fields__ is an instance variable of "DataClass"
+ xarray_dataclasses/v2/specs.py:158: note: ClassVar protocol member _DataclassInstance.__dataclass_fields__ can never be matched by a class object

core (https://github.com/home-assistant/core)
+ homeassistant/components/p1_monitor/diagnostics.py:45: error: Argument 1 to "asdict" has incompatible type "Optional[Any]"; expected "_DataclassInstance"  [arg-type]
+ homeassistant/components/zwave_js/discovery.py:78: error: No overload variant of "asdict" matches argument type "DataclassMustHaveAtLeastOne"  [call-overload]
+ homeassistant/components/zwave_js/discovery.py:78: note: Possible overload variants:
+ homeassistant/components/zwave_js/discovery.py:78: note:     def asdict(obj: _DataclassInstance) -> Dict[str, Any]
+ homeassistant/components/zwave_js/discovery.py:78: note:     def [_T] asdict(obj: _DataclassInstance, *, dict_factory: Callable[[List[Tuple[str, Any]]], _T]) -> _T

[AlexWaygood]

@stroxler / @pradeep90: from @thomkeh's experiments in #9345, it looks like pyre doesn't currently synthesize a __dataclass_fields__ ClassVar for dataclass classes in the same way that mypy and pyright do. That means that if this PR were to be merged, it might cause false positives for pyre users. Would you be okay with it being merged, or would you rather we held off for now?

Same question for @rchen152 -- not sure if pytype synthesizes this attribute for dataclasses or not? Would merging this cause issues for pytype?

[rchen152]

Yep, pytype synthesizes the __dataclass_fields__ attribute! Somewhat amusingly, the entire reason we added it was to get this pytype_extensions.Dataclass protocol working: https://github.com/google/pytype/blob/8d0970083128ffc7f6042b08fad076f8c2ee01a6/pytype_extensions/__init__.py#L67

[tmke8]

Should typing_extensions also get a Dataclass protocol then? 🤔

[AlexWaygood]

Yep, pytype synthesizes the dataclass_fields attribute!

Brilliant, thanks @rchen152!

Should typing_extensions also get a Dataclass protocol then? 🤔

Maybe. But it's probably better to add it to typeshed first and wait a while (a few months, at least) to see if it's stable before adding it to typing_extensions.

[pradeep90]

@stroxler / @pradeep90: from @thomkeh's experiments in #9345, it looks like pyre doesn't currently synthesize a __dataclass_fields__ ClassVar for dataclass classes in the same way that mypy and pyright do. That means that if this PR were to be merged, it might cause false positives for pyre users. Would you be okay with it being merged, or would you rather we held off for now?

Yes, feel free to merge this change.

Pyre doesn't currently synthesize a __dataclass_fields__ ClassVar for dataclasses. I'll fix it on the Pyre side. We vendor a typeshed version manually anyway, so we will ensure the fix has landed before we upgrade next.

[cdce8p]

I've been working on updating Home Assistant for these changes. Some notes

There are a couple of places where I needed to use the dataclass protocol itself. One example are dataclass mixin classes. From that I think it would make sense to either include it in typing_extensions (like others suggested already) or in _typeshed.

At least for mypy, pydantic dataclasses don't seem to work, yet. Haven't had the time to look into it.

import dataclasses
from pydantic.dataclasses import dataclass

@dataclass
class Info:
    var: str

def func(info: Info) -> None:
    dataclasses.asdict(info)

t:9: error: No overload variant of "asdict" matches argument type "Info"  [call-overload]
t:9: note: Possible overload variants:
t:9: note:     def asdict(obj: _DataclassInstance) -> Dict[str, Any]
t:9: note:     def [_T] asdict(obj: _DataclassInstance, *, dict_factory: Callable[[List[Tuple[str, Any]]], _T]) -> _T

Tested with pydantic==1.10.4. It's quite common that an upstream library uses pydantic dataclasses (-> Info), whereas the caller uses dataclasses (-> func).

[tmke8]

As long as pydantic's dataclass has the __dataclass_fields__ attribute, it should work, right?

EDIT: oh wait, nevermind. The type checker needs to know that it's going to be synthesized. So maybe typing.dataclass_transform needs to have the option to synthesize the attribute.

[AlexWaygood]

I think pydantic has a custom mypy plugin, so it's probably an issue with the plugin not synthesising the attribute (haven't dug in to confirm my suspicion).

[AlexWaygood]

Anyway, thanks (as ever) for the very early testing here @cdce8p! Given how new/experimental the protocol is, and how it doesn't work particularly well with pyright due to microsoft/pyright#4339, I think it definitely isn't a suitable candidate to be added to typing_extensions just yet. But adding it to _typeshed is something we can easily do now, so I'll try to get on that (or alternatively, I'll happily accept a PR if you get there first ;)

[AlexWaygood]

(To be clear, this PR shouldn't cause any regressions for pyright users, the pyright issue just means that this PR doesn't provide quite as much of a type-safety improvement for pyright users as it does for mypy users.)

[cdce8p]

Small update regarding pydantic. Enabling it's mypy plugin does resolve the issue

[mypy]
plugins = pydantic.mypy

https://docs.pydantic.dev/mypy_plugin/