From 6df0eb2927d7940ab6de25de7fc82311605e89f0 Mon Sep 17 00:00:00 2001 From: Zixuan James Li Date: Thu, 8 Feb 2024 19:05:12 -0500 Subject: [PATCH] Be consistent with "key", "field", "item", "value type" usage. This follows the style of PEP 705 and the majority of the typing spec to use "key" or "item" instead of "field" depending on the context. It also makes use of "value type" more to be adequately more pedantic. The special key added is now referred to as the "reserved key" to make it clearer that it is not intended for other purposes anymore. It also make use of type aliases like `VT` more for brevity. Signed-off-by: Zixuan James Li --- peps/pep-0728.rst | 194 +++++++++++++++++++++++----------------------- 1 file changed, 98 insertions(+), 96 deletions(-) diff --git a/peps/pep-0728.rst b/peps/pep-0728.rst index b507500fed87..295b012fc296 100644 --- a/peps/pep-0728.rst +++ b/peps/pep-0728.rst @@ -1,5 +1,5 @@ PEP: 728 -Title: TypedDict with Typed Extra Fields +Title: TypedDict with Typed Extra Items Author: Zixuan James Li Sponsor: Jelle Zijlstra Discussions-To: @@ -17,28 +17,28 @@ Post-History: Abstract ======== -This PEP proposes a way to type extra fields for :class:`~typing.TypedDict` using -a using a special ``__extra__: T`` field. -This addresses the need to define a subset of fields that might appear in a -``dict`` while permitting additional fields of a specified type, and the need to -create a closed TypedDict type with ``__extra__: Never``. +This PEP proposes a way to type extra items for :class:`~typing.TypedDict` using +a reserved ``__extra__`` key. This addresses the need to define a subset of +keys that might appear in a ``dict`` while permitting additional items of a +specified type, and the need to create a closed TypedDict type with ``__extra__: +Never``. Motivation ========== -A TypedDict type can annotate the type of each known field from dictionaries. -However, due to structural subtyping, the TypedDict can have extra fields that -are not visible through its type. There is currently not a way to limit or -restrict the type of fields that might be present in the TypedDict type's -structural subtype. +A TypedDict type can annotate the value type of each known item from +dictionaries. However, due to structural subtyping, the TypedDict can have extra +items that are not visible through its type. There is currently not a way to +limit or restrict the value type of items that might be present in the TypedDict +type's structural subtype. Defining a Closed TypedDict Type -------------------------------- In practice, this prevents users from defining a closed TypedDict type -when it is expected that the TypedDict type contains no additional fields. +when it is expected that the TypedDict type contains no additional items. -Due to the possible presence of extra fields, type checker cannot infer more +Due to the possible presence of extra items, type checker cannot infer more precise return types for ``.items()`` and ``.values()`` on a TypedDict. This can also be resolved by `defining a closed TypedDict type `__. @@ -63,7 +63,7 @@ Nothing prevents a ``dict`` that is structurally compatible with ``Movie`` to have the ``author`` key, and it would be correct for the type checker to not narrow its type. -Allowing Extra Fields of a Certain Type +Allowing Extra Items of a Certain Type --------------------------------------- For supporting API interfaces or legacy codebase where only a subset of possible @@ -78,14 +78,14 @@ from doing this:: name: str def fun(movie: MovieBase) -> None: - # movie can have extra fields that are not visible through MovieBase + # movie can have extra items that are not visible through MovieBase ... movie: MovieBase = {"name": "Blade Runner", "year": 1982} # Not OK fun({"name": "Blade Runner", "year": 1982}) # Not OK While the restriction is enforced when constructing a TypedDict, due to -structural subtyping, the TypedDict is may have extra fields that are not +structural subtyping, the TypedDict is may have extra items that are not visible through its type. For example:: class Movie(MovieBase): @@ -94,7 +94,7 @@ visible through its type. For example:: movie: Movie = {"name": "Blade Runner", "year": 1982} fun(movie) # OK -It is not possible to acknowledge the existence of the extra fields through +It is not possible to acknowledge the existence of the extra items through ``in`` check and access them without breaking type safety, even though they might exist from arbitrary structural subtypes of ``MovieBase``:: @@ -115,20 +115,20 @@ Support Additional Keys for ``Unpack`` :pep:`692` adds a way to precisely annotate the types of individual keyword arguments represented by ``**kwargs`` using TypedDict with ``Unpack``. However, -because TypedDict cannot be defined to accept arbitrary extra fields, it is not +because TypedDict cannot be defined to accept arbitrary extra items, it is not possible to `allow additional keyword arguments `__ that are not known at the time the TypedDict is defined. Given the usage of pre-:pep:`692` type annotation for ``**kwargs`` in existing -codebases, it will be valuable to accept and type extra fields on TypedDict so +codebases, it will be valuable to accept and type extra items on TypedDict so that the old typing behavior can be supported in combination with the new ``Unpack`` construct. Rationale ========= -A type that allows extra fields of type ``str`` on a TypedDict can be loosely +A type that allows extra items of type ``str`` on a TypedDict can be loosely described as the intersection between the TypedDict and ``Mapping[str, str]``. `Index Signatures `__ @@ -146,10 +146,10 @@ intersection of types or syntax changes, offering a natural extension to the existing type consistency rules. We propose that we give the dunder attribute ``__extra__`` a special meaning: -When it is defined on a TypedDict type, extra fields are allowed, and their -types should be compatible with the type of ``__extra__``. Different from index -signatures, the types of known fields do not need to be consistent with the type -of ``__extra__``. +When it is defined on a TypedDict type, extra items are allowed, and their types +should be compatible with the value type of ``__extra__``. Different from index +signatures, the types of known items do not need to be consistent with the value +type of ``__extra__``. There are some advantages to this approach: @@ -160,11 +160,10 @@ There are some advantages to this approach: `__. ``__extra__`` can be treated as a pseudo-key in terms of type consistency. -- There is no need to introduce a syntax to specify the type of the extra - fields. +- There is no need to introduce a syntax to specify the type of the extra items. -- We can precisely type the extra fields without making ``__extra__`` the union - of known fields. +- We can precisely type the extra items without making ``__extra__`` the union + of known items. Specification ============= @@ -172,7 +171,7 @@ Specification This specification is structured to parallel :pep:`589` to highlight changes to the original TypedDict specification. -Extra fields are treated as non-required fields having the same type of +Extra items are treated as non-required items having the same type of ``__extra__`` whose keys are allowed when determining `supported and unsupported operations `__. @@ -180,9 +179,9 @@ Extra fields are treated as non-required fields having the same type of Using TypedDict Types --------------------- -For a TypedDict type that has the ``__extra__`` field, during construction, the -value type of each unknown field is expected to be non-required and compatible -with the type of ``__extra__``. For example:: +For a TypedDict type that has the "__extra__" key, during construction, the +value type of each unknown item is expected to be non-required and compatible +with the value type of ``__extra__``. For example:: class Movie(TypedDict): name: str @@ -196,14 +195,14 @@ with the type of ``__extra__``. For example:: In this example, ``__extra__: bool`` does not mean that ``Movie`` has a required string key "__extra__" whose value type is ``bool``. Instead, it specifies that -fields other than "name" have a value type of ``bool`` and are non-required. +keys other than "name" have a value type of ``bool`` and are non-required. The alternative inline syntax is also supported:: Movie = TypedDict("Movie", {"name": str, "__extra__": bool}) -Accessing extra fields is allowed. Type checkers must infer its value type from -the type of ``__extra__``:: +Accessing extra keys is allowed. Type checkers must infer its value type from +the value type of ``__extra__``:: def f(movie: Movie) -> None: reveal_type(movie["name"]) # Revealed type is 'str' @@ -212,23 +211,26 @@ the type of ``__extra__``:: Interaction with PEP 705 ------------------------ -When ``__extra__`` is annotated with ``ReadOnly[]``, the extra fields on the -TypedDict have the properties of read-only fields. This affects subclassing +When ``__extra__`` is annotated with ``ReadOnly[]``, the extra items on the +TypedDict have the properties of read-only items. This affects subclassing according to the inheritance rules specified in :pep:`PEP 705 <705#Inheritance>`. -Notably, a subclass of the TypedDict type may redeclare the type of ``__extra__`` -or extra fields if the TypedDict type declares ``__extra__`` to be read-only. +Notably, a subclass of the TypedDict type may redeclare the value type of +``__extra__`` or extra items if the TypedDict type declares ``__extra__`` to be +read-only. + +More details are discussed in the later sections. Interaction with Totality ------------------------- It is an error to use ``Required[]`` or ``NotRequired[]`` with the special -``__extra__`` field. ``total=False`` and ``total=True`` have no effect on +``__extra__`` item. ``total=False`` and ``total=True`` have no effect on ``__extra__`` itself. -The extra fields are non-required, regardless of the totality of the TypedDict. -Operations that are available to ``NotRequired`` fields should also be available -to the extra fields:: +The extra items are non-required, regardless of the totality of the TypedDict. +Operations that are available to ``NotRequired`` items should also be available +to the extra items:: class Movie(TypedDict): name: str @@ -241,7 +243,7 @@ to the extra fields:: Interaction with ``Unpack`` --------------------------- -For type checking purposes, ``Unpack[TypedDict]`` with extra fields should be +For type checking purposes, ``Unpack[TypedDict]`` with extra items should be treated as its equivalent in regular parameters, and the existing rules for function parameters still apply:: @@ -257,7 +259,7 @@ function parameters still apply:: Inheritance ----------- -``__extra__`` is inherited the same way as a regular ``key: value_type`` field. +``__extra__`` is inherited the same way as a regular ``key: value_type`` item. As with the other keys, the same rules from `the typing spec `__ and :pep:`PEP 705 <705#inheritance>` apply. We interpret the existing rules in the @@ -268,33 +270,33 @@ with it: * Changing a field type of a parent TypedDict class in a subclass is not allowed. -First, it is not allowed to change the type of ``__extra__`` in a subclass unless -it is declared to be ``ReadOnly`` in the superclass:: +First, it is not allowed to change the value type of ``__extra__`` in a subclass +unless it is declared to be ``ReadOnly`` in the superclass:: class Parent(TypedDict): __extra__: int | None class Child(Parent): - __extra__: int # Not OK. Like any other TypedDict field, __extra__'s type cannot be changed + __extra__: int # Not OK. Like any other TypedDict item, __extra__'s type cannot be changed -Second, ``__extra__`` effectively defines the type of any unnamed fields +Second, ``__extra__`` effectively defines the value type of any unnamed items accepted to the TypedDict and marks them as non-required. Thus, the above -restriction applies to any additional fields defined in a subclass. For each -field added in a subclass, all of the following conditions should apply: +restriction applies to any additional items defined in a subclass. For each item +added in a subclass, all of the following conditions should apply: - If ``__extra__`` is read-only - - The field can be either required or non-required + - The item can be either required or non-required - - The field's value type is consistent with the type of ``__extra__`` + - The item's value type is consistent with the value type of ``__extra__`` - If ``__extra__`` is not read-only - - The field is non-required + - The item is non-required - - The field's value type is consistent with the type of ``__extra__`` + - The item's value type is consistent with the value type of ``__extra__`` - - The type of ``__extra__`` is consistent with the field's value type + - The value type of ``__extra__`` is consistent with the item's value type - If ``__extra__`` is not redeclared, the subclass inherits it as-is. @@ -317,7 +319,7 @@ For example:: year: NotRequired[int | None] Due to this nature, an important side effect allows us to define a TypedDict -type that disallows additional fields:: +type that disallows additional items:: class MovieFinal(TypedDict): name: str @@ -329,27 +331,27 @@ there can be no other keys in ``MovieFinal`` other than the known ones. Type Consistency ---------------- -In addition to the set ``S`` of keys of the explicitly defined fields, a -TypedDict type that has ``__extra__`` is considered to have an infinite set of -fields that all satisfy the following conditions: +In addition to the set ``S`` of keys of the explicitly defined items, a +TypedDict type that has the item ``__extra__: T`` is considered to have an +infinite set of items that all satisfy the following conditions: - If ``__extra__`` is read-only - - The field's value type is consistent with the type of ``__extra__`` + - The key's value type is consistent with ``T`` - - The field's key is not in ``S``. + - The key is not in ``S``. - If ``__extra__`` is not read-only - - The field is non-required + - The key is non-required - - The field's value type is consistent with the type of ``__extra__`` + - The key's value type is consistent with ``T`` - - The type of ``__extra__`` is consistent with the field's value type + - ``T`` is consistent with the key's value type - - The field's key is not in ``S``. + - The key is not in ``S``. -For type checking purposes, let "__extra__" be a non-required pseudo-field to be +For type checking purposes, let "__extra__" be a non-required pseudo-key to be included whenever "for each ... item/key" is stated in :pep:`the existing type consistency rules from PEP 705 <705#type-consistency>`, and we modify it as follows: @@ -382,7 +384,7 @@ and we modify it as follows: The following examples illustrate these checks in action. -``__extra__`` puts various restrictions on additional fields for type +``__extra__`` puts various restrictions on additional items for type consistency checks:: class Movie(TypedDict): @@ -399,7 +401,7 @@ consistency checks:: The value type of "year" is not found in ``Movie``, and that it is not "read-only, not required, and of top value type (``ReadOnly[NotRequired[object]]``)" -thus it needs to be consistent with ``__extra__``'s value type and vice versa:: +thus it needs to be consistent with ``__extra__``'s type and vice versa:: class MovieDetails(TypedDict): name: str @@ -414,7 +416,7 @@ corresponding key. "year" being required violates the rule "For each required key in ``B``, the corresponding key is required in ``A``". When ``__extra__`` is defined to be read-only in a TypedDict type, it is possible -for a field to have a narrower type than ``__extra__``'s value type. +for an item to have a narrower type than ``__extra__``'s type. class Movie(TypedDict): name: str @@ -428,9 +430,9 @@ for a field to have a narrower type than ``__extra__``'s value type. movie: Movie = details # OK. 'int' is consistent with 'str | int'. This behaves the same way as :pep:`705` specified if ``year: ReadOnly[str | int]`` -is a field defined in ``Movie``. +is an item defined in ``Movie``. -``__extra__`` as a pseudo-field follows the same rules that other fields have, +``__extra__`` as a pseudo-key follows the same rules that other keys have, so when both TypedDict contains ``__extra__``, this check is naturally enforced:: @@ -444,14 +446,14 @@ enforced:: extra_int: MovieExtraInt = {"name": "No Country for Old Men", "year": 2007} extra_str: MovieExtraStr = {"name": "No Country for Old Men", "description": ""} - extra_int = extra_str # Not OK. 'str' is inconsistent with 'int' for field '__extra__' - extra_str = extra_int # Not OK. 'int' is inconsistent with 'str' for field '__extra__' + extra_int = extra_str # Not OK. 'str' is inconsistent with 'int' for item '__extra__' + extra_str = extra_int # Not OK. 'int' is inconsistent with 'str' for item '__extra__' Interaction with Mapping[KT, VT] -------------------------------- A TypedDict can be consistent with ``Mapping[...]`` types other than -``Mapping[str, object]`` as long as the union of value types on all fields is +``Mapping[str, object]`` as long as the union of value types on all keys is consistent with the value type of the ``Mapping[...]`` type. It is an extension to this rule from the typing spec:: @@ -473,8 +475,8 @@ For example:: int_mapping: Mapping[str, int] = extra_int # Not OK. 'int | str' is not consistent with 'int' int_str_mapping: Mapping[str, int | str] = extra_int # OK -Furthermore, type checkers should be able to infer the precise return types for -``values()`` and ``items()`` on a TypedDict:: +Furthermore, type checkers should be able to infer the precise return types of +``values()`` and ``items()`` on such TypedDict types:: def fun(movie: MovieExtraStr) -> None: reveal_type(movie.items()) # Revealed type is 'dict_items[str, str]' @@ -488,13 +490,13 @@ keys in a TypedDict type's structural subtypes, we can determine if the TypedDict type and its structural subtypes will ever have any required key during static analysis. -If there is no required key, the TypedDict type is consistent with ``dict[KT, VT]`` -and vice versa if all fields on the TypedDict type satisfy the following +If there is no required key, the TypedDict type is consistent with ``dict[KT, +VT]`` and vice versa if all items on the TypedDict type satisfy the following conditions: -- ``VT`` is consistent with the value type of the field +- ``VT`` is consistent with the value type of the item -- The value type of the field is consistent with ``VT`` +- The value type of the item is consistent with ``VT`` For example:: @@ -521,12 +523,12 @@ In this case, methods that are previously unavailable on a TypedDict are allowed Open Issues =========== -Alternatives to the ``__extra__`` Magic Field +Alternatives to the ``__extra__`` Reserved Key --------------------------------------------- As it was pointed out in the `PEP 705 review comment `__, -``__extra__`` as a field has some disadvantages, including not allowing +``__extra__`` as a reserved item has some disadvantages, including not allowing "__extra__" as a regular key, requiring special-handling to disallow ``Required`` and ``NotRequired``. There could be some better alternatives to this without the above mentioned issues. @@ -546,19 +548,19 @@ versions as long as the type checker supports it. Rejected Ideas ============== -Allowing Extra Fields without Specifying the Type +Allowing Extra Items without Specifying the Type ------------------------------------------------- ``extra=True`` was originally proposed for defining a TypedDict accept extra -fields regardless of the type, like how ``total=True`` works:: +items regardless of the type, like how ``total=True`` works:: class TypedDict(extra=True): pass -Because it did not offer a way to specify the type of the extra fields, the type -checkers will need to assume that the type of the extra fields are ``Any``, +Because it did not offer a way to specify the type of the extra items, the type +checkers will need to assume that the type of the extra items are ``Any``, which compromises type safety. Furthermore, the current behavior of TypedDict -already allows untyped extra fields to be present in runtime, due to structural +already allows untyped extra items to be present in runtime, due to structural subtyping. Supporting ``TypedDict(extra=type)`` @@ -566,35 +568,35 @@ Supporting ``TypedDict(extra=type)`` This adds more corner cases to determine whether a type should be treated as a type or a value. And it will require more work to support using special forms to -type the extra fields. +type the extra items. While this saves us from reserving an attribute for special use, it will require extra work to implement inheritance, and it is less natural to integrate with generic TypedDicts. -Support Extra Fields with Intersection +Support Extra Items with Intersection -------------------------------------- Supporting intersections in Python's type system requires a lot of careful considerations, and it can take a long time for the community to reach a consensus on a reasonable design. -Ideally, extra fields in TypedDict should not be blocked by work on +Ideally, extra items in TypedDict should not be blocked by work on intersections, nor does it necessarily need to be supported through intersections. Moreover, the intersection between ``Mapping[...]`` and ``TypedDict`` is not -equivalent to a TypedDict type with the proposed ``__extra__`` special field, as -the value type of all known fields in ``TypedDict`` need to satisfy the +equivalent to a TypedDict type with the proposed ``__extra__`` special item, as +the value type of all known items in ``TypedDict`` need to satisfy the is-subtype-of relation with the value type of ``Mapping[...]``. -Requiring Type Compatibility of the Known Fields with ``__extra__`` +Requiring Type Compatibility of the Known Items with ``__extra__`` ------------------------------------------------------------------- ``__extra__`` restricts the value type for keys that are *unknown* to the -TypedDict type. So the value type of any *known* field is not necessarily +TypedDict type. So the value type of any *known* item is not necessarily consistent with ``__extra__``'s type, and ``__extra__``'s type is not -necessarily consistent with the value types of all known fields. +necessarily consistent with the value types of all known items. This differs from TypeScript's `Index Signatures `__