From 3253c9a69ec461de0a8636c849582ffbc063a728 Mon Sep 17 00:00:00 2001 From: Zixuan James Li Date: Sun, 17 Sep 2023 03:22:32 -0400 Subject: [PATCH 01/19] PEP 728: TypedDict with Typed Extra Fields Signed-off-by: Zixuan James Li --- .github/CODEOWNERS | 1 + peps/pep-0728.rst | 614 +++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 615 insertions(+) create mode 100644 peps/pep-0728.rst diff --git a/.github/CODEOWNERS b/.github/CODEOWNERS index f32ca251fc3..0a57d91da11 100644 --- a/.github/CODEOWNERS +++ b/.github/CODEOWNERS @@ -606,6 +606,7 @@ peps/pep-0724.rst @jellezijlstra peps/pep-0725.rst @pradyunsg peps/pep-0726.rst @AA-Turner peps/pep-0727.rst @JelleZijlstra +peps/pep-0728.rst @AlexWaygood peps/pep-0729.rst @JelleZijlstra @hauntsaninja peps/pep-0730.rst @ned-deily peps/pep-0731.rst @gvanrossum @encukou @vstinner @zooba @iritkatriel diff --git a/peps/pep-0728.rst b/peps/pep-0728.rst new file mode 100644 index 00000000000..d24b5afa03b --- /dev/null +++ b/peps/pep-0728.rst @@ -0,0 +1,614 @@ +PEP: 728 +Title: TypedDict with Typed Extra Fields +Author: Zixuan James Li +Sponsor: Alex Waygood +Discussions-To: https://discuss.python.org/t/pep-692-using-typeddict-for-more-precise-kwargs-typing/17314 +Status: Draft +Type: Standards Track +Topic: Typing +Content-Type: text/x-rst +Created: 12-Sep-2023 +Python-Version: 3.13 +Post-History: + + +.. highlight:: rst + +Abstract +======== + +This PEP proposes a way to type extra fields for :class:`~typing.TypedDict`. +This addresses the need to define a subset of fields that might appear in a +``dict`` while permitting additional fields of a specified type. + +Motivation +========== + +:pep:`589` specified a tight restriction on type checking the construction of a TypedDict:: + + class MovieBase(TypedDict): + name: str + + movie: MovieBase = {"name": "Blade Runner", "year": 1982} # Not OK + +While the restriction is enforced when constructing a TypedDict, due to +structural subtyping, the TypedDict is flexible enough to have extra fields that +are not visible through its type. For example:: + + class Movie(MovieBase): + year: int + + def f(movie: MovieBase) -> None: + # movie can have extra fields that are not visible through MovieBase + ... + + movie: Movie = {"name": "Blade Runner", "year": 1982} + f(movie) # OK + +It is not possible to acknowledge the existence of the extra fields through +``in`` check and access them without breaking type safety, even though they +might exist from arbitrary structural subtypes of ``MovieBase``:: + + def g(movie: MovieBase) -> None: + if "year" in movie: + reveal_type(movie["year"]) # Error: TypedDict 'MovieBase' has no key 'breed' + +There have been requests for allowing extra keys for TypedDict [1]_ [2]_. A common +use case for this is typing complex dictionaries where only a subset of possible +keys are known. The following are some existing workarounds: + +- Suppress type checking error specifically for unknown keys on TypedDict. For + mypy, this can be done with ``--disable-error-code=typeddict-unknown-key`` + [3]_. This sacrifices type safety over flexibility, and it does not offer a + way to restrict the type of additional fields on a TypedDict. + +- Define the extra keys as a nested ``Dict`` or ``Mapping`` inside the + ``TypedDict``:: + + class MovieBase(TypedDict): + name: str + extra_fields: Mapping[str, int | str] + + While this construct is explicit about the type of the extra fields, it cannot + preserve the structure of an otherwise flatter dictionary containing all the + extra fields, adding an extra level of nesting. This is also undesirable when + the TypedDict ought to be used with an external API where the structure is + already defined. + +- Define a subclass of ``MutableMapping`` and override ``__getitem__`` for + specific string literals:: + + class Movie(MutableMapping[str, int | str]): + @overload + def __getitem__(self, key: Literal['name']) -> str: + ... + + This requires a lot of boilerplate code and is harder to read than a plain + TypedDict does not work to precisely type construction or mutation of a + specific field. + +:pep:`692` adds a way to precisely annotate the types of individual keyword +arguments represented by ``**kwargs`` using TypedDict with ``Unpack``. However, +because TypedDict cannot be defined to accept arbitrary extra fields, it is not +possible to allow additional keyword arguments that are not known at the time +the TypedDict is defined. + +Given the usage of pre-:pep:`692` type annotation for ``**kwargs`` in existing +codebases, it will be valuable to accept and type extra fields on TypedDict so +that the old typing behavior can be supported in combination with the new +``Unpack`` construct. [4]_ + +Rationale +========= + +A type that allows extra fields of type ``str`` on a TypedDict can be loosely +described as the intersection between the TypedDict and ``Mapping[str, str]``. + +`Index Signatures `__ +in TypeScript achieve this: + +.. code-block:: typescript + + type Foo = { + a: string + [key: string]: string + } + +This proposal aims to support a similar feature without introducing general +intersection of types or syntax changes. + +We propose that we give the dunder attribute ``__extra__`` a special meaning: +When it is defined on a TypedDict type, extra fields are allowed, and their +types should be compatible with the type of ``__extra__``. Different from index +signatures, the types of known fields don't need to be consistent with the type +of ``__extra__``. + +There are some advantages to this approach: + +- Inheritance works naturally. ``__extra__`` defined on a TypedDict will also + be available to its subclasses. + +- We can build on top of + :pep:`the type consistency rules defined in PEP 589 <589#type-consistency>`. + ``__extra__`` can be treated as a pseudo-key in terms of type consistency. + +- There is no need to introduce a syntax to specify the type of the extra + fields. + +- We can precisely type the extra fields without making ``__extra__`` the union + of known fields. + +Specification +============= + +This specification is structured to parallel :pep:`589` to highlight changes to +the original TypedDict specification. + +Extra fields are treated as non-required fields having the same type of +``__extra__`` whose keys are allowed when determining +:pep:`supported and unsupported operations <589>`. + +Using TypedDict Types +--------------------- + +For a TypedDict type that has the ``__extra__`` field, during construction, the +value type of each unknown field is expected to be non-required and compatible +with the type of ``__extra__``. For example:: + + class Movie(TypedDict): + name: str + __extra__: bool + + a: Movie = {"name": "Blade Runner", "novel_adaption": True} # OK + b: Movie = { + "name": "Blade Runner", + "year": 1982, # Not OK. 'int' is incompatible with 'bool' + } + +In this example, ``__extra__: bool`` does not mean that ``Movie`` has a required +string key "__extra__" whose value type is ``bool``. Instead, it specifies that +fields other than "name" have a value type of ``bool`` and are non-required. + +The alternative syntax is also supported:: + + Movie = TypedDict("Movie", {"name": str, "__extra__": bool}) + +Accessing extra fields is allowed. Type checkers must infer its value type from +the type of ``__extra__``:: + + def f(movie: Movie, key: str) -> None: + reveal_type(movie["name"]) # Revealed type is 'str' + reveal_type(movie["year"]) # Revealed type is 'int' + + +Interaction with Totality +--------------------------- + +It is an error to use ``Required[]`` or ``NotRequired[]`` with the special +``__extra__`` field. ``total=False`` and ``total=True`` have no effect on +``__extra__`` itself. + +The extra fields are non-required, regardless of the totality of the TypedDict. +Operations that are available to ``NotRequired`` fields should also be available +to the extra fields:: + + def f(movie: Movie) -> None: + del movie["name"] # Not OK + del movie["year"] # OK + +Interaction with ``Unpack`` +--------------------------- + +For type checking purposes, ``Unpack[TypedDict]`` with extra fields should be +treated as its equivalent in regular parameters, and the existing rules for +function parameters still apply:: + + class Movie(TypedDict): + name: str + __extra__: int + + def f(**kwargs: Unpack[Movie]) -> None: ... + + # Should be equivalent to + def f(*, name: str, **kwargs: int) -> None: ... + +Inheritance +----------- + +``__extra__`` is inherited the same way as a regular ``key: value_type`` +field. The same rules from :pep:`PEP 589 <589#inheritance>` apply. + +Additionally, we need to reinterpret the following rule to define how +``__extra__`` interacts with it: + + * Changing a field type of a parent TypedDict class in a subclass is not allowed. + +First, it is not allowed to change the type of ``__extra__`` in a subclass:: + + class Parent(TypedDict): + __extra__: int | None + + class Child(Parent): + __extra__: int # Not OK. Like any other TypedDict field, __extra__'s type cannot be changed + +Second, ``__extra__`` effectively defines the type of any unnamed fields +accepted to the TypedDict and marks them as non-required. Thus, the above +restriction applies to any additional fields defined in a subclass. For each +field added in a subclass, all of the following conditions should apply: + +- The field is non-required + +- The field's value type is consistent with the type of ``__extra__`` + +- The type of ``__extra__`` is consistent with the field's value type + +For example:: + + class MovieBase(TypedDict): + name: str + __extra__: int | None + + class AdaptedMovie(MovieBase): # Not OK. 'bool' is not consistent with 'int | None' + adapted_from_novel: bool + + class MovieRequiredYear(MovieBase): # Not OK. Required key 'x' is not known to 'Parent' + year: int | None + + class MovieNotRequiredYear(MovieBase): # Not OK. 'int | None' is not consistent with 'int' + year: NotRequired[int] + + class MovieWithYear(MovieBase): # OK + x: NotRequired[int | None] + +As a side-effect, annotating ``__extra__`` with :class:`typing.Never` for a +TypedDict type will act similarly to decorating the class definition with +``@final``, because existing fields' types cannot be changed and no meaningful +extra fields can be added. + +Type Consistency +---------------- + +In addition to the set ``S`` of keys of the explicitly defined fields, a +TypedDict type that has ``__extra__`` is considered to have an infinite set of +fields that all satisfy the following conditions: + +- The field is non-required + +- The field's value type is consistent with the type of ``__extra__`` + +- The type of ``__extra__`` is consistent with the field's value type + +- The field's key is not in ``S``. + +For type checking purposes, let "__extra__" be a pseudo-key to be included +whenever "for each key" or "for each non-required key" is stated in +:pep:`the existing type consistency rules <589#type-consistency>`: + + Second, a TypedDict type ``A`` is consistent with TypedDict ``B`` if + ``A`` is structurally compatible with ``B``. This is true if and only + if both of these conditions are satisfied: + + * For each key in ``B``, ``A`` has the corresponding key and the + corresponding value type in ``A`` is consistent with the value type in + ``B``. For each key in ``B``, the value type in ``B`` is also consistent + with the corresponding value type in ``A``. [Edit: if the corresponding key + with the same name cannot be found in ``A``, "__extra__" is considered the + corresponding key.] + + * For each required key in ``B``, the corresponding key is required in + ``A``. For each non-required key in ``B``, the corresponding key is not + required in ``A``. [Edit: if the corresponding non-required key with the + same name cannot be found in ``A``, "__extra__" is considered the + corresponding key.] + +The following examples illustrate these checks in action. + +``__extra__`` as a pseudo-field follows the same rules that other fields have, +so when both TypedDict contains ``__extra__``, this check is naturally +enforced:: + + class MovieExtraInt(TypedDict): + name: str + __extra__: int + + class MovieExtraStr(TypedDict): + name: str + __extra__: str + + extra_int: MovieExtraInt = {"name": "No Country for Old Men", "year": 2007} + extra_str: MovieExtraStr = {"name": "No Country for Old Men", "description": ""} + extra_int = extra_str # Not OK. 'str' is inconsistent with 'int' for field '__extra__' + extra_str = extra_int # Not OK. 'int' is inconsistent with 'str' for field '__extra__' + +When only one of the TypedDict ``t`` has ``__extra__``, then the keys unknown to +``t`` in the other TypedDict corresponds to the ``__extra__`` pseudo-field. +Still, the same set of rules from :pep:`589` apply:: + + class MovieNotRequiredYear(TypedDict): + name: str + year: NotRequired[int] + + class MovieRequiredYear(TypedDict): + name: str + year: int + + extra_int: MovieExtraInt = {"name": "Kill Bill Vol. 1"} + required_year: MovieNotRequiredYear = {"name": "Kill Bill Vol. 1"} + not_required_year: MovieRequiredYear = {"name": "Kill Bill Vol. 1", "year": 2003} + + extra_int = required_year # Not OK. Having an extra required field makes + # 'required_year' structurally incompatible with + # extra_int + extra_int = not_required_year # OK. + + required_year = extra_int # Not OK. The required key 'year' is missing in + # 'extra_int' + not_required_year = extra_int # OK. + +A TypedDict can be consistent with ``Mapping[...]`` types other than +``Mapping[str, object]`` as long as the union of value types on all fields is +consistent with the value type of the ``Mapping[...]`` type. It is an extension +to this rule:: + + * A TypedDict with all ``int`` values is not consistent with + ``Mapping[str, int]``, since there may be additional non-``int`` + values not visible through the type, due to structural subtyping. + These can be accessed using the ``values()`` and ``items()`` + methods in ``Mapping`` + +For example:: + + extra_str: MovieExtraStr = {"name": "Blade Runner", "summary": ""} + str_mapping: Mapping[str, str] = extra_str # OK + + int_mapping: Mapping[str, int] = extra_int # Not OK. 'int | str' is not consistent with 'int' + int_str_mapping: Mapping[str, int | str] = extra_int # OK + +Note that because the presence of ``__extra__`` prohibits required keys from +being added through inheritance, it is possible to determine if a TypedDict type +and its structural subtypes will never have any required key during static +analysis. + +If there is no required key, the TypedDict type is consistent with ``Dict[...]`` +and vice versa if all fields on the TypedDict type satisfy the following +conditions: + +- The value type of the ``Dict[...]`` is consistent with the value type of the + field + +- The value type of the field is consistent with the value type of the + ``Dict[...]`` + +For example:: + + class IntDict(TypedDict): + __extra__: int + + class IntDictWithNum(IntDict): + num: NotRequired[int] + + def f(x: IntDict) -> None: + v: Dict[str, int] = x # OK + v.clear() # OK + + not_required_num: IntDictWithNum = {"num": 1, "bar": 2} + regular_dict: Dict[str, int] = not_required_num # OK + f(not_required_num) # OK + +Open Issues +=========== + +Soundness of Structural Subtyping +--------------------------------- + +We refer to TypeScript's support for this feature in this PEP. However, a caveat +here is that the TypeScript construct is unsound due to a design decision noted +in the documentation [5]_. Consider this example: + +.. code-block:: typescript + + interface A { + value: string + [key: string]: string | number + } + + interface B extends A { + foo: number + } + + const x: B = {value: "asd", foo: 12} + function mut(v: A) { + v.foo = "asd" + } + mut(x) + console.log(x) // {"value": "asd", "foo": "asd"} + +Because ``A`` is a subtype of ``B``, ``mut`` accepts ``x`` and mutates the value +of the property ``foo``, changing it to a ``string``. However, ``x`` is defined +to be ``B``, which expects ``foo`` to be a ``number``. As a result, the value of +``x`` in runtime is no longer consistent with its type during static analysis. + +To support this in Python, we need to determine if we need this level of +flexibility to open up this hole of soundness. + +This PEP as written is more restrictive to match the type consistency rules +defined in :pep:`589`. A more relaxed design may, for example, no longer +prohibit subclasses of a TypedDict type that defines ``__extra__`` from adding +more required fields:: + + class IntDict(TypedDict): + __extra__: int + + class IntDictWithNum(IntDict): + num: int + + def f(x: IntDict) -> None: + del x["num"] # OK + + required_num: IntDictWithNum = {"num": 1} + f(required_num) # OK + required_num["num"] # KeyError in runtime + +Should ``__extra__`` Exclude the Known Keys? +-------------------------------------------- + +``__extra__`` specifies the value type for fields of keys that are *unknown* to +the TypedDict type. So the value type of any known field is not necessarily +consistent with ``__extra__``'s type, and ``__extra__``'s type is not +necessarily consistent with the value types of all known fields. + +This differs from TypeScript's Index Signatures [6]_ syntax, which requires all +properties' types to match the string index's type. For example: + +.. code-block:: typescript + + interface MovieWithExtraNumber { + name: string // Property 'name' of type 'string' is not assignable to 'string' index type 'number'. + [index: string]: number + } + + interface MovieWithExtraNumberOrString { + name: string // OK + [index: string]: number | string + } + +This is a known limitation is discussed in TypeScript's issue tracker [7]_, +where it is suggested that there should be a way to exclude the defined keys +from the index signature, so that it is possible to define a type like +``MovieWithExtraNumber``. + +A workaround is to use intersection: + +.. code-block:: typescript + + interface MovieExtras { + [index: string]: number + } + + interface MovieBase { + name: string + } + + type Movie = MovieExtras & MovieBase + +However, this does not work for assignment: + +.. code-block:: typescript + + const movie: Movie = { + name: "No Country for Old Men", // Type 'string' is not assignable to type 'number' + year: 2007, + } + +How Else Will This Interact with PEP 692? +----------------------------------------- + +Quoting from a comment on a Pull Request that implements part of :pep:`692` [8]_ +in mypy, we want it to be as simple as possible, reusing the existing type +checking logic: + + There is nothing cumbersome in reducing the PEP to just one paragraph that + would explain that `Unpack[SomeTD]` is a syntactic sugar for (and is + considered equivalent to) the expanded signature. This has a number of + benefits: + + * This will not add any new unsafety that is not already present for + existing uses of TypedDicts in `**` contexts. (And type checkers may + handle this unsafety in a uniform way, say in mypy we _may_ use existing + `--extra-checks` flag to prohibit some techincally unsafe calls as I + mentioned before.) + + * This is actually easy to remember and to reason about. + + * This will allow people who want subtyping between callables to easily + achieve this using `total=False`, which follows from existing rules for + expanded callables. + +Backwards Compatibility +======================= + +Dunder attributes like ``__extra__`` are reserved for stdlib, we need not +concerned about using it in this proposal. If the proposal is accepted, neither +``__required_keys__`` and ``__optional_keys__`` should include ``__extra__`` in +runtime. + +Because this is a type-checking feature, it can be made available to older +versions when TypedDict is imported from ``typing_extensions``, and type +checkers should handle ``__extra__`` in that case. + +Rejected Ideas +============== + +Allowing Extra Fields without Specifying the Type +------------------------------------------------- + +``extra=True`` was originally proposed for defining a TypedDict accept extra +fields regardless of the type, like how ``total=True`` works:: + + class TypedDict(extra=True): + pass + +Because it did not offer a way to specify the type of the extra fields, the type +checkers will need to assume that the type of the extra fields are ``Any``, +which compromises type safety. Furthermore, the current behavior of TypedDict +already allows untyped extra fields to be present in runtime, due to structural +subtyping. + +Supporting ``TypedDict(extra=type)`` +------------------------------------ + +This adds more corner cases to determine whether a type should be treated as a +type or a value. And it will require more work to support using special forms to +type the extra fields. + +While this saves us from reserving an attribute for special use, it will require +extra work to implement inheritance, and it is less natural to integrate with +generics if we support generic TypedDict types in the future. + +Support Extra Fields with Intersection +-------------------------------------- + +Supporting intersections in Python's type system requires a lot of careful +considerations, and it can take a long time for the community to reach a +consensus on a reasonable design. + +Ideally, extra fields in TypedDict should not be blocked by work on +intersections, nor does it necessarily need to be supported through +intersections. + +Moreover, the intersection between ``Mapping[...]`` and ``TypedDict`` is not +equivalent to a TypedDict type with the proposed ``__extra__`` special field, as +the value type of all known fields in ``TypedDict`` need to satisfy the +is-subtype-of relation with the value type of ``Mapping[...]``. + +Reference Implementation +======================== + +pyanalyze has +`experimental support `__ +for a similar feature. + +Reference implementation for this specific proposal, however, is not currently available. + +References +========== + +.. [1] https://github.com/python/mypy/issues/4617 +.. [2] https://mail.python.org/archives/list/typing-sig@python.org/thread/66RITIHDQHVTUMJHH2ORSNWZ6DOPM367/ +.. [3] https://github.com/python/mypy/pull/14225 +.. [4] https://discuss.python.org/t/pep-692-using-typeddict-for-more-precise-kwargs-typing/17314/87 +.. [5] https://www.typescriptlang.org/docs/handbook/type-compatibility.html#a-note-on-soundness +.. [6] https://www.typescriptlang.org/docs/handbook/2/objects.html#index-signatures +.. [7] https://github.com/microsoft/TypeScript/issues/17867 +.. [8] https://github.com/python/mypy/pull/15612#discussion_r1298300340 + +Acknowledgments +================ + +TBD. + +Copyright +========= + +This document is placed in the public domain or under the +CC0-1.0-Universal license, whichever is more permissive. From 887ecb3cbd5fc68fa7ab60b036820c017ff51aba Mon Sep 17 00:00:00 2001 From: Zixuan James Li Date: Thu, 21 Sep 2023 18:37:31 -0400 Subject: [PATCH 02/19] Address review comments. Signed-off-by: Zixuan James Li --- peps/pep-0728.rst | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/peps/pep-0728.rst b/peps/pep-0728.rst index d24b5afa03b..fd3f3545cdb 100644 --- a/peps/pep-0728.rst +++ b/peps/pep-0728.rst @@ -2,7 +2,7 @@ PEP: 728 Title: TypedDict with Typed Extra Fields Author: Zixuan James Li Sponsor: Alex Waygood -Discussions-To: https://discuss.python.org/t/pep-692-using-typeddict-for-more-precise-kwargs-typing/17314 +Discussions-To: Status: Draft Type: Standards Track Topic: Typing @@ -159,7 +159,7 @@ with the type of ``__extra__``. For example:: name: str __extra__: bool - a: Movie = {"name": "Blade Runner", "novel_adaption": True} # OK + a: Movie = {"name": "Blade Runner", "novel_adaptation": True} # OK b: Movie = { "name": "Blade Runner", "year": 1982, # Not OK. 'int' is incompatible with 'bool' @@ -533,8 +533,8 @@ concerned about using it in this proposal. If the proposal is accepted, neither runtime. Because this is a type-checking feature, it can be made available to older -versions when TypedDict is imported from ``typing_extensions``, and type -checkers should handle ``__extra__`` in that case. +versions as long as the type checker supports it without requiring imports from +``typing_extensions``. Rejected Ideas ============== @@ -563,7 +563,7 @@ type the extra fields. While this saves us from reserving an attribute for special use, it will require extra work to implement inheritance, and it is less natural to integrate with -generics if we support generic TypedDict types in the future. +generic TypedDicts. Support Extra Fields with Intersection -------------------------------------- From 9bc68c2682a900e8e08186f82208edfde78ba573 Mon Sep 17 00:00:00 2001 From: Zixuan James Li Date: Fri, 2 Feb 2024 17:35:07 -0500 Subject: [PATCH 03/19] Update sponsor information. Signed-off-by: Zixuan James Li --- .github/CODEOWNERS | 2 +- peps/pep-0728.rst | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/.github/CODEOWNERS b/.github/CODEOWNERS index 0a57d91da11..e3561fcbb79 100644 --- a/.github/CODEOWNERS +++ b/.github/CODEOWNERS @@ -606,7 +606,7 @@ peps/pep-0724.rst @jellezijlstra peps/pep-0725.rst @pradyunsg peps/pep-0726.rst @AA-Turner peps/pep-0727.rst @JelleZijlstra -peps/pep-0728.rst @AlexWaygood +peps/pep-0728.rst @JelleZijlstra peps/pep-0729.rst @JelleZijlstra @hauntsaninja peps/pep-0730.rst @ned-deily peps/pep-0731.rst @gvanrossum @encukou @vstinner @zooba @iritkatriel diff --git a/peps/pep-0728.rst b/peps/pep-0728.rst index fd3f3545cdb..ad198cf7a84 100644 --- a/peps/pep-0728.rst +++ b/peps/pep-0728.rst @@ -1,7 +1,7 @@ PEP: 728 Title: TypedDict with Typed Extra Fields Author: Zixuan James Li -Sponsor: Alex Waygood +Sponsor: Jelle Zijlstra Discussions-To: Status: Draft Type: Standards Track From 1855bf5ae0f9102f1842e45f9633ddf5711f85a9 Mon Sep 17 00:00:00 2001 From: Zixuan James Li Date: Fri, 2 Feb 2024 22:11:47 -0500 Subject: [PATCH 04/19] Extend on the "final" side-effect. Signed-off-by: Zixuan James Li --- peps/pep-0728.rst | 61 +++++++++++++++++++++++++++-------------------- 1 file changed, 35 insertions(+), 26 deletions(-) diff --git a/peps/pep-0728.rst b/peps/pep-0728.rst index ad198cf7a84..336c60f5394 100644 --- a/peps/pep-0728.rst +++ b/peps/pep-0728.rst @@ -53,13 +53,14 @@ might exist from arbitrary structural subtypes of ``MovieBase``:: if "year" in movie: reveal_type(movie["year"]) # Error: TypedDict 'MovieBase' has no key 'breed' -There have been requests for allowing extra keys for TypedDict [1]_ [2]_. A common -use case for this is typing complex dictionaries where only a subset of possible -keys are known. The following are some existing workarounds: +There have been requests to disallow extra keys explicitly by defining a "final" TypedDict +type [1]_ [2]_. There are also requests to allow extra keys for TypedDict [3]_ [4]_, +which is needed when typing a subset of possible keys in complex dictionaries. +The following are some existing workarounds: - Suppress type checking error specifically for unknown keys on TypedDict. For mypy, this can be done with ``--disable-error-code=typeddict-unknown-key`` - [3]_. This sacrifices type safety over flexibility, and it does not offer a + [5]_. This sacrifices type safety over flexibility, and it does not offer a way to restrict the type of additional fields on a TypedDict. - Define the extra keys as a nested ``Dict`` or ``Mapping`` inside the @@ -96,7 +97,7 @@ the TypedDict is defined. Given the usage of pre-:pep:`692` type annotation for ``**kwargs`` in existing codebases, it will be valuable to accept and type extra fields on TypedDict so that the old typing behavior can be supported in combination with the new -``Unpack`` construct. [4]_ +``Unpack`` construct. [6]_ Rationale ========= @@ -169,7 +170,7 @@ In this example, ``__extra__: bool`` does not mean that ``Movie`` has a required string key "__extra__" whose value type is ``bool``. Instead, it specifies that fields other than "name" have a value type of ``bool`` and are non-required. -The alternative syntax is also supported:: +The alternative inline syntax is also supported:: Movie = TypedDict("Movie", {"name": str, "__extra__": bool}) @@ -182,7 +183,7 @@ the type of ``__extra__``:: Interaction with Totality ---------------------------- +------------------------- It is an error to use ``Required[]`` or ``NotRequired[]`` with the special ``__extra__`` field. ``total=False`` and ``total=True`` have no effect on @@ -251,19 +252,25 @@ For example:: class AdaptedMovie(MovieBase): # Not OK. 'bool' is not consistent with 'int | None' adapted_from_novel: bool - class MovieRequiredYear(MovieBase): # Not OK. Required key 'x' is not known to 'Parent' + class MovieRequiredYear(MovieBase): # Not OK. Required key 'year' is not known to 'Parent' year: int | None class MovieNotRequiredYear(MovieBase): # Not OK. 'int | None' is not consistent with 'int' year: NotRequired[int] class MovieWithYear(MovieBase): # OK - x: NotRequired[int | None] + year: NotRequired[int | None] + +Due to this nature, an important side effect allows us to define a TypedDict +type that acts similarly to decorating the class definition with ``@final``:: -As a side-effect, annotating ``__extra__`` with :class:`typing.Never` for a -TypedDict type will act similarly to decorating the class definition with -``@final``, because existing fields' types cannot be changed and no meaningful -extra fields can be added. + class MovieFinal(TypedDict): + name: str + __extra__: Never + +Here, annotating ``__extra__`` with :class:`typing.Never` prevents ``MovieFinal`` +from being inheritted, because existing fields' types cannot be changed and +no meaningful extra fields can be added. Type Consistency ---------------- @@ -338,7 +345,7 @@ Still, the same set of rules from :pep:`589` apply:: extra_int = required_year # Not OK. Having an extra required field makes # 'required_year' structurally incompatible with - # extra_int + # 'extra_int' extra_int = not_required_year # OK. required_year = extra_int # Not OK. The required key 'year' is missing in @@ -403,7 +410,7 @@ Soundness of Structural Subtyping We refer to TypeScript's support for this feature in this PEP. However, a caveat here is that the TypeScript construct is unsound due to a design decision noted -in the documentation [5]_. Consider this example: +in the documentation [7]_. Consider this example: .. code-block:: typescript @@ -457,7 +464,7 @@ the TypedDict type. So the value type of any known field is not necessarily consistent with ``__extra__``'s type, and ``__extra__``'s type is not necessarily consistent with the value types of all known fields. -This differs from TypeScript's Index Signatures [6]_ syntax, which requires all +This differs from TypeScript's Index Signatures [8]_ syntax, which requires all properties' types to match the string index's type. For example: .. code-block:: typescript @@ -472,7 +479,7 @@ properties' types to match the string index's type. For example: [index: string]: number | string } -This is a known limitation is discussed in TypeScript's issue tracker [7]_, +This is a known limitation is discussed in TypeScript's issue tracker [9]_, where it is suggested that there should be a way to exclude the defined keys from the index signature, so that it is possible to define a type like ``MovieWithExtraNumber``. @@ -503,7 +510,7 @@ However, this does not work for assignment: How Else Will This Interact with PEP 692? ----------------------------------------- -Quoting from a comment on a Pull Request that implements part of :pep:`692` [8]_ +Quoting from a comment on a Pull Request that implements part of :pep:`692` [10]_ in mypy, we want it to be as simple as possible, reusing the existing type checking logic: @@ -593,14 +600,16 @@ Reference implementation for this specific proposal, however, is not currently a References ========== -.. [1] https://github.com/python/mypy/issues/4617 -.. [2] https://mail.python.org/archives/list/typing-sig@python.org/thread/66RITIHDQHVTUMJHH2ORSNWZ6DOPM367/ -.. [3] https://github.com/python/mypy/pull/14225 -.. [4] https://discuss.python.org/t/pep-692-using-typeddict-for-more-precise-kwargs-typing/17314/87 -.. [5] https://www.typescriptlang.org/docs/handbook/type-compatibility.html#a-note-on-soundness -.. [6] https://www.typescriptlang.org/docs/handbook/2/objects.html#index-signatures -.. [7] https://github.com/microsoft/TypeScript/issues/17867 -.. [8] https://github.com/python/mypy/pull/15612#discussion_r1298300340 +.. [1] https://discuss.python.org/t/do-we-want-an-exact-typeddict-if-so-how-final-extras-never/44418 +.. [2] https://github.com/python/mypy/issues/7981 +.. [3] https://github.com/python/mypy/issues/4617 +.. [4] https://mail.python.org/archives/list/typing-sig@python.org/thread/66RITIHDQHVTUMJHH2ORSNWZ6DOPM367/ +.. [5] https://github.com/python/mypy/pull/14225 +.. [6] https://discuss.python.org/t/pep-692-using-typeddict-for-more-precise-kwargs-typing/17314/87 +.. [7] https://www.typescriptlang.org/docs/handbook/type-compatibility.html#a-note-on-soundness +.. [8] https://www.typescriptlang.org/docs/handbook/2/objects.html#index-signatures +.. [9] https://github.com/microsoft/TypeScript/issues/17867 +.. [10] https://github.com/python/mypy/pull/15612#discussion_r1298300340 Acknowledgments ================ From 72e2a6252fb300533d32d931ac23352314599cf7 Mon Sep 17 00:00:00 2001 From: Zixuan James Li Date: Sat, 3 Feb 2024 17:42:00 -0500 Subject: [PATCH 05/19] Mention the limitation of the current "__extra__" approach. I left out the rewrite to other proposed syntax such as using "OtherFields[]" with an arbitrary key before there is sufficient discussion on the alternatives available to us. Signed-off-by: Zixuan James Li --- peps/pep-0728.rst | 18 ++++++++++++++---- 1 file changed, 14 insertions(+), 4 deletions(-) diff --git a/peps/pep-0728.rst b/peps/pep-0728.rst index 336c60f5394..e600cd9c449 100644 --- a/peps/pep-0728.rst +++ b/peps/pep-0728.rst @@ -531,13 +531,22 @@ checking logic: achieve this using `total=False`, which follows from existing rules for expanded callables. +Alternatives to the ``__extra__`` Magic Field +--------------------------------------------- + +As it was pointed out in the PEP 705 review comment [11]_, ``__extra__`` +as a field has some disadvantages, including not allowing "__extra__" +as a regular key, requiring special-handling to disallow ``Required`` +and ``NotRequired``. There could be some better alternatives to this +without the above mentioned issues. + Backwards Compatibility ======================= -Dunder attributes like ``__extra__`` are reserved for stdlib, we need not -concerned about using it in this proposal. If the proposal is accepted, neither -``__required_keys__`` and ``__optional_keys__`` should include ``__extra__`` in -runtime. +While dunder attributes like ``__extra__`` are reserved for stdlib, it is still +a limitation that ``__extra__`` is no longer usable as a regular key. If the +proposal is accepted, neither ``__required_keys__`` and ``__optional_keys__`` +should include ``__extra__`` in runtime. Because this is a type-checking feature, it can be made available to older versions as long as the type checker supports it without requiring imports from @@ -610,6 +619,7 @@ References .. [8] https://www.typescriptlang.org/docs/handbook/2/objects.html#index-signatures .. [9] https://github.com/microsoft/TypeScript/issues/17867 .. [10] https://github.com/python/mypy/pull/15612#discussion_r1298300340 +.. [11] https://discuss.python.org/t/pep-705-typeddict-read-only-and-other-keys/36457/6 Acknowledgments ================ From 5012fc62e588f645ba70587eef8cade5663e2f96 Mon Sep 17 00:00:00 2001 From: Zixuan James Li Date: Mon, 5 Feb 2024 00:30:40 -0500 Subject: [PATCH 06/19] Address some feedback (WIP). TODOs: - rewrite the motivation section - use inline references instead - extend on how this PEP interacts with PEP 705 --- peps/pep-0728.rst | 101 ++++++++++++++++++++-------------------------- 1 file changed, 44 insertions(+), 57 deletions(-) diff --git a/peps/pep-0728.rst b/peps/pep-0728.rst index e600cd9c449..9d294c3e4a5 100644 --- a/peps/pep-0728.rst +++ b/peps/pep-0728.rst @@ -17,13 +17,21 @@ Post-History: Abstract ======== -This PEP proposes a way to type extra fields for :class:`~typing.TypedDict`. +This PEP proposes a way to type extra fields for :class:`~typing.TypedDict` using +a using a special ``__extra__: T`` field. This addresses the need to define a subset of fields that might appear in a -``dict`` while permitting additional fields of a specified type. +``dict`` while permitting additional fields of a specified type, and the need to +create a closed TypedDict type with ``__extra__: Never``. Motivation ========== +There have been requests to disallow extra keys explicitly by defining a "final" TypedDict +type [1]_ [2]_. + +There are also requests to allow extra keys for TypedDict [3]_ [4]_, +which is needed when typing a subset of possible keys in complex dictionaries. + :pep:`589` specified a tight restriction on type checking the construction of a TypedDict:: class MovieBase(TypedDict): @@ -51,11 +59,8 @@ might exist from arbitrary structural subtypes of ``MovieBase``:: def g(movie: MovieBase) -> None: if "year" in movie: - reveal_type(movie["year"]) # Error: TypedDict 'MovieBase' has no key 'breed' + reveal_type(movie["year"]) # Error: TypedDict 'MovieBase' has no key 'year' -There have been requests to disallow extra keys explicitly by defining a "final" TypedDict -type [1]_ [2]_. There are also requests to allow extra keys for TypedDict [3]_ [4]_, -which is needed when typing a subset of possible keys in complex dictionaries. The following are some existing workarounds: - Suppress type checking error specifically for unknown keys on TypedDict. For @@ -63,7 +68,7 @@ The following are some existing workarounds: [5]_. This sacrifices type safety over flexibility, and it does not offer a way to restrict the type of additional fields on a TypedDict. -- Define the extra keys as a nested ``Dict`` or ``Mapping`` inside the +- Define the extra keys as a nested ``dict`` or ``Mapping`` inside the ``TypedDict``:: class MovieBase(TypedDict): @@ -84,8 +89,8 @@ The following are some existing workarounds: def __getitem__(self, key: Literal['name']) -> str: ... - This requires a lot of boilerplate code and is harder to read than a plain - TypedDict does not work to precisely type construction or mutation of a + This requires a lot of boilerplate code, is harder to read than a plain + TypedDict, and does not work to precisely type construction or mutation of a specific field. :pep:`692` adds a way to precisely annotate the types of individual keyword @@ -177,10 +182,19 @@ The alternative inline syntax is also supported:: Accessing extra fields is allowed. Type checkers must infer its value type from the type of ``__extra__``:: - def f(movie: Movie, key: str) -> None: - reveal_type(movie["name"]) # Revealed type is 'str' - reveal_type(movie["year"]) # Revealed type is 'int' + def f(movie: Movie) -> None: + reveal_type(movie["name"]) # Revealed type is 'str' + reveal_type(movie["novel_adaptation"]) # Revealed type is 'bool' + +Interaction with PEP 705 +------------------------ +When ``__extra__`` is annotated with ``ReadOnly[]``, the extra fields on the +TypedDict have the properties of read-only fields. This affects subclassing +according to the inheritance rules specified in :pep:`PEP 705 <589#Inheritance>`. + +Notably, a subclass of the TypedDict type may redeclare the type of ``__extra__`` +or extra fields if the TypedDict type declares ``__extra__`` to be read-only. Interaction with Totality ------------------------- @@ -224,7 +238,8 @@ Additionally, we need to reinterpret the following rule to define how * Changing a field type of a parent TypedDict class in a subclass is not allowed. -First, it is not allowed to change the type of ``__extra__`` in a subclass:: +First, it is not allowed to change the type of ``__extra__`` in a subclass unless +it is declared to be ``ReadOnly`` in the superclass:: class Parent(TypedDict): __extra__: int | None @@ -262,15 +277,14 @@ For example:: year: NotRequired[int | None] Due to this nature, an important side effect allows us to define a TypedDict -type that acts similarly to decorating the class definition with ``@final``:: +type that disallows additional fields:: class MovieFinal(TypedDict): name: str __extra__: Never -Here, annotating ``__extra__`` with :class:`typing.Never` prevents ``MovieFinal`` -from being inheritted, because existing fields' types cannot be changed and -no meaningful extra fields can be added. +Here, annotating ``__extra__`` with :class:`typing.Never` specifies that +there can be no other keys in ``MovieFinal`` other than the known ones. Type Consistency ---------------- @@ -328,7 +342,7 @@ enforced:: extra_str = extra_int # Not OK. 'int' is inconsistent with 'str' for field '__extra__' When only one of the TypedDict ``t`` has ``__extra__``, then the keys unknown to -``t`` in the other TypedDict corresponds to the ``__extra__`` pseudo-field. +``t`` in the other TypedDict correspond to the ``__extra__`` pseudo-field. Still, the same set of rules from :pep:`589` apply:: class MovieNotRequiredYear(TypedDict): @@ -371,20 +385,18 @@ For example:: int_mapping: Mapping[str, int] = extra_int # Not OK. 'int | str' is not consistent with 'int' int_str_mapping: Mapping[str, int | str] = extra_int # OK -Note that because the presence of ``__extra__`` prohibits required keys from -being added through inheritance, it is possible to determine if a TypedDict type -and its structural subtypes will never have any required key during static +Note that because the presence of ``__extra__`` prohibits additional required keys in a +TypedDict type's structural subtypes, we can determine if the TypedDict type +and its structural subtypes will ever have any required key during static analysis. -If there is no required key, the TypedDict type is consistent with ``Dict[...]`` +If there is no required key, the TypedDict type is consistent with ``dict[KT, VT]`` and vice versa if all fields on the TypedDict type satisfy the following conditions: -- The value type of the ``Dict[...]`` is consistent with the value type of the - field +- ``VT`` is consistent with the value type of the field -- The value type of the field is consistent with the value type of the - ``Dict[...]`` +- The value type of the field is consistent with ``VT`` For example:: @@ -395,11 +407,11 @@ For example:: num: NotRequired[int] def f(x: IntDict) -> None: - v: Dict[str, int] = x # OK + v: dict[str, int] = x # OK v.clear() # OK not_required_num: IntDictWithNum = {"num": 1, "bar": 2} - regular_dict: Dict[str, int] = not_required_num # OK + regular_dict: dict[str, int] = not_required_num # OK f(not_required_num) # OK Open Issues @@ -507,30 +519,6 @@ However, this does not work for assignment: year: 2007, } -How Else Will This Interact with PEP 692? ------------------------------------------ - -Quoting from a comment on a Pull Request that implements part of :pep:`692` [10]_ -in mypy, we want it to be as simple as possible, reusing the existing type -checking logic: - - There is nothing cumbersome in reducing the PEP to just one paragraph that - would explain that `Unpack[SomeTD]` is a syntactic sugar for (and is - considered equivalent to) the expanded signature. This has a number of - benefits: - - * This will not add any new unsafety that is not already present for - existing uses of TypedDicts in `**` contexts. (And type checkers may - handle this unsafety in a uniform way, say in mypy we _may_ use existing - `--extra-checks` flag to prohibit some techincally unsafe calls as I - mentioned before.) - - * This is actually easy to remember and to reason about. - - * This will allow people who want subtyping between callables to easily - achieve this using `total=False`, which follows from existing rules for - expanded callables. - Alternatives to the ``__extra__`` Magic Field --------------------------------------------- @@ -545,12 +533,12 @@ Backwards Compatibility While dunder attributes like ``__extra__`` are reserved for stdlib, it is still a limitation that ``__extra__`` is no longer usable as a regular key. If the -proposal is accepted, neither ``__required_keys__`` and ``__optional_keys__`` -should include ``__extra__`` in runtime. +proposal is accepted, none of ``__required_keys__``, ``__optional_keys__``, +``__readonly_keys__`` and ``__mutable_keys__`` should include ``__extra__`` in +runtime. Because this is a type-checking feature, it can be made available to older -versions as long as the type checker supports it without requiring imports from -``typing_extensions``. +versions as long as the type checker supports it. Rejected Ideas ============== @@ -618,7 +606,6 @@ References .. [7] https://www.typescriptlang.org/docs/handbook/type-compatibility.html#a-note-on-soundness .. [8] https://www.typescriptlang.org/docs/handbook/2/objects.html#index-signatures .. [9] https://github.com/microsoft/TypeScript/issues/17867 -.. [10] https://github.com/python/mypy/pull/15612#discussion_r1298300340 .. [11] https://discuss.python.org/t/pep-705-typeddict-read-only-and-other-keys/36457/6 Acknowledgments From 2bb2da54615c6fc206927b0a03c909743a11ec8d Mon Sep 17 00:00:00 2001 From: Zixuan James Li Date: Wed, 7 Feb 2024 21:34:05 -0500 Subject: [PATCH 07/19] Address feedbacks that require some more substantial rewrites. In this update: - The Motivation section has been rewritten to expand more on the compelling use cases and focus less on the current workarounds. - Update the "Open Issues" section to remove the issues that have been resolved. - Add an entry to "Rejected Ideas" after removing it from "Open Issues", and keep the discussion relevant to Python as opposed to TypeScript. - Remove the "References" and use inline hyperlinks instead throughout the document. Signed-off-by: Zixuan James Li --- peps/pep-0728.rst | 277 ++++++++++++++++++---------------------------- 1 file changed, 108 insertions(+), 169 deletions(-) diff --git a/peps/pep-0728.rst b/peps/pep-0728.rst index 9d294c3e4a5..8e3f8fe2e12 100644 --- a/peps/pep-0728.rst +++ b/peps/pep-0728.rst @@ -26,18 +26,63 @@ create a closed TypedDict type with ``__extra__: Never``. Motivation ========== -There have been requests to disallow extra keys explicitly by defining a "final" TypedDict -type [1]_ [2]_. +A TypedDict type can annotate the type of each known field from dictionaries. +However, due to structural subtyping, the TypedDict can have extra fields that +are not visible through its type. There is currently not a way to limit or +restrict the type of fields that might be present in the TypedDict type's +structural subtype. -There are also requests to allow extra keys for TypedDict [3]_ [4]_, -which is needed when typing a subset of possible keys in complex dictionaries. +Defining a Closed TypedDict Type +-------------------------------- -:pep:`589` specified a tight restriction on type checking the construction of a TypedDict:: +In practice, this prevents users from defining a closed TypedDict type +when it is expected that the TypedDict type contains no additional fields. + +Due to the possible presence of extra fields, type checker cannot infer more +precise return types for ``.items()`` and ``.values()`` on a TypedDict. This can +also be resolved by +`defining a closed TypedDict type `__. + +Another possible use case for this is a sound way to +`enable type narrowing `__ with the +``in`` check:: + + class Movie(TypedDict): + name: str + director: str + + class Book(TypedDict): + name: str + author: str + + def fun(entry: Movie | Book) -> None: + if "author" in entry: + add_book(entry) # Not OK + +Nothing prevents a ``dict`` that is structurally compatible with ``Movie`` to +have the ``author`` key, and it would be correct for the type checker to reject +the ``add_book`` call. + +Allowing Extra Fields of a Certain Type +--------------------------------------- + +For supporting API interfaces or legacy codebase where only a subset of possible +keys are known, it would be useful to explicitly expect additional keys of +certain value types. + +However, :pep:`589` is more restrictive on type checking the construction of a +TypedDict, `preventing the users `__ +from doing this:: class MovieBase(TypedDict): name: str + def fun(movie: MovieBase) -> None: + # movie can have extra fields that are not visible through MovieBase + ... + movie: MovieBase = {"name": "Blade Runner", "year": 1982} # Not OK + fun({"name": "Blade Runner", "year": 1982}) # Not OK While the restriction is enforced when constructing a TypedDict, due to structural subtyping, the TypedDict is flexible enough to have extra fields that @@ -46,12 +91,8 @@ are not visible through its type. For example:: class Movie(MovieBase): year: int - def f(movie: MovieBase) -> None: - # movie can have extra fields that are not visible through MovieBase - ... - movie: Movie = {"name": "Blade Runner", "year": 1982} - f(movie) # OK + fun(movie) # OK It is not possible to acknowledge the existence of the extra fields through ``in`` check and access them without breaking type safety, even though they @@ -61,48 +102,28 @@ might exist from arbitrary structural subtypes of ``MovieBase``:: if "year" in movie: reveal_type(movie["year"]) # Error: TypedDict 'MovieBase' has no key 'year' -The following are some existing workarounds: - -- Suppress type checking error specifically for unknown keys on TypedDict. For - mypy, this can be done with ``--disable-error-code=typeddict-unknown-key`` - [5]_. This sacrifices type safety over flexibility, and it does not offer a - way to restrict the type of additional fields on a TypedDict. - -- Define the extra keys as a nested ``dict`` or ``Mapping`` inside the - ``TypedDict``:: +Some workarounds have already been implemented in response to the need of +allowing extra keys, but none of them is ideal. For mypy, +``--disable-error-code=typeddict-unknown-key`` +`suppresses type checking error `__ +specifically for unknown keys on TypedDict. This sacrifices type safety over +flexibility, and it does not offer a way to specify that the TypedDict type +expects additional keys compatible with a certain type. - class MovieBase(TypedDict): - name: str - extra_fields: Mapping[str, int | str] - - While this construct is explicit about the type of the extra fields, it cannot - preserve the structure of an otherwise flatter dictionary containing all the - extra fields, adding an extra level of nesting. This is also undesirable when - the TypedDict ought to be used with an external API where the structure is - already defined. - -- Define a subclass of ``MutableMapping`` and override ``__getitem__`` for - specific string literals:: - - class Movie(MutableMapping[str, int | str]): - @overload - def __getitem__(self, key: Literal['name']) -> str: - ... - - This requires a lot of boilerplate code, is harder to read than a plain - TypedDict, and does not work to precisely type construction or mutation of a - specific field. +Support Additional Keys for ``Unpack`` +-------------------------------------- :pep:`692` adds a way to precisely annotate the types of individual keyword arguments represented by ``**kwargs`` using TypedDict with ``Unpack``. However, because TypedDict cannot be defined to accept arbitrary extra fields, it is not -possible to allow additional keyword arguments that are not known at the time -the TypedDict is defined. +possible to +`allow additional keyword arguments `__ +that are not known at the time the TypedDict is defined. Given the usage of pre-:pep:`692` type annotation for ``**kwargs`` in existing codebases, it will be valuable to accept and type extra fields on TypedDict so that the old typing behavior can be supported in combination with the new -``Unpack`` construct. [6]_ +``Unpack`` construct. Rationale ========= @@ -121,12 +142,13 @@ in TypeScript achieve this: } This proposal aims to support a similar feature without introducing general -intersection of types or syntax changes. +intersection of types or syntax changes, offering a natural extension to the +existing type consistency rules. We propose that we give the dunder attribute ``__extra__`` a special meaning: When it is defined on a TypedDict type, extra fields are allowed, and their types should be compatible with the type of ``__extra__``. Different from index -signatures, the types of known fields don't need to be consistent with the type +signatures, the types of known fields do not need to be consistent with the type of ``__extra__``. There are some advantages to this approach: @@ -417,116 +439,15 @@ For example:: Open Issues =========== -Soundness of Structural Subtyping ---------------------------------- - -We refer to TypeScript's support for this feature in this PEP. However, a caveat -here is that the TypeScript construct is unsound due to a design decision noted -in the documentation [7]_. Consider this example: - -.. code-block:: typescript - - interface A { - value: string - [key: string]: string | number - } - - interface B extends A { - foo: number - } - - const x: B = {value: "asd", foo: 12} - function mut(v: A) { - v.foo = "asd" - } - mut(x) - console.log(x) // {"value": "asd", "foo": "asd"} - -Because ``A`` is a subtype of ``B``, ``mut`` accepts ``x`` and mutates the value -of the property ``foo``, changing it to a ``string``. However, ``x`` is defined -to be ``B``, which expects ``foo`` to be a ``number``. As a result, the value of -``x`` in runtime is no longer consistent with its type during static analysis. - -To support this in Python, we need to determine if we need this level of -flexibility to open up this hole of soundness. - -This PEP as written is more restrictive to match the type consistency rules -defined in :pep:`589`. A more relaxed design may, for example, no longer -prohibit subclasses of a TypedDict type that defines ``__extra__`` from adding -more required fields:: - - class IntDict(TypedDict): - __extra__: int - - class IntDictWithNum(IntDict): - num: int - - def f(x: IntDict) -> None: - del x["num"] # OK - - required_num: IntDictWithNum = {"num": 1} - f(required_num) # OK - required_num["num"] # KeyError in runtime - -Should ``__extra__`` Exclude the Known Keys? --------------------------------------------- - -``__extra__`` specifies the value type for fields of keys that are *unknown* to -the TypedDict type. So the value type of any known field is not necessarily -consistent with ``__extra__``'s type, and ``__extra__``'s type is not -necessarily consistent with the value types of all known fields. - -This differs from TypeScript's Index Signatures [8]_ syntax, which requires all -properties' types to match the string index's type. For example: - -.. code-block:: typescript - - interface MovieWithExtraNumber { - name: string // Property 'name' of type 'string' is not assignable to 'string' index type 'number'. - [index: string]: number - } - - interface MovieWithExtraNumberOrString { - name: string // OK - [index: string]: number | string - } - -This is a known limitation is discussed in TypeScript's issue tracker [9]_, -where it is suggested that there should be a way to exclude the defined keys -from the index signature, so that it is possible to define a type like -``MovieWithExtraNumber``. - -A workaround is to use intersection: - -.. code-block:: typescript - - interface MovieExtras { - [index: string]: number - } - - interface MovieBase { - name: string - } - - type Movie = MovieExtras & MovieBase - -However, this does not work for assignment: - -.. code-block:: typescript - - const movie: Movie = { - name: "No Country for Old Men", // Type 'string' is not assignable to type 'number' - year: 2007, - } - Alternatives to the ``__extra__`` Magic Field --------------------------------------------- -As it was pointed out in the PEP 705 review comment [11]_, ``__extra__`` -as a field has some disadvantages, including not allowing "__extra__" -as a regular key, requiring special-handling to disallow ``Required`` -and ``NotRequired``. There could be some better alternatives to this -without the above mentioned issues. +As it was pointed out in the `PEP 705 review comment +`__, +``__extra__`` as a field has some disadvantages, including not allowing +"__extra__" as a regular key, requiring special-handling to disallow +``Required`` and ``NotRequired``. There could be some better alternatives to +this without the above mentioned issues. Backwards Compatibility ======================= @@ -585,28 +506,46 @@ equivalent to a TypedDict type with the proposed ``__extra__`` special field, as the value type of all known fields in ``TypedDict`` need to satisfy the is-subtype-of relation with the value type of ``Mapping[...]``. +Requiring Type Compatibility of the Known Fields with ``__extra__`` +------------------------------------------------------------------- + +``__extra__`` restricts the value type for keys that are *unknown* to the +TypedDict type. So the value type of any *known* field is not necessarily +consistent with ``__extra__``'s type, and ``__extra__``'s type is not +necessarily consistent with the value types of all known fields. + +This differs from TypeScript's `Index Signatures +`__ +syntax, which requires all properties' types to match the string index's type. +For example: + +.. code-block:: typescript + + interface MovieWithExtraNumber { + name: string // Property 'name' of type 'string' is not assignable to 'string' index type 'number'. + [index: string]: number + } + + interface MovieWithExtraNumberOrString { + name: string // OK + [index: string]: number | string + } + +This is a known limitation is discussed in `TypeScript's issue tracker +`__, +where it is suggested that there should be a way to exclude the defined keys +from the index signature, so that it is possible to define a type like +``MovieWithExtraNumber``. + Reference Implementation ======================== -pyanalyze has -`experimental support `__ +pyanalyze has `experimental support +`__ for a similar feature. -Reference implementation for this specific proposal, however, is not currently available. - -References -========== - -.. [1] https://discuss.python.org/t/do-we-want-an-exact-typeddict-if-so-how-final-extras-never/44418 -.. [2] https://github.com/python/mypy/issues/7981 -.. [3] https://github.com/python/mypy/issues/4617 -.. [4] https://mail.python.org/archives/list/typing-sig@python.org/thread/66RITIHDQHVTUMJHH2ORSNWZ6DOPM367/ -.. [5] https://github.com/python/mypy/pull/14225 -.. [6] https://discuss.python.org/t/pep-692-using-typeddict-for-more-precise-kwargs-typing/17314/87 -.. [7] https://www.typescriptlang.org/docs/handbook/type-compatibility.html#a-note-on-soundness -.. [8] https://www.typescriptlang.org/docs/handbook/2/objects.html#index-signatures -.. [9] https://github.com/microsoft/TypeScript/issues/17867 -.. [11] https://discuss.python.org/t/pep-705-typeddict-read-only-and-other-keys/36457/6 +Reference implementation for this specific proposal, however, is not currently +available. Acknowledgments ================ From 7e140ce1cba0bf3e853fb9e69c4d5cec214985a5 Mon Sep 17 00:00:00 2001 From: Zixuan James Li Date: Thu, 8 Feb 2024 13:04:49 -0500 Subject: [PATCH 08/19] Update inheritance on interaction with PEP 705. Signed-off-by: Zixuan James Li --- peps/pep-0728.rst | 26 +++++++++++++++++++------- 1 file changed, 19 insertions(+), 7 deletions(-) diff --git a/peps/pep-0728.rst b/peps/pep-0728.rst index 8e3f8fe2e12..e45ac696631 100644 --- a/peps/pep-0728.rst +++ b/peps/pep-0728.rst @@ -252,11 +252,13 @@ function parameters still apply:: Inheritance ----------- -``__extra__`` is inherited the same way as a regular ``key: value_type`` -field. The same rules from :pep:`PEP 589 <589#inheritance>` apply. +``__extra__`` is inherited the same way as a regular ``key: value_type`` field. +As with the other keys, the same rules from :pep:`PEP 589 <589#inheritance>` and +:pep:`PEP 705 <705#inheritance>` apply. We interpret the existing rules in the +context of ``__extra__``. -Additionally, we need to reinterpret the following rule to define how -``__extra__`` interacts with it: +We need to reinterpret the following rule to define how ``__extra__`` interacts +with it: * Changing a field type of a parent TypedDict class in a subclass is not allowed. @@ -274,11 +276,21 @@ accepted to the TypedDict and marks them as non-required. Thus, the above restriction applies to any additional fields defined in a subclass. For each field added in a subclass, all of the following conditions should apply: -- The field is non-required +- If ``__extra__`` is read-only -- The field's value type is consistent with the type of ``__extra__`` + - The field can be either required or non-required -- The type of ``__extra__`` is consistent with the field's value type + - The field's value type is consistent with the type of ``__extra__`` + +- If ``__extra__`` is not read-only + + - The field is non-required + + - The field's value type is consistent with the type of ``__extra__`` + + - The type of ``__extra__`` is consistent with the field's value type + +- If ``__extra__`` is not redeclared, the subclass inherits it as-is. For example:: From 4412012339726b7b41da55ec69679228b0942877 Mon Sep 17 00:00:00 2001 From: Zixuan James Li Date: Thu, 8 Feb 2024 13:47:19 -0500 Subject: [PATCH 09/19] Redo "type consistency" addressing PEP 705 interaction. Also split parts of it into individual sections dicussing changes to TypedDict type consistency with Mapping and dict. Signed-off-by: Zixuan James Li --- peps/pep-0728.rst | 166 ++++++++++++++++++++++++++++++++-------------- peps/test.py | 12 ++++ 2 files changed, 129 insertions(+), 49 deletions(-) create mode 100644 peps/test.py diff --git a/peps/pep-0728.rst b/peps/pep-0728.rst index e45ac696631..d07454dde9f 100644 --- a/peps/pep-0728.rst +++ b/peps/pep-0728.rst @@ -327,37 +327,103 @@ In addition to the set ``S`` of keys of the explicitly defined fields, a TypedDict type that has ``__extra__`` is considered to have an infinite set of fields that all satisfy the following conditions: -- The field is non-required +- If ``__extra__`` is read-only + + - The field's value type is consistent with the type of ``__extra__`` + + - The field's key is not in ``S``. -- The field's value type is consistent with the type of ``__extra__`` +- If ``__extra__`` is not read-only -- The type of ``__extra__`` is consistent with the field's value type + - The field is non-required -- The field's key is not in ``S``. + - The field's value type is consistent with the type of ``__extra__`` -For type checking purposes, let "__extra__" be a pseudo-key to be included -whenever "for each key" or "for each non-required key" is stated in -:pep:`the existing type consistency rules <589#type-consistency>`: + - The type of ``__extra__`` is consistent with the field's value type - Second, a TypedDict type ``A`` is consistent with TypedDict ``B`` if - ``A`` is structurally compatible with ``B``. This is true if and only - if both of these conditions are satisfied: + - The field's key is not in ``S``. - * For each key in ``B``, ``A`` has the corresponding key and the - corresponding value type in ``A`` is consistent with the value type in - ``B``. For each key in ``B``, the value type in ``B`` is also consistent - with the corresponding value type in ``A``. [Edit: if the corresponding key +For type checking purposes, let "__extra__" be a non-required pseudo-field to be +included whenever "for each ... item/key" is stated in +:pep:`the existing type consistency rules from PEP 705 <705#type-consistency>`, +and we modify it as follows: + + A TypedDict type ``A`` is consistent with TypedDict ``B`` if ``A`` is + structurally compatible with ``B``. This is true if and only if all of the + following are satisfied: + + * For each item in ``B``, ``A`` has the corresponding key, unless the item + in ``B`` is read-only, not required, and of top value type + (``ReadOnly[NotRequired[object]]``). [Edit: Otherwise, if the + corresponding key with the same name cannot be found in ``A``, "__extra__" + is considered the corresponding key.] + + * For each item in ``B``, if ``A`` has the corresponding key [Edit: or + "__extra__"], the corresponding value type in ``A`` is consistent with the + value type in ``B``. + + * For each non-read-only item in ``B``, its value type is consistent with + the corresponding value type in ``A``. [Edit: if the corresponding key with the same name cannot be found in ``A``, "__extra__" is considered the corresponding key.] - * For each required key in ``B``, the corresponding key is required in - ``A``. For each non-required key in ``B``, the corresponding key is not - required in ``A``. [Edit: if the corresponding non-required key with the - same name cannot be found in ``A``, "__extra__" is considered the - corresponding key.] + * For each required key in ``B``, the corresponding key is required in ``A``. + For each non-required key in ``B``, if the item is not read-only in ``B``, + the corresponding key is not required in ``A``. + [Edit: if the corresponding key with the same name cannot be found in + ``A``, "__extra__" is considered to be non-required as the corresponding + key.] The following examples illustrate these checks in action. +``__extra__`` puts various restrictions on additional fields for type +consistency checks:: + + class Movie(TypedDict): + name: str + __extra__: int | None + + class MovieDetails(TypedDict): + name: str + year: NotRequired[int] + + details: MovieDetails = {"name": "Kill Bill Vol. 1", "year": 2003} + movie: Movie = details # Not OK. While 'int' is consistent with 'int | None', + # 'int | None' is not consistent with 'int' + +The value type of "year" is not found in ``Movie``, and that it is not +"read-only, not required, and of top value type (``ReadOnly[NotRequired[object]]``)" +thus it needs to be consistent with ``__extra__``'s value type and vice versa:: + + class MovieDetails(TypedDict): + name: str + year: int | None + + details: MovieDetails = {"name": "Kill Bill Vol. 1", "year": 2003} + movie: Movie = details # Not OK. 'year' is not required in 'Movie', + # so it shouldn't be required in 'MovieDetails' either + +Because "year" is absent in ``Movie``, "__extra__" is considered the +corresponding key. "year" being required violates the rule "For each required +key in ``B``, the corresponding key is required in ``A``". + +When ``__extra__`` is defined to be read-only in a TypedDict type, it is possible +for a field to have a narrower type than ``__extra__``'s value type. + + class Movie(TypedDict): + name: str + __extra__: ReadOnly[str | int] + + class MovieDetails(TypedDict): + name: str + year: NotRequired[int] + + details: MovieDetails = {"name": "Kill Bill Vol. 2", "year": 2004} + movie: Movie = details # OK. 'int' is consistent with 'str | int'. + +This behaves the same way as :pep:`705` specified if ``year: ReadOnly[str | int]`` +is a field defined in ``Movie``. + ``__extra__`` as a pseudo-field follows the same rules that other fields have, so when both TypedDict contains ``__extra__``, this check is naturally enforced:: @@ -375,35 +441,13 @@ enforced:: extra_int = extra_str # Not OK. 'str' is inconsistent with 'int' for field '__extra__' extra_str = extra_int # Not OK. 'int' is inconsistent with 'str' for field '__extra__' -When only one of the TypedDict ``t`` has ``__extra__``, then the keys unknown to -``t`` in the other TypedDict correspond to the ``__extra__`` pseudo-field. -Still, the same set of rules from :pep:`589` apply:: - - class MovieNotRequiredYear(TypedDict): - name: str - year: NotRequired[int] - - class MovieRequiredYear(TypedDict): - name: str - year: int - - extra_int: MovieExtraInt = {"name": "Kill Bill Vol. 1"} - required_year: MovieNotRequiredYear = {"name": "Kill Bill Vol. 1"} - not_required_year: MovieRequiredYear = {"name": "Kill Bill Vol. 1", "year": 2003} - - extra_int = required_year # Not OK. Having an extra required field makes - # 'required_year' structurally incompatible with - # 'extra_int' - extra_int = not_required_year # OK. - - required_year = extra_int # Not OK. The required key 'year' is missing in - # 'extra_int' - not_required_year = extra_int # OK. +Interaction with Mapping[KT, VT] +-------------------------------- A TypedDict can be consistent with ``Mapping[...]`` types other than ``Mapping[str, object]`` as long as the union of value types on all fields is consistent with the value type of the ``Mapping[...]`` type. It is an extension -to this rule:: +to this rule from :pep:`589`:: * A TypedDict with all ``int`` values is not consistent with ``Mapping[str, int]``, since there may be additional non-``int`` @@ -413,16 +457,30 @@ to this rule:: For example:: + class MovieExtraStr(TypedDict): + name: str + __extra__: str + extra_str: MovieExtraStr = {"name": "Blade Runner", "summary": ""} str_mapping: Mapping[str, str] = extra_str # OK int_mapping: Mapping[str, int] = extra_int # Not OK. 'int | str' is not consistent with 'int' int_str_mapping: Mapping[str, int | str] = extra_int # OK -Note that because the presence of ``__extra__`` prohibits additional required keys in a -TypedDict type's structural subtypes, we can determine if the TypedDict type -and its structural subtypes will ever have any required key during static -analysis. +Furthermore, type checkers should be able to infer the precise return types for +``values()`` and ``items()`` on a TypedDict:: + + def fun(movie: MovieExtraStr) -> None: + reveal_type(movie.items()) # Revealed type is 'dict_items[str, str]' + reveal_type(movie.values()) # Revealed type is 'dict_values[str, str]' + +Interaction with dict[KT, VT] +-------------------------------- + +Note that because the presence of ``__extra__`` prohibits additional required +keys in a TypedDict type's structural subtypes, we can determine if the +TypedDict type and its structural subtypes will ever have any required key +during static analysis. If there is no required key, the TypedDict type is consistent with ``dict[KT, VT]`` and vice versa if all fields on the TypedDict type satisfy the following @@ -448,6 +506,12 @@ For example:: regular_dict: dict[str, int] = not_required_num # OK f(not_required_num) # OK +In this case, methods that are previously unavailable on a TypedDict are allowed. + + not_required_num.clear() # OK + + reveal_type(not_required_num.popitem()) # OK. Revealed type is tuple[str, int] + Open Issues =========== @@ -562,7 +626,11 @@ available. Acknowledgments ================ -TBD. +Thanks to Jelle Zijlstra for sponsoring this PEP and providing review feedbacks, +Eric Traut who `proposed the original design +` +this PEP iterates on, and Alice Purcell for offering perspective as the author +of :pep:`705`. Copyright ========= diff --git a/peps/test.py b/peps/test.py new file mode 100644 index 00000000000..6c7cab2b082 --- /dev/null +++ b/peps/test.py @@ -0,0 +1,12 @@ +from typing import NotRequired, TypedDict, Unpack + +class Animal(TypedDict): + foo: int + + __extra__: str + + def __extra_field__(self, key: str) -> int: + ... + +animal = Animal({"foo": 10}) +print(Animal.__required_keys__) From 9453e2d0e58443df6df5bc93cc8c6205e19f63aa Mon Sep 17 00:00:00 2001 From: Zixuan James Li Date: Thu, 8 Feb 2024 14:12:27 -0500 Subject: [PATCH 10/19] Minor wording and example changes. Addresses some review feedback. Signed-off-by: Zixuan James Li --- peps/pep-0728.rst | 16 ++++++++++------ 1 file changed, 10 insertions(+), 6 deletions(-) diff --git a/peps/pep-0728.rst b/peps/pep-0728.rst index d07454dde9f..179c2033697 100644 --- a/peps/pep-0728.rst +++ b/peps/pep-0728.rst @@ -57,11 +57,11 @@ Another possible use case for this is a sound way to def fun(entry: Movie | Book) -> None: if "author" in entry: - add_book(entry) # Not OK + reveal_type(entry) # Revealed type is 'Movie | Book' Nothing prevents a ``dict`` that is structurally compatible with ``Movie`` to -have the ``author`` key, and it would be correct for the type checker to reject -the ``add_book`` call. +have the ``author`` key, and it would be correct for the type checker to not +narrow its type. Allowing Extra Fields of a Certain Type --------------------------------------- @@ -85,8 +85,8 @@ from doing this:: fun({"name": "Blade Runner", "year": 1982}) # Not OK While the restriction is enforced when constructing a TypedDict, due to -structural subtyping, the TypedDict is flexible enough to have extra fields that -are not visible through its type. For example:: +structural subtyping, the TypedDict is may have extra fields that are not +visible through its type. For example:: class Movie(MovieBase): year: int @@ -213,7 +213,7 @@ Interaction with PEP 705 When ``__extra__`` is annotated with ``ReadOnly[]``, the extra fields on the TypedDict have the properties of read-only fields. This affects subclassing -according to the inheritance rules specified in :pep:`PEP 705 <589#Inheritance>`. +according to the inheritance rules specified in :pep:`PEP 705 <705#Inheritance>`. Notably, a subclass of the TypedDict type may redeclare the type of ``__extra__`` or extra fields if the TypedDict type declares ``__extra__`` to be read-only. @@ -229,6 +229,10 @@ The extra fields are non-required, regardless of the totality of the TypedDict. Operations that are available to ``NotRequired`` fields should also be available to the extra fields:: + class Movie(TypedDict): + name: str + __extra__: int + def f(movie: Movie) -> None: del movie["name"] # Not OK del movie["year"] # OK From 75916b589c8eedd559358376415574f65859961a Mon Sep 17 00:00:00 2001 From: Zixuan James Li Date: Thu, 8 Feb 2024 14:13:01 -0500 Subject: [PATCH 11/19] Mention the typing spec instead of PEP 589 more when possible. Signed-off-by: Zixuan James Li --- peps/pep-0728.rst | 16 +++++++++------- 1 file changed, 9 insertions(+), 7 deletions(-) diff --git a/peps/pep-0728.rst b/peps/pep-0728.rst index 179c2033697..b507500fed8 100644 --- a/peps/pep-0728.rst +++ b/peps/pep-0728.rst @@ -70,7 +70,7 @@ For supporting API interfaces or legacy codebase where only a subset of possible keys are known, it would be useful to explicitly expect additional keys of certain value types. -However, :pep:`589` is more restrictive on type checking the construction of a +However, the typing spec is more restrictive on type checking the construction of a TypedDict, `preventing the users `__ from doing this:: @@ -156,8 +156,8 @@ There are some advantages to this approach: - Inheritance works naturally. ``__extra__`` defined on a TypedDict will also be available to its subclasses. -- We can build on top of - :pep:`the type consistency rules defined in PEP 589 <589#type-consistency>`. +- We can build on top of the `type consistency rules defined in the typing spec + `__. ``__extra__`` can be treated as a pseudo-key in terms of type consistency. - There is no need to introduce a syntax to specify the type of the extra @@ -174,7 +174,8 @@ the original TypedDict specification. Extra fields are treated as non-required fields having the same type of ``__extra__`` whose keys are allowed when determining -:pep:`supported and unsupported operations <589>`. +`supported and unsupported operations +`__. Using TypedDict Types --------------------- @@ -257,8 +258,9 @@ Inheritance ----------- ``__extra__`` is inherited the same way as a regular ``key: value_type`` field. -As with the other keys, the same rules from :pep:`PEP 589 <589#inheritance>` and -:pep:`PEP 705 <705#inheritance>` apply. We interpret the existing rules in the +As with the other keys, the same rules from +`the typing spec `__ +and :pep:`PEP 705 <705#inheritance>` apply. We interpret the existing rules in the context of ``__extra__``. We need to reinterpret the following rule to define how ``__extra__`` interacts @@ -451,7 +453,7 @@ Interaction with Mapping[KT, VT] A TypedDict can be consistent with ``Mapping[...]`` types other than ``Mapping[str, object]`` as long as the union of value types on all fields is consistent with the value type of the ``Mapping[...]`` type. It is an extension -to this rule from :pep:`589`:: +to this rule from the typing spec:: * A TypedDict with all ``int`` values is not consistent with ``Mapping[str, int]``, since there may be additional non-``int`` From 2ffed81d2461b146021837957803b7ae7e837ae0 Mon Sep 17 00:00:00 2001 From: Zixuan James Li Date: Thu, 8 Feb 2024 14:15:33 -0500 Subject: [PATCH 12/19] Remove an accident. Signed-off-by: Zixuan James Li --- peps/test.py | 12 ------------ 1 file changed, 12 deletions(-) delete mode 100644 peps/test.py diff --git a/peps/test.py b/peps/test.py deleted file mode 100644 index 6c7cab2b082..00000000000 --- a/peps/test.py +++ /dev/null @@ -1,12 +0,0 @@ -from typing import NotRequired, TypedDict, Unpack - -class Animal(TypedDict): - foo: int - - __extra__: str - - def __extra_field__(self, key: str) -> int: - ... - -animal = Animal({"foo": 10}) -print(Animal.__required_keys__) From 3106c1587002cd548f3dbfa9e5fb5bb8162efc93 Mon Sep 17 00:00:00 2001 From: Zixuan James Li Date: Thu, 8 Feb 2024 19:05:12 -0500 Subject: [PATCH 13/19] Be consistent with "key", "item", "value type" and word choices. This follows the style of PEP 705 and the majority of the typing spec to use "key" or "item" instead of "field" depending on the context. It also makes use of "value type" more to be adequately more pedantic. The special key added is now referred to as the "reserved key" to make it clearer that it is not intended for other purposes anymore. It also make use of type aliases like `VT` more for brevity. Signed-off-by: Zixuan James Li --- peps/pep-0728.rst | 202 +++++++++++++++++++++++----------------------- 1 file changed, 102 insertions(+), 100 deletions(-) diff --git a/peps/pep-0728.rst b/peps/pep-0728.rst index b507500fed8..e4e3924dd6b 100644 --- a/peps/pep-0728.rst +++ b/peps/pep-0728.rst @@ -1,5 +1,5 @@ PEP: 728 -Title: TypedDict with Typed Extra Fields +Title: TypedDict with Typed Extra Items Author: Zixuan James Li Sponsor: Jelle Zijlstra Discussions-To: @@ -17,28 +17,28 @@ Post-History: Abstract ======== -This PEP proposes a way to type extra fields for :class:`~typing.TypedDict` using -a using a special ``__extra__: T`` field. -This addresses the need to define a subset of fields that might appear in a -``dict`` while permitting additional fields of a specified type, and the need to -create a closed TypedDict type with ``__extra__: Never``. +This PEP proposes a way to type extra items for :class:`~typing.TypedDict` using +a reserved ``__extra__`` key. This addresses the need to define a subset of +keys that might appear in a ``dict`` while permitting additional items of a +specified type, and the need to create a closed TypedDict type with ``__extra__: +Never``. Motivation ========== -A TypedDict type can annotate the type of each known field from dictionaries. -However, due to structural subtyping, the TypedDict can have extra fields that -are not visible through its type. There is currently not a way to limit or -restrict the type of fields that might be present in the TypedDict type's -structural subtype. +A TypedDict type can annotate the value type of each known item from +dictionaries. However, due to structural subtyping, the TypedDict can have extra +items that are not visible through its type. There is currently not a way to +limit or restrict the value type of items that might be present in the TypedDict +type's structural subtype. Defining a Closed TypedDict Type -------------------------------- In practice, this prevents users from defining a closed TypedDict type -when it is expected that the TypedDict type contains no additional fields. +when it is expected that the TypedDict type contains no additional items. -Due to the possible presence of extra fields, type checker cannot infer more +Due to the possible presence of extra items, type checker cannot infer more precise return types for ``.items()`` and ``.values()`` on a TypedDict. This can also be resolved by `defining a closed TypedDict type `__. @@ -63,7 +63,7 @@ Nothing prevents a ``dict`` that is structurally compatible with ``Movie`` to have the ``author`` key, and it would be correct for the type checker to not narrow its type. -Allowing Extra Fields of a Certain Type +Allowing Extra Items of a Certain Type --------------------------------------- For supporting API interfaces or legacy codebase where only a subset of possible @@ -78,14 +78,14 @@ from doing this:: name: str def fun(movie: MovieBase) -> None: - # movie can have extra fields that are not visible through MovieBase + # movie can have extra items that are not visible through MovieBase ... movie: MovieBase = {"name": "Blade Runner", "year": 1982} # Not OK fun({"name": "Blade Runner", "year": 1982}) # Not OK While the restriction is enforced when constructing a TypedDict, due to -structural subtyping, the TypedDict is may have extra fields that are not +structural subtyping, the TypedDict is may have extra items that are not visible through its type. For example:: class Movie(MovieBase): @@ -94,7 +94,7 @@ visible through its type. For example:: movie: Movie = {"name": "Blade Runner", "year": 1982} fun(movie) # OK -It is not possible to acknowledge the existence of the extra fields through +It is not possible to acknowledge the existence of the extra items through ``in`` check and access them without breaking type safety, even though they might exist from arbitrary structural subtypes of ``MovieBase``:: @@ -115,20 +115,20 @@ Support Additional Keys for ``Unpack`` :pep:`692` adds a way to precisely annotate the types of individual keyword arguments represented by ``**kwargs`` using TypedDict with ``Unpack``. However, -because TypedDict cannot be defined to accept arbitrary extra fields, it is not +because TypedDict cannot be defined to accept arbitrary extra items, it is not possible to `allow additional keyword arguments `__ that are not known at the time the TypedDict is defined. Given the usage of pre-:pep:`692` type annotation for ``**kwargs`` in existing -codebases, it will be valuable to accept and type extra fields on TypedDict so +codebases, it will be valuable to accept and type extra items on TypedDict so that the old typing behavior can be supported in combination with the new ``Unpack`` construct. Rationale ========= -A type that allows extra fields of type ``str`` on a TypedDict can be loosely +A type that allows extra items of type ``str`` on a TypedDict can be loosely described as the intersection between the TypedDict and ``Mapping[str, str]``. `Index Signatures `__ @@ -146,10 +146,10 @@ intersection of types or syntax changes, offering a natural extension to the existing type consistency rules. We propose that we give the dunder attribute ``__extra__`` a special meaning: -When it is defined on a TypedDict type, extra fields are allowed, and their -types should be compatible with the type of ``__extra__``. Different from index -signatures, the types of known fields do not need to be consistent with the type -of ``__extra__``. +When it is defined on a TypedDict type, extra items are allowed, and their types +should be compatible with the value type of ``__extra__``. Different from index +signatures, the types of known items do not need to be consistent with the value +type of ``__extra__``. There are some advantages to this approach: @@ -158,13 +158,12 @@ There are some advantages to this approach: - We can build on top of the `type consistency rules defined in the typing spec `__. - ``__extra__`` can be treated as a pseudo-key in terms of type consistency. + ``__extra__`` can be treated as a pseudo-item in terms of type consistency. -- There is no need to introduce a syntax to specify the type of the extra - fields. +- There is no need to introduce a syntax to specify the type of the extra items. -- We can precisely type the extra fields without making ``__extra__`` the union - of known fields. +- We can precisely type the extra items without making ``__extra__`` the union + of known items. Specification ============= @@ -172,7 +171,7 @@ Specification This specification is structured to parallel :pep:`589` to highlight changes to the original TypedDict specification. -Extra fields are treated as non-required fields having the same type of +Extra items are treated as non-required items having the same type of ``__extra__`` whose keys are allowed when determining `supported and unsupported operations `__. @@ -180,9 +179,9 @@ Extra fields are treated as non-required fields having the same type of Using TypedDict Types --------------------- -For a TypedDict type that has the ``__extra__`` field, during construction, the -value type of each unknown field is expected to be non-required and compatible -with the type of ``__extra__``. For example:: +For a TypedDict type that has the "__extra__" key, during construction, the +value type of each unknown item is expected to be non-required and compatible +with the value type of ``__extra__``. For example:: class Movie(TypedDict): name: str @@ -196,14 +195,14 @@ with the type of ``__extra__``. For example:: In this example, ``__extra__: bool`` does not mean that ``Movie`` has a required string key "__extra__" whose value type is ``bool``. Instead, it specifies that -fields other than "name" have a value type of ``bool`` and are non-required. +keys other than "name" have a value type of ``bool`` and are non-required. The alternative inline syntax is also supported:: Movie = TypedDict("Movie", {"name": str, "__extra__": bool}) -Accessing extra fields is allowed. Type checkers must infer its value type from -the type of ``__extra__``:: +Accessing extra keys is allowed. Type checkers must infer its value type from +the value type of ``__extra__``:: def f(movie: Movie) -> None: reveal_type(movie["name"]) # Revealed type is 'str' @@ -212,23 +211,26 @@ the type of ``__extra__``:: Interaction with PEP 705 ------------------------ -When ``__extra__`` is annotated with ``ReadOnly[]``, the extra fields on the -TypedDict have the properties of read-only fields. This affects subclassing +When ``__extra__`` is annotated with ``ReadOnly[]``, the extra items on the +TypedDict have the properties of read-only items. This affects subclassing according to the inheritance rules specified in :pep:`PEP 705 <705#Inheritance>`. -Notably, a subclass of the TypedDict type may redeclare the type of ``__extra__`` -or extra fields if the TypedDict type declares ``__extra__`` to be read-only. +Notably, a subclass of the TypedDict type may redeclare the value type of +``__extra__`` or extra items if the TypedDict type declares ``__extra__`` to be +read-only. + +More details are discussed in the later sections. Interaction with Totality ------------------------- It is an error to use ``Required[]`` or ``NotRequired[]`` with the special -``__extra__`` field. ``total=False`` and ``total=True`` have no effect on +``__extra__`` item. ``total=False`` and ``total=True`` have no effect on ``__extra__`` itself. -The extra fields are non-required, regardless of the totality of the TypedDict. -Operations that are available to ``NotRequired`` fields should also be available -to the extra fields:: +The extra items are non-required, regardless of the totality of the TypedDict. +Operations that are available to ``NotRequired`` items should also be available +to the extra items:: class Movie(TypedDict): name: str @@ -241,7 +243,7 @@ to the extra fields:: Interaction with ``Unpack`` --------------------------- -For type checking purposes, ``Unpack[TypedDict]`` with extra fields should be +For type checking purposes, ``Unpack[TypedDict]`` with extra items should be treated as its equivalent in regular parameters, and the existing rules for function parameters still apply:: @@ -257,7 +259,7 @@ function parameters still apply:: Inheritance ----------- -``__extra__`` is inherited the same way as a regular ``key: value_type`` field. +``__extra__`` is inherited the same way as a regular ``key: value_type`` item. As with the other keys, the same rules from `the typing spec `__ and :pep:`PEP 705 <705#inheritance>` apply. We interpret the existing rules in the @@ -268,33 +270,33 @@ with it: * Changing a field type of a parent TypedDict class in a subclass is not allowed. -First, it is not allowed to change the type of ``__extra__`` in a subclass unless -it is declared to be ``ReadOnly`` in the superclass:: +First, it is not allowed to change the value type of ``__extra__`` in a subclass +unless it is declared to be ``ReadOnly`` in the superclass:: class Parent(TypedDict): __extra__: int | None class Child(Parent): - __extra__: int # Not OK. Like any other TypedDict field, __extra__'s type cannot be changed + __extra__: int # Not OK. Like any other TypedDict item, __extra__'s type cannot be changed -Second, ``__extra__`` effectively defines the type of any unnamed fields +Second, ``__extra__: T`` effectively defines the value type of any unnamed items accepted to the TypedDict and marks them as non-required. Thus, the above -restriction applies to any additional fields defined in a subclass. For each -field added in a subclass, all of the following conditions should apply: +restriction applies to any additional items defined in a subclass. For each item +added in a subclass, all of the following conditions should apply: - If ``__extra__`` is read-only - - The field can be either required or non-required + - The item can be either required or non-required - - The field's value type is consistent with the type of ``__extra__`` + - The item's value type is consistent with ``T`` - If ``__extra__`` is not read-only - - The field is non-required + - The item is non-required - - The field's value type is consistent with the type of ``__extra__`` + - The item's value type is consistent with ``T`` - - The type of ``__extra__`` is consistent with the field's value type + - ``T`` is consistent with the item's value type - If ``__extra__`` is not redeclared, the subclass inherits it as-is. @@ -317,7 +319,7 @@ For example:: year: NotRequired[int | None] Due to this nature, an important side effect allows us to define a TypedDict -type that disallows additional fields:: +type that disallows additional items:: class MovieFinal(TypedDict): name: str @@ -329,28 +331,28 @@ there can be no other keys in ``MovieFinal`` other than the known ones. Type Consistency ---------------- -In addition to the set ``S`` of keys of the explicitly defined fields, a -TypedDict type that has ``__extra__`` is considered to have an infinite set of -fields that all satisfy the following conditions: +In addition to the set ``S`` of keys of the explicitly defined items, a +TypedDict type that has the item ``__extra__: T`` is considered to have an +infinite set of items that all satisfy the following conditions: - If ``__extra__`` is read-only - - The field's value type is consistent with the type of ``__extra__`` + - The key's value type is consistent with ``T`` - - The field's key is not in ``S``. + - The key is not in ``S``. - If ``__extra__`` is not read-only - - The field is non-required + - The key is non-required - - The field's value type is consistent with the type of ``__extra__`` + - The key's value type is consistent with ``T`` - - The type of ``__extra__`` is consistent with the field's value type + - ``T`` is consistent with the key's value type - - The field's key is not in ``S``. + - The key is not in ``S``. -For type checking purposes, let "__extra__" be a non-required pseudo-field to be -included whenever "for each ... item/key" is stated in +For type checking purposes, let ``__extra__`` be a non-required pseudo-item to +be included whenever "for each ... item/key" is stated in :pep:`the existing type consistency rules from PEP 705 <705#type-consistency>`, and we modify it as follows: @@ -382,7 +384,7 @@ and we modify it as follows: The following examples illustrate these checks in action. -``__extra__`` puts various restrictions on additional fields for type +``__extra__`` puts various restrictions on additional items for type consistency checks:: class Movie(TypedDict): @@ -414,7 +416,7 @@ corresponding key. "year" being required violates the rule "For each required key in ``B``, the corresponding key is required in ``A``". When ``__extra__`` is defined to be read-only in a TypedDict type, it is possible -for a field to have a narrower type than ``__extra__``'s value type. +for an item to have a narrower type than ``__extra__``'s value type. class Movie(TypedDict): name: str @@ -428,9 +430,9 @@ for a field to have a narrower type than ``__extra__``'s value type. movie: Movie = details # OK. 'int' is consistent with 'str | int'. This behaves the same way as :pep:`705` specified if ``year: ReadOnly[str | int]`` -is a field defined in ``Movie``. +is an item defined in ``Movie``. -``__extra__`` as a pseudo-field follows the same rules that other fields have, +``__extra__`` as a pseudo-item follows the same rules that other items have, so when both TypedDict contains ``__extra__``, this check is naturally enforced:: @@ -444,16 +446,16 @@ enforced:: extra_int: MovieExtraInt = {"name": "No Country for Old Men", "year": 2007} extra_str: MovieExtraStr = {"name": "No Country for Old Men", "description": ""} - extra_int = extra_str # Not OK. 'str' is inconsistent with 'int' for field '__extra__' - extra_str = extra_int # Not OK. 'int' is inconsistent with 'str' for field '__extra__' + extra_int = extra_str # Not OK. 'str' is inconsistent with 'int' for item '__extra__' + extra_str = extra_int # Not OK. 'int' is inconsistent with 'str' for item '__extra__' Interaction with Mapping[KT, VT] -------------------------------- -A TypedDict can be consistent with ``Mapping[...]`` types other than -``Mapping[str, object]`` as long as the union of value types on all fields is -consistent with the value type of the ``Mapping[...]`` type. It is an extension -to this rule from the typing spec:: +A TypedDict type can be consistent with ``Mapping[KT, VT]`` types other than +``Mapping[str, object]`` as long as the union of value types on the TypedDict +type is consistent with ``VT``. It is an extension to this rule from the typing +spec:: * A TypedDict with all ``int`` values is not consistent with ``Mapping[str, int]``, since there may be additional non-``int`` @@ -473,8 +475,8 @@ For example:: int_mapping: Mapping[str, int] = extra_int # Not OK. 'int | str' is not consistent with 'int' int_str_mapping: Mapping[str, int | str] = extra_int # OK -Furthermore, type checkers should be able to infer the precise return types for -``values()`` and ``items()`` on a TypedDict:: +Furthermore, type checkers should be able to infer the precise return types of +``values()`` and ``items()`` on such TypedDict types:: def fun(movie: MovieExtraStr) -> None: reveal_type(movie.items()) # Revealed type is 'dict_items[str, str]' @@ -488,13 +490,13 @@ keys in a TypedDict type's structural subtypes, we can determine if the TypedDict type and its structural subtypes will ever have any required key during static analysis. -If there is no required key, the TypedDict type is consistent with ``dict[KT, VT]`` -and vice versa if all fields on the TypedDict type satisfy the following +If there is no required key, the TypedDict type is consistent with ``dict[KT, +VT]`` and vice versa if all items on the TypedDict type satisfy the following conditions: -- ``VT`` is consistent with the value type of the field +- ``VT`` is consistent with the value type of the item -- The value type of the field is consistent with ``VT`` +- The value type of the item is consistent with ``VT`` For example:: @@ -521,12 +523,12 @@ In this case, methods that are previously unavailable on a TypedDict are allowed Open Issues =========== -Alternatives to the ``__extra__`` Magic Field +Alternatives to the ``__extra__`` Reserved Key --------------------------------------------- As it was pointed out in the `PEP 705 review comment `__, -``__extra__`` as a field has some disadvantages, including not allowing +``__extra__`` as a reserved item has some disadvantages, including not allowing "__extra__" as a regular key, requiring special-handling to disallow ``Required`` and ``NotRequired``. There could be some better alternatives to this without the above mentioned issues. @@ -546,19 +548,19 @@ versions as long as the type checker supports it. Rejected Ideas ============== -Allowing Extra Fields without Specifying the Type +Allowing Extra Items without Specifying the Type ------------------------------------------------- ``extra=True`` was originally proposed for defining a TypedDict accept extra -fields regardless of the type, like how ``total=True`` works:: +items regardless of the type, like how ``total=True`` works:: class TypedDict(extra=True): pass -Because it did not offer a way to specify the type of the extra fields, the type -checkers will need to assume that the type of the extra fields are ``Any``, +Because it did not offer a way to specify the type of the extra items, the type +checkers will need to assume that the type of the extra items are ``Any``, which compromises type safety. Furthermore, the current behavior of TypedDict -already allows untyped extra fields to be present in runtime, due to structural +already allows untyped extra items to be present in runtime, due to structural subtyping. Supporting ``TypedDict(extra=type)`` @@ -566,35 +568,35 @@ Supporting ``TypedDict(extra=type)`` This adds more corner cases to determine whether a type should be treated as a type or a value. And it will require more work to support using special forms to -type the extra fields. +type the extra items. While this saves us from reserving an attribute for special use, it will require extra work to implement inheritance, and it is less natural to integrate with generic TypedDicts. -Support Extra Fields with Intersection +Support Extra Items with Intersection -------------------------------------- Supporting intersections in Python's type system requires a lot of careful considerations, and it can take a long time for the community to reach a consensus on a reasonable design. -Ideally, extra fields in TypedDict should not be blocked by work on +Ideally, extra items in TypedDict should not be blocked by work on intersections, nor does it necessarily need to be supported through intersections. Moreover, the intersection between ``Mapping[...]`` and ``TypedDict`` is not -equivalent to a TypedDict type with the proposed ``__extra__`` special field, as -the value type of all known fields in ``TypedDict`` need to satisfy the +equivalent to a TypedDict type with the proposed ``__extra__`` special item, as +the value type of all known items in ``TypedDict`` need to satisfy the is-subtype-of relation with the value type of ``Mapping[...]``. -Requiring Type Compatibility of the Known Fields with ``__extra__`` +Requiring Type Compatibility of the Known Items with ``__extra__`` ------------------------------------------------------------------- ``__extra__`` restricts the value type for keys that are *unknown* to the -TypedDict type. So the value type of any *known* field is not necessarily +TypedDict type. So the value type of any *known* item is not necessarily consistent with ``__extra__``'s type, and ``__extra__``'s type is not -necessarily consistent with the value types of all known fields. +necessarily consistent with the value types of all known items. This differs from TypeScript's `Index Signatures `__ From c1028be0f181cca7f6e9f1596f907cbda33b61fd Mon Sep 17 00:00:00 2001 From: Zixuan James Li Date: Thu, 8 Feb 2024 20:29:57 -0500 Subject: [PATCH 14/19] Apply edits from Jelle. Signed-off-by: Zixuan James Li --- peps/pep-0728.rst | 63 ++++++++++++++++++++++------------------------- 1 file changed, 29 insertions(+), 34 deletions(-) diff --git a/peps/pep-0728.rst b/peps/pep-0728.rst index e4e3924dd6b..6626e37d5e8 100644 --- a/peps/pep-0728.rst +++ b/peps/pep-0728.rst @@ -26,17 +26,17 @@ Never``. Motivation ========== -A TypedDict type can annotate the value type of each known item from -dictionaries. However, due to structural subtyping, the TypedDict can have extra -items that are not visible through its type. There is currently not a way to -limit or restrict the value type of items that might be present in the TypedDict -type's structural subtype. +A :py:data:`typing.TypedDict` type can annotate the value type of each known +item in a dictionary. However, due to structural subtyping, a TypedDict can have +extra items that are not visible through its type. There is currently no way to +restrict the types of items that might be present in the TypedDict type's +structural subtypes. Defining a Closed TypedDict Type -------------------------------- -In practice, this prevents users from defining a closed TypedDict type -when it is expected that the TypedDict type contains no additional items. +The current behavior of TypedDict prevents users from defining a closed +TypedDict type when it is expected that the type contains no additional items. Due to the possible presence of extra items, type checker cannot infer more precise return types for ``.items()`` and ``.values()`` on a TypedDict. This can @@ -60,8 +60,8 @@ Another possible use case for this is a sound way to reveal_type(entry) # Revealed type is 'Movie | Book' Nothing prevents a ``dict`` that is structurally compatible with ``Movie`` to -have the ``author`` key, and it would be correct for the type checker to not -narrow its type. +have the ``author`` key, and under the current specification it would be +incorrect for the type checker to narrow its type. Allowing Extra Items of a Certain Type --------------------------------------- @@ -71,7 +71,7 @@ keys are known, it would be useful to explicitly expect additional keys of certain value types. However, the typing spec is more restrictive on type checking the construction of a -TypedDict, `preventing the users `__ +TypedDict, `preventing users `__ from doing this:: class MovieBase(TypedDict): @@ -85,8 +85,8 @@ from doing this:: fun({"name": "Blade Runner", "year": 1982}) # Not OK While the restriction is enforced when constructing a TypedDict, due to -structural subtyping, the TypedDict is may have extra items that are not -visible through its type. For example:: +structural subtyping, the TypedDict may have extra items that are not visible +through its type. For example:: class Movie(MovieBase): year: int @@ -95,7 +95,7 @@ visible through its type. For example:: fun(movie) # OK It is not possible to acknowledge the existence of the extra items through -``in`` check and access them without breaking type safety, even though they +``in`` checks and access them without breaking type safety, even though they might exist from arbitrary structural subtypes of ``MovieBase``:: def g(movie: MovieBase) -> None: @@ -179,7 +179,7 @@ Extra items are treated as non-required items having the same type of Using TypedDict Types --------------------- -For a TypedDict type that has the "__extra__" key, during construction, the +For a TypedDict type that has the ``__extra__`` key, during construction, the value type of each unknown item is expected to be non-required and compatible with the value type of ``__extra__``. For example:: @@ -194,7 +194,7 @@ with the value type of ``__extra__``. For example:: } In this example, ``__extra__: bool`` does not mean that ``Movie`` has a required -string key "__extra__" whose value type is ``bool``. Instead, it specifies that +string key ``"__extra__"`` whose value type is ``bool``. Instead, it specifies that keys other than "name" have a value type of ``bool`` and are non-required. The alternative inline syntax is also supported:: @@ -216,8 +216,8 @@ TypedDict have the properties of read-only items. This affects subclassing according to the inheritance rules specified in :pep:`PEP 705 <705#Inheritance>`. Notably, a subclass of the TypedDict type may redeclare the value type of -``__extra__`` or extra items if the TypedDict type declares ``__extra__`` to be -read-only. +``__extra__`` or of additional non-extra items if the TypedDict type declares +``__extra__`` to be read-only. More details are discussed in the later sections. @@ -399,21 +399,17 @@ consistency checks:: movie: Movie = details # Not OK. While 'int' is consistent with 'int | None', # 'int | None' is not consistent with 'int' -The value type of "year" is not found in ``Movie``, and that it is not -"read-only, not required, and of top value type (``ReadOnly[NotRequired[object]]``)" -thus it needs to be consistent with ``__extra__``'s value type and vice versa:: - - class MovieDetails(TypedDict): + class MovieWithYear(TypedDict): name: str year: int | None - details: MovieDetails = {"name": "Kill Bill Vol. 1", "year": 2003} + details: MovieWithYear = {"name": "Kill Bill Vol. 1", "year": 2003} movie: Movie = details # Not OK. 'year' is not required in 'Movie', - # so it shouldn't be required in 'MovieDetails' either + # so it shouldn't be required in 'MovieWithYear' either -Because "year" is absent in ``Movie``, "__extra__" is considered the -corresponding key. "year" being required violates the rule "For each required -key in ``B``, the corresponding key is required in ``A``". +Because "year" is absent in ``Movie``, ``__extra__`` is considered the +corresponding key. ``"year"`` being required violates the rule "For each +required key in ``B``, the corresponding key is required in ``A``". When ``__extra__`` is defined to be read-only in a TypedDict type, it is possible for an item to have a narrower type than ``__extra__``'s value type. @@ -432,9 +428,8 @@ for an item to have a narrower type than ``__extra__``'s value type. This behaves the same way as :pep:`705` specified if ``year: ReadOnly[str | int]`` is an item defined in ``Movie``. -``__extra__`` as a pseudo-item follows the same rules that other items have, -so when both TypedDict contains ``__extra__``, this check is naturally -enforced:: +``__extra__`` as a pseudo-item follows the same rules that other items have, so +when both TypedDicts contain ``__extra__``, this check is naturally enforced:: class MovieExtraInt(TypedDict): name: str @@ -483,7 +478,7 @@ Furthermore, type checkers should be able to infer the precise return types of reveal_type(movie.values()) # Revealed type is 'dict_values[str, str]' Interaction with dict[KT, VT] --------------------------------- +----------------------------- Note that because the presence of ``__extra__`` prohibits additional required keys in a TypedDict type's structural subtypes, we can determine if the @@ -634,11 +629,11 @@ available. Acknowledgments ================ -Thanks to Jelle Zijlstra for sponsoring this PEP and providing review feedbacks, +Thanks to Jelle Zijlstra for sponsoring this PEP and providing review feedback, Eric Traut who `proposed the original design ` -this PEP iterates on, and Alice Purcell for offering perspective as the author -of :pep:`705`. +this PEP iterates on, and Alice Purcell for offering their perspective as the +author of :pep:`705`. Copyright ========= From e1a37a123c624304827f236f86b327de663824ed Mon Sep 17 00:00:00 2001 From: Zixuan James Li Date: Thu, 8 Feb 2024 20:34:17 -0500 Subject: [PATCH 15/19] Another grammatical pass. Signed-off-by: Zixuan James Li --- peps/pep-0728.rst | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) diff --git a/peps/pep-0728.rst b/peps/pep-0728.rst index 6626e37d5e8..48f9de23e57 100644 --- a/peps/pep-0728.rst +++ b/peps/pep-0728.rst @@ -38,7 +38,7 @@ Defining a Closed TypedDict Type The current behavior of TypedDict prevents users from defining a closed TypedDict type when it is expected that the type contains no additional items. -Due to the possible presence of extra items, type checker cannot infer more +Due to the possible presence of extra items, type checkers cannot infer more precise return types for ``.items()`` and ``.values()`` on a TypedDict. This can also be resolved by `defining a closed TypedDict type `__. @@ -102,8 +102,8 @@ might exist from arbitrary structural subtypes of ``MovieBase``:: if "year" in movie: reveal_type(movie["year"]) # Error: TypedDict 'MovieBase' has no key 'year' -Some workarounds have already been implemented in response to the need of -allowing extra keys, but none of them is ideal. For mypy, +Some workarounds have already been implemented in response to the need to allow +extra keys, but none of them is ideal. For mypy, ``--disable-error-code=typeddict-unknown-key`` `suppresses type checking error `__ specifically for unknown keys on TypedDict. This sacrifices type safety over @@ -449,7 +449,7 @@ Interaction with Mapping[KT, VT] A TypedDict type can be consistent with ``Mapping[KT, VT]`` types other than ``Mapping[str, object]`` as long as the union of value types on the TypedDict -type is consistent with ``VT``. It is an extension to this rule from the typing +type is consistent with ``VT``. It is an extension of this rule from the typing spec:: * A TypedDict with all ``int`` values is not consistent with @@ -524,9 +524,9 @@ Alternatives to the ``__extra__`` Reserved Key As it was pointed out in the `PEP 705 review comment `__, ``__extra__`` as a reserved item has some disadvantages, including not allowing -"__extra__" as a regular key, requiring special-handling to disallow +"__extra__" as a regular key, requiring special handling to disallow ``Required`` and ``NotRequired``. There could be some better alternatives to -this without the above mentioned issues. +this without the above-mentioned issues. Backwards Compatibility ======================= @@ -553,9 +553,9 @@ items regardless of the type, like how ``total=True`` works:: pass Because it did not offer a way to specify the type of the extra items, the type -checkers will need to assume that the type of the extra items are ``Any``, -which compromises type safety. Furthermore, the current behavior of TypedDict -already allows untyped extra items to be present in runtime, due to structural +checkers will need to assume that the type of the extra items is ``Any``, which +compromises type safety. Furthermore, the current behavior of TypedDict already +allows untyped extra items to be present in runtime, due to structural subtyping. Supporting ``TypedDict(extra=type)`` @@ -582,7 +582,7 @@ intersections. Moreover, the intersection between ``Mapping[...]`` and ``TypedDict`` is not equivalent to a TypedDict type with the proposed ``__extra__`` special item, as -the value type of all known items in ``TypedDict`` need to satisfy the +the value type of all known items in ``TypedDict`` needs to satisfy the is-subtype-of relation with the value type of ``Mapping[...]``. Requiring Type Compatibility of the Known Items with ``__extra__`` From e8ac84e3ed4731d05dc1edeab357f327ee345ac8 Mon Sep 17 00:00:00 2001 From: Zixuan James Li Date: Thu, 8 Feb 2024 20:37:16 -0500 Subject: [PATCH 16/19] More heading alignment fixes. Signed-off-by: Zixuan James Li --- peps/pep-0728.rst | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/peps/pep-0728.rst b/peps/pep-0728.rst index 48f9de23e57..21bd5e09eec 100644 --- a/peps/pep-0728.rst +++ b/peps/pep-0728.rst @@ -64,7 +64,7 @@ have the ``author`` key, and under the current specification it would be incorrect for the type checker to narrow its type. Allowing Extra Items of a Certain Type ---------------------------------------- +-------------------------------------- For supporting API interfaces or legacy codebase where only a subset of possible keys are known, it would be useful to explicitly expect additional keys of @@ -519,7 +519,7 @@ Open Issues =========== Alternatives to the ``__extra__`` Reserved Key ---------------------------------------------- +---------------------------------------------- As it was pointed out in the `PEP 705 review comment `__, @@ -544,7 +544,7 @@ Rejected Ideas ============== Allowing Extra Items without Specifying the Type -------------------------------------------------- +------------------------------------------------ ``extra=True`` was originally proposed for defining a TypedDict accept extra items regardless of the type, like how ``total=True`` works:: @@ -570,7 +570,7 @@ extra work to implement inheritance, and it is less natural to integrate with generic TypedDicts. Support Extra Items with Intersection --------------------------------------- +------------------------------------- Supporting intersections in Python's type system requires a lot of careful considerations, and it can take a long time for the community to reach a @@ -586,7 +586,7 @@ the value type of all known items in ``TypedDict`` needs to satisfy the is-subtype-of relation with the value type of ``Mapping[...]``. Requiring Type Compatibility of the Known Items with ``__extra__`` -------------------------------------------------------------------- +------------------------------------------------------------------ ``__extra__`` restricts the value type for keys that are *unknown* to the TypedDict type. So the value type of any *known* item is not necessarily @@ -627,7 +627,7 @@ Reference implementation for this specific proposal, however, is not currently available. Acknowledgments -================ +=============== Thanks to Jelle Zijlstra for sponsoring this PEP and providing review feedback, Eric Traut who `proposed the original design From 928fef6d8a9a11fddfef2dfeff0a550893bd26ca Mon Sep 17 00:00:00 2001 From: Jelle Zijlstra Date: Thu, 8 Feb 2024 18:30:17 -0800 Subject: [PATCH 17/19] Update peps/pep-0728.rst --- peps/pep-0728.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/peps/pep-0728.rst b/peps/pep-0728.rst index 21bd5e09eec..7f94eac5a07 100644 --- a/peps/pep-0728.rst +++ b/peps/pep-0728.rst @@ -26,7 +26,7 @@ Never``. Motivation ========== -A :py:data:`typing.TypedDict` type can annotate the value type of each known +A :py:class:`typing.TypedDict` type can annotate the value type of each known item in a dictionary. However, due to structural subtyping, a TypedDict can have extra items that are not visible through its type. There is currently no way to restrict the types of items that might be present in the TypedDict type's From 92bc6e18023f50b8d3ea18b85b72ede2b0dde5c7 Mon Sep 17 00:00:00 2001 From: Jelle Zijlstra Date: Thu, 8 Feb 2024 18:31:46 -0800 Subject: [PATCH 18/19] Remove Discussions-To for now, can re-add after it's merged --- peps/pep-0728.rst | 2 -- 1 file changed, 2 deletions(-) diff --git a/peps/pep-0728.rst b/peps/pep-0728.rst index 7f94eac5a07..0a4df6d213e 100644 --- a/peps/pep-0728.rst +++ b/peps/pep-0728.rst @@ -2,14 +2,12 @@ PEP: 728 Title: TypedDict with Typed Extra Items Author: Zixuan James Li Sponsor: Jelle Zijlstra -Discussions-To: Status: Draft Type: Standards Track Topic: Typing Content-Type: text/x-rst Created: 12-Sep-2023 Python-Version: 3.13 -Post-History: .. highlight:: rst From 0d6db289a9e9bc0e90c77ac9610c677f6893ff45 Mon Sep 17 00:00:00 2001 From: Jelle Zijlstra Date: Thu, 8 Feb 2024 18:49:09 -0800 Subject: [PATCH 19/19] Update peps/pep-0728.rst --- peps/pep-0728.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/peps/pep-0728.rst b/peps/pep-0728.rst index 0a4df6d213e..b5d7b2a4581 100644 --- a/peps/pep-0728.rst +++ b/peps/pep-0728.rst @@ -544,7 +544,7 @@ Rejected Ideas Allowing Extra Items without Specifying the Type ------------------------------------------------ -``extra=True`` was originally proposed for defining a TypedDict accept extra +``extra=True`` was originally proposed for defining a TypedDict that accepts extra items regardless of the type, like how ``total=True`` works:: class TypedDict(extra=True):