From 5b383429eedf4a998a7c154855135a103f8e3821 Mon Sep 17 00:00:00 2001 From: Alice Purcell Date: Fri, 20 Oct 2023 22:42:29 +0100 Subject: [PATCH] Remove other_keys and readonly flag --- peps/pep-0705.rst | 612 ++++++++++++++++++++++------------------------ 1 file changed, 295 insertions(+), 317 deletions(-) diff --git a/peps/pep-0705.rst b/peps/pep-0705.rst index 36eb020091ea..8021d9f117c2 100644 --- a/peps/pep-0705.rst +++ b/peps/pep-0705.rst @@ -1,5 +1,5 @@ PEP: 705 -Title: TypedDict: Read-only and other keys +Title: TypedDict: Read-only items Author: Alice Purcell Sponsor: Pablo Galindo Discussions-To: https://discuss.python.org/t/pep-705-typeddict-read-only-and-other-keys/36457 @@ -20,14 +20,12 @@ Abstract :pep:`589` defines the structural type :class:`~typing.TypedDict` for dictionaries with a fixed set of keys. As ``TypedDict`` is a mutable type, it is difficult to correctly annotate methods which accept read-only parameters in a way that doesn't prevent valid inputs. -As structural subtypes can add other keys in, it is also difficult for type-checkers to safely define covariant methods like ``update``, or support type narrowing. -This PEP proposes two new ``TypedDict`` flags, ``readonly`` and ``other_keys``, plus an associated type qualifier, ``typing.ReadOnly``. +This PEP proposes a new type qualifier, ``typing.ReadOnly``, to support these usages. Motivation ========== -Representing structured data using (potentially nested) dictionaries with string keys is a common pattern in Python programs. :pep:`589` allows these values to be type checked when the exact type is known up-front, but it is hard to write read-only code that accepts more specific variants: for instance, where fields may be subtypes or restrict a union of possible types. This is an especially common issue when writing APIs for services, which may support a wide range of input structures, and typically do not need to modify their input. - +Representing structured data using (potentially nested) dictionaries with string keys is a common pattern in Python programs. :pep:`589` allows these values to be type checked when the exact type is known up-front, but it is hard to write read-only code that accepts more specific variants: for instance, where values may be subtypes or restrict a union of possible types. This is an especially common issue when writing APIs for services, which may support a wide range of input structures, and typically do not need to modify their input. Pure functions -------------- @@ -86,7 +84,7 @@ This is very repetitive, easy to get wrong, and is still missing important metho Updating nested dicts --------------------- -The structural typing of ``TypedDict`` is supposed to permit writing update functions that only constrain the types of entries they modify:: +The structural typing of ``TypedDict`` is supposed to permit writing update functions that only constrain the types of items they modify:: class HasTimestamp(TypedDict): timestamp: float @@ -118,175 +116,84 @@ However, this no longer works once you start nesting dictionaries:: d["name"] = name update_metadata_timestamp(d) # Type check error: "metadata" is not of type HasTimestamp -This looks like an error, but is simply due to the (unwanted) ability to overwrite the ``metadata`` entry held by the ``HasTimestampedMetadata`` instance with a different ``HasTimestamp`` instance, that may no longer be a ``UserAudit`` instance. +This looks like an error, but is simply due to the (unwanted) ability to overwrite the ``metadata`` item held by the ``HasTimestampedMetadata`` instance with a different ``HasTimestamp`` instance, that may no longer be a ``UserAudit`` instance. It is possible to work around this issue with generics (as of Python 3.11), but it is very complicated, requiring a type parameter for every nested dict. -Type discrimination -------------------- - -Another common idiom in JSON APIs is to discriminate between mutually exclusive choices with a single-entry dictionary, where the key on the dictionary distinguishes between choices, and constrains the associated value type:: - - class Movie(TypedDict): - name: str - director: str - - class Book(TypedDict): - name: str - author: str +Rationale +========= - class EntertainmentMovie(TypedDict): - movie: Movie +These problems can be resolved by removing the ability to update one or more of the items in a ``TypedDict``. This does not mean the items are immutable: a reference to the underlying dictionary could still exist with a different but compatible type in which those items have mutator operations. As such, these are not "final" items; using this term would risk confusion with final attributes, which are fully immutable. These items are "read-only", and we introduce a new ``typing.ReadOnly`` type qualifier for this purpose. - class EntertainmentBook(TypedDict): - book: Book +The ``movie_string`` function in the first motivating example can then be typed as follows:: - Entertainment = EntertainmentMovie | EntertainmentBook + from typing import NotRequired, ReadOnly, TypedDict -Users of this pattern expect type-checkers to allow the following pattern:: + class Movie(TypedDict): + name: ReadOnly[str] + year: ReadOnly[NotRequired[int | None]] - def get_name(entertainment: Entertainment) -> str: - if "movie" in entertainment: - return entertainment["movie"]["name"] - elif "book" in entertainment: - return entertainment["book"]["name"] + def movie_string(movie: Movie) -> str: + if movie.get("year") is None: + return movie["name"] else: - # Theoretically unreachable but common defensive coding - raise ValueError("Unexpected entertainment type") - -However, type-checkers will actually raise an error on this code; mypy, for instance, will complain that ``TypedDict "EntertainmentBook" has no key "movie"`` on the third line. This is because ``TypedDict`` does not prevent instances from having keys not specified in the type, and so the check ``"movie" in entertainment`` can return True for an ``EntertainmentBook``. - -Users can alternatively use a non-total ``TypedDict`` instead of a union:: - - class Entertainment(TypedDict, total=False): - movie: Movie - book: Book - -This ensures the ``get_name`` example type-checks correctly, but it no longer encodes the constraint that exactly one key must be present, meaning other valid code raises spurious type-check failures. In practice, we tend to see code using types like this either casting to the correct type, with the associated risk of mistakes, or moving the ``in`` checks to dedicated ``TypeGuard`` functions, reducing readability. - -Rationale -========= - -The first two motivating examples can be solved by removing the ability to update one or more of the entries in a ``TypedDict``. This does not mean the entries are immutable; a reference to the underlying dictionary could still exist with a different but compatible type in which those entries have mutator operations. As such, these are not "final" entries; using this term would risk confusion with final attributes, which are fully immutable. These entries are "readonly". + return f'{movie["name"]} ({movie["year"]})' -To support this, we propose adding a new boolean flag to ``TypedDict``, ``readonly``, which when set to True, removes all mutator operations from the type:: +A mixture of read-only and non-read-only items is permitted, allowing the second motivating example to be correctly type-hinted:: - from typing import NotRequired, TypedDict - - class Movie(TypedDict, readonly=True): - name: str - director: str + class HasTimestamp(TypedDict): + timestamp: float - class Book(TypedDict, readonly=True): - name: str - author: str + class HasTimestampedMetadata(TypedDict): + metadata: ReadOnly[HasTimestamp] -In addition to these benefits, by flagging arguments of a function as read-only (by using a read-only ``TypedDict`` like ``Movie`` or ``Book``), it makes explicit not just to typecheckers but also to users that the function is not going to modify its inputs, which is usually a desireable property of a function interface. + def update_metadata_timestamp(d: HasTimestampedMetadata) -> None: + d["metadata"]["timestamp"] = now() -A new ``typing.ReadOnly`` type qualifier allows removing the ability to mutate individual entries, permitting a mixture of readonly and mutable entries. This is necessary for supporting the second motivating example, updating nested dicts:: + class Logs(HasTimestamp): + loglines: list[str] class UserAudit(TypedDict): name: str - metadata: ReadOnly[Logs] - -This PEP only proposes making ``ReadOnly`` valid in a ``TypedDict``. A possible future extension would be to support it in additional contexts, such as in protocols. - -Finally, to support type discrimination, we add a second flag to ``TypedDict``, ``other_keys``, which when set to ``typing.Never``, prevents instances from holding any key not explicitly listed in the type:: - - class EntertainmentMovie(TypedDict, readonly=True, other_keys=Never): - movie: Movie + metadata: Logs - class EntertainmentBook(TypedDict, readonly=True, other_keys=Never): - book: Book + def rename_user(d: UserAudit, name: str) -> None: + d["name"] = name + update_metadata_timestamp(d) # Now OK - Entertainment = EntertainmentMovie | EntertainmentBook +In addition to these benefits, by flagging arguments of a function as read-only (by using a ``TypedDict`` like ``Movie`` with read-only items), it makes explicit not just to typecheckers but also to users that the function is not going to modify its inputs, which is usually a desireable property of a function interface. - def get_name(entertainment: Entertainment) -> str: - if "movie" in entertainment: - return entertainment["movie"]["name"] - elif "book" in entertainment: - return entertainment["book"]["name"] - else: - raise ValueError("Unexpected entertainment type") +This PEP only proposes making ``ReadOnly`` valid in a ``TypedDict``. A possible future extension would be to support it in additional contexts, such as in protocols. -Note this is a subset of the functionality of the `unmerged proposal of PEP-728 `_. Specification ============= -``TypedDict`` will gain two new flags: ``other_keys`` and ``readonly``. A new ``typing.ReadOnly`` type qualifier is added. - -``other_keys`` flag -------------------- - -The optional ``other_keys`` flag to ``TypedDict`` can have the value ``typing.Never``, indicating that instances may only contain keys explicitly listed in the type:: - - class Album(TypedDict, other_keys=Never): - name: str - year: int - - class AlbumExtra(Album, TypedDict): - band: str # Runtime error - -Type-checkers may rely on this restriction:: - - def album_keys(album: Album) -> Collection[Literal['name', 'year']]: - # Type checkers may permit this, but should error if Album did not specify `other_keys=Never` - return album.keys() - -Type-checkers should prevent operations that would violate this restriction:: - - class AlbumExtra(TypedDict, other_keys=Never): - name: str - year: int - band: str - - album: AlbumExtra = { "name": "Flood", year: 1990, band: "They Might Be Giants" } - album_keys(album) # Type check error: extra key 'band' - -This PEP does not propose supporting any other values than ``other_keys=Never``. Future or concurrent PEPs may extend this flag to permit other types. - -``readonly`` flag ------------------ - -The optional boolean ``readonly`` flag to ``TypedDict``, when ``True``, indicates that no mutator operations (``__setitem__``, ``__delitem__``, ``update``, etc.) will be permitted:: - - class NamedDict(TypedDict, readonly=True): - name: str - - def get_name(d: NamedDict) -> str: - return d["name"] - - def set_name(d: NamedDict, name: str) -> None: - d["name"] = name # Type check error: cannot modify a read-only entry - -The ``readonly`` flag defaults to ``False``. +A new ``typing.ReadOnly`` type qualifier is added. ``typing.ReadOnly`` type qualifier ---------------------------------- -The ``typing.ReadOnly`` type qualifier is used to indicate that a variable declared in a ``TypedDict`` definition may not be mutated by any operation performed on instances of the ``TypedDict``:: +The ``typing.ReadOnly`` type qualifier is used to indicate that an item declared in a ``TypedDict`` definition may not be mutated (added, modified, or removed) by any operation performed on variables of that type:: from typing import ReadOnly - class BandAndAlbum(TypedDict): - band: str - album: ReadOnly[Album] - -The ``readonly`` flag is equivalent to marking all entries as ``ReadOnly[]``, guaranteeing no entries are missed by mistake. To avoid potential confusion, it is an error to use both ``readonly=True`` and ``ReadOnly[]``:: + class Band(TypedDict): + name: str + members: ReadOnly[list[str]] - class Band(TypedDict, readonly=True): - name: ReadOnly[str] # Runtime error: redundant ReadOnly qualifier - members: Collection[str] + blur: Band = {"name": "blur", members: []} + blur["name"] = "Blur" # OK: "name" is not read-only + blur["members"] = ["Damon Albarn"] # Type check error: "members" is read-only + blur["members"].append("Damon Albarn") # OK: list is mutable Alternative functional syntax ----------------------------- -The :pep:`alternative functional syntax <589#alternative-syntax>` for TypedDict also supports these features:: +The :pep:`alternative functional syntax <589#alternative-syntax>` for TypedDict also supports the new type qualifier:: - EntityBand = TypedDict('EntityBand', {'band': Band}, readonly=True, other_keys=Never) - BandAndAlbum = TypedDict(`BandAndAlbum', {'band': str, 'album': ReadOnly[Album]}) + Band = TypedDict("Band", {"name": str, "members": ReadOnly[list[str]]}) Interaction with other special types ------------------------------------ @@ -310,235 +217,260 @@ This is consistent with the behavior introduced in :pep:`655`. Inheritance ----------- -To avoid potential confusion, it is an error to have a read-only type extend a non-read-only type:: - - class BandAndAlbum(TypedDict): - band: str - album: ReadOnly[Album] - - class BandAlbumAndLabel(BandAndAlbum, readonly=True): # Runtime error - label: str +Subclasses can redeclare read-only items as non-read-only, allowing them to be mutated:: + class NamedDict(TypedDict): + name: ReadOnly[str] -It is also an error to have a type without ``other_keys`` specified extend a type with ``other_keys=Never``:: - - class Building(TypedDict, other_keys=Never): + class Album(NamedDict): name: str - address: str - - class Museum(Building): # Runtime error - pass - -It is valid to have a non-read-only type extend a read-only one. The subclass will not be read-only, but any keys not redeclared in the subclass will remain read-only:: - - class NamedDict(TypedDict, readonly=True): - name: str - - class Album(NamedDict, TypedDict): year: int album: Album = { name: "Flood", year: 1990 } - album["year"] = 1973 # OK - album["name"] = "Dark Side Of The Moon" # Type check error: "name" is read-only + album["year"] = 1973 + album["name"] = "Dark Side Of The Moon" # OK: "name" is not read-only in Album -Subclasses can redeclare read-only entries as non-read-only, allowing them to be mutated:: +If a read-only item is not redeclared, it remains read-only:: - class Album(NamedDict, TypedDict): - name: str + class Album(NamedDict): year: int album: Album = { name: "Flood", year: 1990 } - album["year"] = 1973 # OK - album["name"] = "Dark Side Of The Moon" # Also OK now + album["name"] = "Dark Side Of The Moon" # Type check error: "name" is read-only in Album -Subclasses can narrow value types of read-only entries:: +Subclasses can narrow value types of read-only items:: - class AlbumCollection(TypedDict, readonly=True): - albums: Collection[Album] + class AlbumCollection(TypedDict): + albums: ReadOnly[Collection[Album]] - class RecordShop(AlbumCollection, TypedDict): + class RecordShop(AlbumCollection): name: str - albums: list[Album] + albums: ReadOnly[list[Album]] # OK: "albums" is read-only in AlbumCollection -Subclasses can also require keys that are read-only but not required in the superclass:: +Subclasses can require items that are read-only but not required in the superclass:: - class OptionalName(TypedDict, readonly=True): - name: NotRequired[str] + class OptionalName(TypedDict): + name: ReadOnly[NotRequired[str]] - class Person(OptionalName, TypedDict): - name: Required[str] + class RequiredName(OptionalName): + name: ReadOnly[Required[str]] - person: Person = {} # Type check error: "name" required + d: RequiredName = {} # Type check error: "name" required -Note that these are just consequences of structural typing, but they are highlighted here as the behavior now differs from the rules specified in :pep:`589`. +Subclasses can combine these rules:: + + class OptionalIdent(TypedDict): + ident: ReadOnly[NotRequired[str | int]] -Finally, subclasses can have ``other_keys=Never`` even if the superclass does not:: + class User(OptionalIdent): + ident: str # Required, mutable, and not an int - class Person(OptionalName, other_keys=Never): - name: Required[str] +Note that these are just consequences of structural typing, but they are highlighted here as the behavior now differs from the rules specified in :pep:`589`. Type consistency ---------------- -*This section updates the type consistency rules introduced in* :pep:`589` *to cover the new features in this PEP. In particular, any pair of types that do not use the new features will be consistent under these new rules if (and only if) they were already consistent.* - -A TypedDict type with ``other_keys=Never`` is consistent with ``Mapping[str, V]``, where ``V`` is the union of all its value types. For instance, the following type is consistent with ``Mapping[str, int | str]``:: - - class Person(TypedDict, other_keys=Never): - name: str - age: int +*This section updates the type consistency rules introduced in* :pep:`589` *to cover the new feature in this PEP. In particular, any pair of types that do not use the new feature will be consistent under these new rules if (and only if) they were already consistent.* A TypedDict type ``A`` is consistent with TypedDict ``B`` if ``A`` is structurally compatible with ``B``. This is true if and only if all of the following are satisfied: -* For each key in ``B``, ``A`` has the corresponding key and the corresponding value type in ``A`` is consistent with the value type in ``B``, unless the key in ``B`` is of type ``ReadOnly[NotRequired[Any]]``, in which case it may be missing in ``A`` provided ``A`` allows other keys. -* For each non-read-only key in ``B``, the corresponding value type in ``B`` is also consistent with the corresponding value type in ``A``. +* For each item in ``B``, ``A`` has the corresponding key, unless the item in ``B`` is read-only, not required, and of top value type (e.g. ``ReadOnly[NotRequired[Any]]``). +* For each item in ``B``, if ``A`` has the corresponding key, the corresponding value type in ``A`` is consistent with the value type in ``B``. +* For each non-read-only item in ``B``, its value type is consistent with the corresponding value type in ``A``. * For each required key in ``B``, the corresponding key is required in ``A``. -* For each non-read-only, non-required key in ``B``, the corresponding key is not required in ``A``. -* If ``B`` does not allow other keys, then ``A`` does not allow other keys. -* If ``B`` does not allow other keys, then for each key in ``A``, ``B`` has the corresponding key. +* For each non-required key in ``B``, if the item is not read-only in ``B``, the corresponding key is not required in ``A``. Discussion: -* All non-specified keys in a type that allows other keys are implicitly of type ``ReadOnly[NotRequired[Any]]`` (or ``ReadOnly[NotRequired[Unknown]]`` in pyright). +* All non-specified items in a TypedDict implicitly have value type ``ReadOnly[NotRequired[Any]]`` (or ``ReadOnly[NotRequired[Unknown]]`` in pyright). -* Read-only value types behave covariantly, as they cannot be mutated. This is similar to container types such as ``Sequence``, and different from non-read-only value types, which behave invariantly. Example:: +* Read-only items behave covariantly, as they cannot be mutated. This is similar to container types such as ``Sequence``, and different from non-read-only items, which behave invariantly. Example:: - class A(TypedDict, readonly=True): - x: int | None + class A(TypedDict): + x: ReadOnly[int | None] class B(TypedDict): x: int def f(a: A) -> None: - print(a['x'] or 0) + print(a["x"] or 0) - b: B = {'x': 1} + b: B = {"x": 1} f(b) # Accepted by type checker -* A TypedDict type ``A`` with no explicit key ``'x'`` that allows other keys is not consistent with a TypedDict type with a non-required key ``'x'``, since at runtime the key ``'x'`` could be present and have an incompatible type (which may not be visible through ``A`` due to structural subtyping). The only exception to this rule is if ``'x'`` is non-required, read-only and of type ``object`` (or ``Any`` or pylance's ``Unknown``). - -* A TypedDict type ``A`` with no key ``'x'`` that does not allow other keys may be consistent with a TypedDict type with a read-only, non-required key ``'x'``. Example:: - - class A(TypedDict, total=False, readonly=True): - y: int +* A TypedDict type ``A`` with no explicit key ``'x'`` is not consistent with a TypedDict type ``B`` with a non-required key ``'x'``, since at runtime the key ``'x'`` could be present and have an incompatible type (which may not be visible through ``A`` due to structural subtyping). The only exception to this rule is if the item in ``B`` is read-only, and the value type is a top type (``object``, ``Any`` or pylance's ``Unknown``). For example:: - class B(TypedDict, other_keys=Never): + class A(TypedDict): x: int - def f(a: A) -> int: - return a.get("y", 0) + class B(TypedDict): + x: int + y: ReadOnly[NotRequired[object]] - def g(b: B) -> None: - b["x"] = f(b) # Accepted by type checker + a: A = { "x": 1 } + b: B = a # Accepted by type checker -Union Operation +Merge operation --------------- -The union operation creates a new dictionary with the merged keys and values of its two operands. As such, the result should be consistent with any type that can hold the possible key-value pairs, not just types compatible with the operand types. For example:: +The merge operation (``d1 | d2``) creates a new dictionary with the merged keys and values of its two operands. As such, the result should be consistent with any type that can hold the possible items. For example:: - class A(TypedDict, readonly=True, other_keys=Never): - x: int + class A(TypedDict): + x: ReadOnly[int] - class B(TypedDict, total=False, readonly=True, other_keys=Never): - x: str + class B(TypedDict): + x: ReadOnly[NotRequired[str]] class C(TypedDict): x: int | str def union_a_b(a: A, b: B) -> C: - # Accepted by type-checker, even though C is not read-only and - # allows other keys: + # Accepted by type-checker, even though A, B and C are mutually inconsistent return a | b -This is different from the usual compatibility rules, where the result of an operation has a defined type which the variable it is assigned to must be consistent with. A similar situation occurs with ``TypedDict`` and ``copy()`` or ``deepcopy()``. +Type checkers should conform to the following rules, even if they are not expressed correctly in the typeshed due to language constraints. -If the union of two TypedDict objects of type ``A`` and ``B`` are assigned to a TypedDict of type ``C``, the type checker should verify that: +If the merger of two TypedDict objects of type ``A`` and ``B`` are assigned to a TypedDict of type ``C``, the type checker should verify that: -* if ``C`` does not allow other keys, neither ``A`` nor ``B`` allow other keys -* if ``C`` does not allow other keys, it contains all keys found in either ``A`` or ``B`` -* if a key ``'x'`` is found in ``A`` and ``C``, its type in ``A`` is consistent with its type in ``C``. -* if a key ``'x'`` is found in ``B`` and ``C``, its type in ``B`` is consistent with its type in ``C``. -* if a key ``'x'`` is required in ``C``, it is required in either ``A`` or ``B``. +* for each key in ``C``, one of the following holds: + + * the associated value type in ``C`` is of top type + * the key is required in ``B`` + * the key is in both ``A`` and ``B`` + +* if a key is in both ``A`` and ``C``, but not required in ``B``, the associated value type in ``A`` is consistent with the value type in ``C`` +* if a key is in both ``B`` and ``C``, the associated value type in ``B`` is consistent with the value type in ``C`` +* if a key is required in ``C``, it is required in either ``A`` or ``B`` Notes: -* The read-only status of the keys does not matter. A key can be read-only on just ``A``, just ``B``, or just ``C``, or any combination. -* A key found on ``A`` or ``B`` may be missed off ``C`` if it allows other keys. Type-checkers may however choose to flag this edge-case with a warning or error in some circumstances, if it is found to be a source of mistakes. +* The read-only status of the items does not matter. An item can be read-only on just ``A``, just ``B``, or just ``C``, or any combination. +* A key found on ``A`` or ``B`` may theoretically be missed off ``C``. Type-checkers or other linters may however choose to flag this edge-case with a warning or error, if it is found to be a source of mistakes. +* An item in ``A`` can be inconsistent with ``C`` if it is required in ``B``. For example:: + + class A(TypedDict): + x: str + y: int + + class B(TypedDict): + x: int + + class C(TypedDict): + x: int # Note: A's definition of "x" is not consistent with this + y: int + + a: A = { "x": "three", "y": 4 } + b: B = { "x": 3 } + c: C = a | b # Accepted by type checker: "x" will always come from b + +Update method +------------- + +In addition to existing type checking rules, type checkers should error if a TypedDict with a read-only item is updated with another TypedDict that declares that key:: -Update Operations + class A(TypedDict): + x: ReadOnly[int] + y: int + + a1: A = { "x": 1, "y": 2 } + a2: A = { "x": 3, "y": 4 } + a1.update(a2) # Type check error: "x" is read-only in A + +Unless the declared value is of bottom type:: + + class B(TypedDict): + x: NotRequired[typing.Never] + y: ReadOnly[int] + + def update_a(a: A, b: B) -> None: + a.update(b) # Accepted by type checker: "x" cannot be set on b + +copy and deepcopy ----------------- -Previously, ``clear()`` and ``popitem()`` were rejected by type checkers on TypedDict objects, as they could remove required keys, some of which may not be directly visible because of structural subtyping. However, these methods should be allowed on TypedDicts objects with all keys non-read-only and non-required and with no other keys allowed:: +As with the merge operation, the result of applying the ``copy`` method, ``copy.copy`` or ``copy.deepcopy`` to a ``TypedDict`` should be consistent with any type that can hold the possible items, not just types the operand is consistent with. For example:: - class A(TypedDict, total=False, other_keys=Never): - x: int - y: str + class A(TypedDict): + x: ReadOnly[int] - a: A = { "x": 1, "y": "foo" } - a.popitem() # Accepted by type checker - a.clear() # Accepted by type checker + class B(TypedDict): + x: int | str -``update`` has been difficult to type correctly due to the open nature of TypedDict objects. Keys not specified on the type could still be present (and constrained) due to structural subtyping, meaning type safety could be accidentally violated. For instance:: + a: A = { "x": 3 } + b: B = copy(a) # Accepted by type checker: items are compatible - class B(TypedDict, total=False): - x: int +Type checkers should conform to the following rules, even if they are not expressed correctly in the typeshed due to language constraints. - def update_b(b1: B, b2: B) -> None: - b1.update(b2) +If a shallow copy of TypedDict type ``A`` is assigned to TypedDict type ``B``, the type checker should verify that: - class C(B, TypedDict, total=False): - y: int +* for each key in ``B``, one of the following holds: + + * the associated value type in of top type + * the key is in ``A``, and the associated value type in ``A`` is consistent with the value type in ``B`` + +* if a key is required in ``B``, it is required in ``A`` + +If a deep copy of TypedDict type ``A`` is assigned to TypedDict type ``B``, the type checker should verify that: + +* for each key in ``B``, one of the following holds: + + * the associated value type in of top type + * the key is in ``A``, and a deep copy of the associated value type in ``A`` could be assigned to the value type in ``B`` + +* if a key is required in ``B``, it is required in ``A`` + +Notes: - class D(B, TypedDict, total=False): - y: str +* The read-only status of the items does not matter. An item can be read-only on just ``A`` or just ``B``. +* A key found on ``A`` may theoretically be missed off ``B``. Type-checkers or other linters may however choose to flag this edge-case with a warning or error, if it is found to be a source of mistakes. +* Type checkers should support a deep copy of a recursive structural type to a different recursive structural type. For instance:: - c: C = { "x": 1, "y": 2 } - d: D = { "x": 3, "y": "foo" } - update_b(c, d) # c is no longer a C at runtime + class A(TypedDict): + v: int | str + x: NotRequired[A] -Both mypy and pyright currectly permit this usage, however, as the only viable alternative has been to prevent calling ``update`` at all. + class B(TypedDict): + v: int + x: NotRequired[C] -With the addition of ``other_keys``, it becomes possible to more accurately type the update method: + class C(TypedDict): + v: str + x: NotRequired[B] -* Declare a new read-only TypedDict type that does not allow other keys -* Copy all non-read-only entries to it -* Make all entries read-only and non-required -* Union this with an iterable of matching key-value pairs + b: B = { "v": 1, "x": { "v": "two", "x": { "v": 3 } } } + a: A = deepcopy(b) # OK -For instance:: + class D(TypedDict): + v: int + x: NotRequired[E] - class Example(TypedDict): - a: int - b: NotRequired[str] - c: ReadOnly[int] + class E(TypedDict): + v: float + x: NotRequired[D] - class ExampleUpdateDict(TypedDict, total=False, readonly=True, other_keys=Never): - a: int - b: str - # c is not present as it is read-only in Example + d: D = { "v": 1, "x": { "v": 1.1 } } + a: A = deepcopy(d) # Type check error: "v" is of type float in E - ExampleUpdateEntry = tuple[Literal["a"], int] | tuple[Literal["b"], str] - ExampleUpdate = ExampleUpdateDict | Iterable[ExampleUpdateEntry] +* Type checkers may choose to reject deep copying between complex recursive types for algorithmic complexity reasons. However, the error should be explicit that this is a type checker limitation, and not due to an unsound assignment in the user code. -Type checkers should permit any type compatible with this TypedDict to be passed into the update operation. As with :pep:`589`, they may choose to continue permitting TypedDict types that allow other keys as well, to avoid generating false positives. Keyword argument typing ----------------------- -:pep:`692` introduced ``Unpack`` to annotate ``**kwargs`` with a ``TypedDict``. Marking one or more of the entries of a ``TypedDict`` used in this way as read-only will have no effect on the type signature of the method, since all keyword arguments are read-only by design in Python. However, it *will* prevent the entry from being modified in the body of the function:: +:pep:`692` introduced ``Unpack`` to annotate ``**kwargs`` with a ``TypedDict``. Marking one or more of the items of a ``TypedDict`` used in this way as read-only will have no effect on the type signature of the method, since all keyword arguments are read-only by design in Python. However, it *will* prevent the item from being modified in the body of the function:: class Args(TypedDict): key1: int key2: str - class ReadonlyArgs(TypedDict, readonly=True): - key1: int - key2: str + class ReadOnlyArgs(TypedDict): + key1: ReadOnly[int] + key2: ReadOnly[str] class Function(Protocol): def __call__(self, **kwargs: Unpack[Args]) -> None: ... - def impl(self, **kwargs: Unpack[ReadonlyArgs]) -> None: + def impl(self, **kwargs: Unpack[ReadOnlyArgs]) -> None: kwargs["key1"] = 3 # Type check error: key1 is readonly fn: Function = impl # Accepted by type checker: function signatures are identical @@ -547,14 +479,14 @@ Keyword argument typing Backwards compatibility ======================= -This PEP adds new features to ``TypedDict``, so code that inspects ``TypedDict`` types will have to change to support types using the new features. This is expected to mainly affect type-checkers. +This PEP adds a new feature to ``TypedDict``, so code that inspects ``TypedDict`` types will have to change to support types using it. This is expected to mainly affect type-checkers. Security implications ===================== There are no known security consequences arising from this PEP. -How to Teach This +How to teach this ================= Suggestion for changes to the :mod:`typing` module, in line with current practice: @@ -563,88 +495,134 @@ Suggestion for changes to the :mod:`typing` module, in line with current practic * Add ``typing.ReadOnly``, linked to TypedDict and this PEP. * Add the following text to the TypedDict entry: -By default, keys not specified in a TypedDict may still be present. Instances can be restricted to only the named keys with the ``other_keys`` flag. *insert example, perhaps using ``in`` to illustrate the benefit* +Individual items can be excluded from mutate operations using ReadOnly, allowing them to be read but not changed. This is useful when the exact type of the value is not known yet, and so modifying it would break structural subtypes. *insert example* -Individual keys can be excluded from mutate operations using ReadOnly, allowing them to be read but not changed. This is useful when the exact type of the value is not known yet, and so modifying it would break structural subtypes. *insert example* - -If all keys on a TypedDict should be read-only, the ``readonly`` flag can be used as a shorthand. *insert example* - -Reference Implementation +Reference implementation ======================== No complete reference implementation exists yet. pyright 1.1.310 ships with a partial implementation of the ReadOnly qualifier. -Rejected Alternatives +Rejected alternatives ===================== A TypedMapping protocol type ---------------------------- -An earlier version of :pep:`705` proposed a ``TypedMapping`` protocol type, behaving much like a read-only TypedDict but without the constraint that the runtime type be a ``dict``. The behavior described in the current version of this PEP could then be obtained by inheriting a TypedDict from a TypedMapping. This has been set aside for now as more complex, without a strong use-case motivating the additional complexity. +An earlier version of this PEP proposed a ``TypedMapping`` protocol type, behaving much like a read-only TypedDict but without the constraint that the runtime type be a ``dict``. The behavior described in the current version of this PEP could then be obtained by inheriting a TypedDict from a TypedMapping. This has been set aside for now as more complex, without a strong use-case motivating the additional complexity. -A higher-order Readonly type +A higher-order ReadOnly type ---------------------------- -A generalized higher-order type could be added that removes mutator methods from its parameter, e.g. ``ReadOnly[MovieRecord]``. For a TypedDict, this would be like adding ``readonly=True`` to the declaration. This would naturally want to be defined for a wider set of types than just TypedDict subclasses, and also raises questions about whether and how it applies to nested types. We decided to keep the scope of this PEP narrower. +A generalized higher-order type could be added that removes mutator methods from its parameter, e.g. ``ReadOnly[MovieRecord]``. For a TypedDict, this would be like adding ``ReadOnly`` to every item, including those declared in superclasses. This would naturally want to be defined for a wider set of types than just TypedDict subclasses, and also raises questions about whether and how it applies to nested types. We decided to keep the scope of this PEP narrower. -Preventing other keys with the typing.final decorator ------------------------------------------------------ +Calling the type ``Readonly`` +----------------------------- -Instead of adding an ``other_keys`` flag to TypedDict, treat classes decorated with :func:`~typing.final` as disallowing other keys. This makes intuitive sense for TypedDict as it stands now: preventing adding any other keys guarantees no other types will be structurally compatible, so it is effectively final. There is also partial support for this idiom in mypy and pyright, which both use it as a way to achieve type discrimination. However, if any keys are read-only, preventing adding any other keys does **not** make the type final any more, so using the decorator this way seems incorrect. For example:: +``Read-only`` is generally hyphenated, and it appears to be common convention to put initial caps onto words separated by a dash when converting to CamelCase. This appears consistent with the definition of CamelCase on Wikipedia: CamelCase uppercases the first letter of each word. That said, Python examples or counter-examples, ideally from the core Python libraries, or better explicit guidance on the convention, would be greatly appreciated. - class Foo: ... - class Bar(Foo): ... +A readonly flag +--------------- - @final - class FooHolder(TypedDict, readonly=True): - item: Foo +Earlier versions of this PEP introduced a boolean flag that would ensure all items in a TypedDict were read-only:: - @final - class BarHolder(FooHolder, readonly=True): - item: Bar + class Movie(TypedDict, readonly=True): + name: str + year: NotRequired[int | None] + + movie: Movie = { "name": "A Clockwork Orange" } + movie["year"] = 1971 # Type check error: "year" is read-only -Extending a ``TypedDict`` to refine the types is a reasonable feature, but the above code looks like it should raise a runtime error. Should ``@final`` be modified to allow inheritance? Should users be prevented from using this pattern? +However, this led to confusion when inheritance was introduced:: -More context for this can be found on `pyright issue 5254 `_. + class A(TypedDict): + key1: int + + class B(A, TypedDict, readonly=True): + key2: int -We recommend type checkers treat decorating a TypedDict type with final as identical to setting ``other_keys=Never``, if they continue to support the idiom for backwards compatibility, but reject any use of final on a TypedDict with read-only keys. Once ``other_keys`` is adopted, they may also wish to deprecate use of final on TypedDicts entirely. + b: B = { "key1": 1, "key2": 2 } + b["key1"] = 4 # Accepted by type checker: "key1" is not read-only -Using different casing for ``readonly`` keyword or ``ReadOnly`` type --------------------------------------------------------------------- +It would be reasonable for someone familiar with ``frozen``, on seeing just the definition of B, to assume that the whole type was read-only. On the other hand, it would be reasonable for someone familiar with ``total`` to assume that read-only only applies to the current type. -It appears to be common convention to put an initial caps onto words separated by a dash when converting to CamelCase, but to drop the dash completely when converting to snake_case. Django uses ``readonly``, for instance. This appears consistent with the definition of both on Wikipedia: snake_case replaces spaces with dashes, while CamelCase uppercases the first letter of each word. That said, more examples or counterexamples, ideally from the core Python libraries, or better explicit guidance on the convention, would be greatly appreciated. +The original proposal attempted to eliminate this ambiguity by making it both a type check and a runtime error to define ``B`` in this way. This was still a source of surprise to people expecting it to work like ``total``. -Mandate unsound type narrowing ------------------------------- +Given that no extra types could be expressed with the ``readonly`` flag, it has been removed from the proposal to avoid ambiguity and surprise. -The main use-case we are aware of for ``other_keys=Never`` (and the current workaround of final-decorated TypedDict types) is to simplify type discrimination, as shown in the motivation section. +Preventing unspecified keys in TypedDicts +----------------------------------------- -By comparison, TypeScript handles this edge-case by ignoring the possibility of instances of one type in the union having undeclared keys. If a variable is known to be of type ``A | B`` and an ``in`` check is done using a key not explicitly declared on ``B``, it is assumed no instance of ``B`` will pass that check. While technically unsound, this a common enough idiom that it could fall under the recommendation in :pep:`589` that "potentially unsafe operations may be accepted if the alternative is to generate false positive errors for idiomatic code". +Consider the following "type discrimination" code:: -This user request has been rejected multiple times by type checkers, however, suggesting the community prefers strict type-safety over idiomatic code here. + class A(TypedDict): + foo: int -Make the ``other_keys`` flag a boolean --------------------------------------- + class B(TypedDict): + bar: int -Since ``other_keys`` can only effectively take two values, ``Never`` or absent, it was originally proposed as a boolean flag, with ``other_keys=False`` equivalent to the current ``other_keys=Never``. However, the `unmerged proposal of PEP-728 `_ provides equivalent functionality when restricting other types to ``Never``, so this proposal was updated to use comparable syntax, to make it clearer how the proposals intersect. + def get_field(d: A | B) -> int: + if "foo" in d: + return d["foo"] # !!! + else: + return d["bar"] -Use a reserved ``__extra__`` key --------------------------------- +This is a common idiom, and other languages like Typescript allow it. Technically, however, this code is unsound: ``B`` does not declare ``foo``, but instances of ``B`` may still have the key present, and the associated value may be of any type:: + + class C(TypedDict): + foo: str + bar: int -The `unmerged proposal of PEP-728 `_ proposes different syntax for disallowing other keys:: + c: C = { "foo": "hi", "bar" 3 } + b: B = c # OK: C is structurally compatible with B + v = get_field(b) # Returns a string at runtime, not an int! - class EntertainmentMovie(TypedDict, readonly=True): - movie: Movie - __extra__: Never +mypy rejects the definition of ``get_field`` on the marked line with the error ``TypedDict "B" has no key "foo"``, which is a rather confusing error message, but is caused by this unsoundness. + +One option for correcting this would be to explicitly prevent ``B`` from holding a ``foo``:: + + class B(TypedDict): + foo: NotRequired[Never] + bar: int + + b: B = c # Type check error: key "foo" not allowed in B + +However, this requires every possible key that might be used to discriminate on to be explicitly declared in every type, which is not generally feasible. A better option would be to have a way of preventing all unspecified keys from being included in ``B``. mypy supports this using the ``@final`` decorator from :pep:`591`:: + + @final + class B(TypedDict): + bar: int + +The reasoning here is that this prevents ``C`` or any other type from being considered a "subclass" of ``B``, so instances of ``B`` can now be relied on to never hold the key ``foo``, even though it is not explicitly declared to be of bottom type. + +With the introduction of read-only items, however, this reasoning would imply type-checkers should ban the following:: + + @final + class D(TypedDict): + field: ReadOnly[Collection[str]] + + @final + class E(TypedDict): + field: list[str] + + e: E = { "field": ["value1", "value2"] } + d: D = e # Error? + +The conceptual problem here is that TypedDicts are structural types: they cannot really be subclassed. As such, using ``@final`` on them is not well-defined; it is certainly not mentioned in :pep:`591`. + +An earlier version of this PEP proposed resolving this by adding a new flag to ``TypedDict`` that would explicitly prevent other keys from being used, but not other kinds of structural compatibility:: + + class B(TypedDict, other_keys=Never): + bar: int -This new key does not function like other keys -- for instance, it is implicitly ``NotRequired`` but cannot be explicitly marked as such. The author of this PEP prefers the asymmetry of using a keyword argument to set expectations that it does not behave like other key declarations, and others have provided similar feedback on the PR. + b: B = c # Type check error: key "foo" not allowed in B -However, this PEP will be updated to match whatever syntax the PEP-728 author decides to go with. +However, during the process of drafting, the situation changed: -Leave other_keys to PEP-728 ---------------------------- +* pyright, which previously worked similarly to mypy in this type discrimination case, `changed to allow the original example without error `_, despite the unsoundness, due to it being a common idiom +* mypy has `an open issue `_ to follow the lead of pyright and Typescript and permit the idiom as well +* a `draft of PEP-728 `_ was created that is a superset of the ``other_keys`` functionality -This PEP could drop the ``other_keys`` proposal entirely rather than propose a limited subset of it. However, as this PEP affects the unofficial status-quo of using final to disallow other keys, it seems important to both highlight that issue and propose a solution. +As such, there is less urgency to address this issue in this PEP, and it has been deferred to PEP-728. Copyright