Return Annotated[Foo, ...] from Foo.__class_getitem__

[jfktrey]

I have a class Foo. It's not generic, so the normal Generic[] syntax isn't useful. It's a common pattern that Foo needs to be annotated, and typing Annotated[Foo, 'some_changing_value'] everywhere is very verbose. I want to be able to write Foo['bar'] and get Annotated[Foo, 'bar'] - is this currently possible?

I've tried this:

P = TypeVar('P')
class Foo:
    @staticmethod
    def __class_getitem__(item: P) -> Annotated[Type[Foo], P]:
        return Annotated[Foo, item]

But there's the obvious complaint from mypy that error: "Foo" expects no type arguments, but 1 given.

There's the related concept of an Annotated alias with a generic type but fixed parameters in PEP 593 - is there a way to express Annotated with fixed type but variable parameters? Even better if it can be done directly on the class itself.

---

This is the closest I've gotten to having something work - seems like it's pretty obviously wrong direction, though, abusing Generic[T].

# annotation.py
from __future__ import annotations

from typing import Annotated, Any, Type, TypeVar, get_type_hints, Generic

B = TypeVar('B')
T = TypeVar('T')
class Foo(Generic[T]):
    @staticmethod
    def __class_getitem__(item: B) -> Annotated[Type[Foo], B]:
        result = Annotated[Foo, item]
        return result

class C:
    foo: Foo['asdf']

$ mypy annotation.py
annotation.py:19: error: Name "asdf" is not defined

[erictraut]

There's currently no way to use non-type annotations in a generic type alias. Type aliases are intended to be used with types, not other forms of annotation. When you use Annotated, you need to spell out the entire annotation.

Could you describe in more detail what you are using the annotations for in this case? What does 'bar' and 'some_changing_value' represent in your use case? I'm wondering if they could be modeled as types within the type system, which would allow you to use normal type annotations and type aliases rather than falling back on Annotated.

[jfktrey]

I see, thank you. In my case, I wanted to avoid creating a descriptor to represent metadata about a field. I can give two example use cases where I've thought about this pattern.

The first is a declarative way to represent bytes in a struct, using an abridged version of a tar file header.

class TarFileHeader:
    name: Bytes[100]
    mode: Bytes[8]
    uid: Bytes[8]
    gid: Bytes[8]
    size: Bytes[12]
    mtime: Bytes[12]
    chksum: Bytes[8]
    typeflag: Bytes[1]
    linkname: Bytes[100]

Where Bytes[12] decomposes into something like Annotated[Bytes, "<12c"]. I then have a wrapper on top of struct.pack and struct.unpack to give a very clean pack/unpack interface.

The second example is a bit more of a stretch and can be solved in better ways - if you wanted to represent the (largely JavaScript) concept of reactivity within a Python class. Each member will be marked as dirty if it depends on another member which has changed.

class SomeWebComponent:
    textInput: Element
    errorMessage: ReactiveElement['textInput']
    clearInputButton: ReactiveElement['textInput']
    sendButton: ReactiveElement['textInput']

# I'd really like to abuse delayed processing of annotations... maybe too far? :)

class SomeWebComponent:
    textInput: Element
    errorMessage: ReactiveElement[textInput]
    clearInputButton: ReactiveElement[textInput]
    sendButton: ReactiveElement[textInput]

I started stubbing out an implementation for this, but there's no way mypy would ever let it fly.

from __future__ import annotations

from pprint import pprint
from typing import Annotated, Generic, TypeVar, cast, get_type_hints

_PT = TypeVar('_PT')
T = TypeVar('T')


class Element:
    ...


class ReactiveElement(Element, Generic[T]):
    @staticmethod
    def __class_getitem__(item: _PT) -> Annotated[ReactiveElement, _PT]:
        return cast(Annotated[ReactiveElement, _PT], Annotated[ReactiveElement, item])


class SomeWebComponent:
    textInput: Element
    errorMessage: ReactiveElement[textInput]
    clearInputButton: ReactiveElement[textInput]
    sendButton: ReactiveElement[textInput]

    """
    mypy hates this.
    
    bigger_annotation_hack.py:22: error: Variable "bigger_annotation_hack.SomeWebComponent.textInput" is not valid as a type
    bigger_annotation_hack.py:22: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
    bigger_annotation_hack.py:23: error: Variable "bigger_annotation_hack.SomeWebComponent.textInput" is not valid as a type
    bigger_annotation_hack.py:23: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
    bigger_annotation_hack.py:24: error: Variable "bigger_annotation_hack.SomeWebComponent.textInput" is not valid as a type
    bigger_annotation_hack.py:24: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
    """

hints = get_type_hints(
    SomeWebComponent,
    locals() | {f: f for f in SomeWebComponent.__annotations__},
    include_extras=True
)
pprint(hints)

"""
{'clearInputButton': typing.Annotated[__main__.ReactiveElement, 'textInput'],
 'errorMessage': typing.Annotated[__main__.ReactiveElement, 'textInput'],
 'sendButton': typing.Annotated[__main__.ReactiveElement, 'textInput'],
 'textInput': <class '__main__.Element'>}
 """

This specific example would do better to have the reactive dependency information in an instance of each ReactiveElement, since there could be cases where one of these fields might not require reactivity. However, I've hit similar cases in other domains where it makes sense to store dependencies in the class itself, rather than in instances of fields of the class.

[jfktrey]

The main idea I'm stuck on is that I find that the pattern used in dataclass's field() totally makes sense given the constraints, but a little clunky. We use the act of assigning to a field in the class as a way to specify metadata about a field, which is what type annotations and Annotated seem to be very useful for.

This rings especially in use cases where you just want to attach metadata, but you may not normally need as much control over how properties are initialized and set for a given class. If you're not careful, what you meant as metadata is now set on all instances of your class.

The final point is that you're lying to mypy, and to do that well, you have to be creative. In the case of

class _SomeMetadata:
    def __init__(self, size: int) -> None:
        self.size: int = size


def some_metadata(size=1) -> Any:
    return _SomeMetadata(size)


class Foo:
    a: bytes = some_metadata(size=100)

If you're going with this pattern, the wrapper function is required in order to make mypy happy - even if you implement __get__ for _SomeMetadata so that it's only returned if you're accessing it via the class and not via instances. There's also no way to lie about the type returned via __new__, thus the wrapper method is required.

That's mostly my struggle - I want to be able to represent the metadata in a compact form via Annotated, but there isn't a convenient way to do so.

[jfktrey]

I actually got this to work, though I've just traded Annotated for Literal. This probably isn't advisable to use.

---

from __future__ import annotations

from dataclasses import dataclass
from typing import Annotated, Generic, Literal, TYPE_CHECKING, TypeVar, cast, get_args, get_type_hints


_PT = TypeVar('_PT')
T = TypeVar('T')


class Foo(Generic[T]):
    @staticmethod
    def __class_getitem__(item: _PT) -> Annotated[Foo, _PT]:
        """
        The `cast` is required. Doing `return Annotated[Foo, item]` directly causes mypy to error:

            annotation_hack.py:14: error: Incompatible return value type (got "object", expected "Foo[Any]")
            annotation_hack.py:37: note: Revealed type is "annotation_hack.Foo[Literal['zzz']]"
        """
        return cast(Annotated[Foo, _PT], Annotated[Foo, item])


@dataclass
class Bar:
    foo: Foo[Literal['zzz']]


bar = Bar(Foo())
type_hints = get_type_hints(Bar, include_extras=True)
annotated_type, annotated_value = get_args(type_hints['foo'])

print(f"Annotations:  {Bar.__annotations__}")
print(f"Hints:        {type_hints}")
print(f"Bar.foo:      {annotated_type, annotated_value}")
"""
Annotations:  {'foo': "Foo[Literal['zzz']]"}
Hints:        {'foo': typing.Annotated[__main__.Foo, typing.Literal['zzz']]}
Bar.foo:      (<class '__main__.Foo'>, typing.Literal['zzz'])
"""

if TYPE_CHECKING:
    reveal_type(Bar.foo)
    """
    annotation_hack.py:42: note: Revealed type is "annotation_hack.Foo[Literal['zzz']]"
    """