Indexing TypeVars / need a workaround for higher-kindedness

[bwo]

I'm using mypy 0.630.

I have a container, which can contain various types, some of which contain further types. I want the root of the container to be able to return grandchild types, or failing that to be able to return fully-specified child types. My first thought doesn't work:

from typing import TypeVar, Generic, List

ContainedType = TypeVar('ContainedType', bound='ContainedCls')
SubContained = TypeVar('SubContained')
T = TypeVar('T')

class Root_wrong(Generic[ContainedType[SubContained]]):
    def __init__(self, ct):
        # type: (ContainedType[SubContained]) -> None
        self.contained = ct

    def iter_subcontained(self):
        # type: () -> List[SubContained]
        return self.contained.items

class ContainedCls(Generic[T]):
    def __init__(self, items):
        # type: (List[T]) -> None
        self.items = items

Because TypeVars can't be indexed.

This also doesn't work, because iter_subcontained can't be typed:

class Root(Generic[ContainedType]):
    def __init__(self, ct):
        # type: (ContainedType) -> None
        # I know because of the bound that ContainedType will have a parameter of its own, but I
        # can't refer to it in the `class Root` declaration because TypeVars can't be indexed.
        # and the index to Generic has to be a typevar
        self.contained = ct

    def iter_subcontained(self):
        # what goes here?
        return self.contained.items


reveal_type(Root(ContainedCls([1])).iter_subcontained())  # Any

Alas, this also doesn't work:

class Root2(Generic[ContainedType, SubContained]):
    def __init__(self, ct):
        # type: (ContainedType) -> None
        # I know because of the bound that ContainedType will have a parameter of its own, but I
        # can't refer to it in the `class Root` declaration because TypeVars can't be indexed.
        # and the index to Generic has to be a typevar
        self.contained = ct

    def iter_subcontained(self):
        # type: () -> List[SubContained]
        return self.contained.items


reveal_type(Root2(ContainedCls([1])).iter_subcontained())  # builtins.list[<nothing>]

# not matching is not an error
reveal_type(Root2[ContainedType[int], str](ContainedCls([1])).iter_subcontained())  # builtins.list[str]

This almost works:

class Root3(Generic[T]):
    def __init__(self, ct1, ct2):
        # type: (ContainedCls[T], ContainedCls[T]) -> None
        self.contained = ct1
        self.contained2 = ct2

    def iter_subcontained(self):
        # type: () -> List[T]
        return self.contained.items


class ContainedClsPrime(ContainedCls):
    pass


# no way to ensure that ct1 and ct2 are the same container type
# in fact, they aren't even constrained to have the same T!
x = Root3[int](ContainedCls([1]), ContainedClsPrime(['definitely not an int']))
reveal_type(x.iter_subcontained())  # builtins.list[str]
reveal_type(x.contained2.items[0])  # builtins.int* !!!

But it doesn't constrain the two ContainedCls[T]s to be the same ContainedCls subtype, and it doesn't (this really surprised me) even constrain them to be containing the same T! It also seems wrong to be able to make Root3 generic only in the leaf types, and not in any of the intermediate types.

[ilevkivskyi]

I'm using mypy 0.630

Why not 0.650?

I am not sure I understand what do you actually need here. Do you really need Root to be generic in the ContainedCls (which is obviously not possible since we don't support higher kinds yet)? Or you just want to type iter_subcontained() well? If only the latter, then your example with Root3 actually works if you fix an obvious error (or even better start using --disallow-any-generics):

class Root(Generic[T]):
    def __init__(self, ct, cto):
        # type: (ContainedCls[T], ContainedCls[T]) -> None
        self.contained = ct
        self.contained_other = cto

    def iter_subcontained(self):
        # type: () -> List[T]
        return self.contained.items

class ContainedCls(Generic[T]):
    def __init__(self, items):
        # type: (List[T]) -> None
        self.items = items

class ContainedClsPrime(ContainedCls[T]):  # <- type variable was missing
    pass

# List item 0 has incompatible type "str"; expected "int"
bad = Root[int](ContainedCls([1]), ContainedClsPrime(['definitely not an int']))

good = Root(ContainedCls([1]), ContainedClsPrime([2]))
reveal_type(good.iter_subcontained())  # Revealed type is 'builtins.list[builtins.int*]'

[rohan]

I wanted to add a plug for adding support for higher kinds. Here's my use case:

S = TypeVar('S')
T = TypeVar('T', bound='MyClass')

class MyClass(Generic[S]):
  def __init__(self, obj):
    # type: (S) -> None
    self.obj = obj

  @classmethod
  def create(cls, obj):
    # type: (Type[T], S) -> T[S]
    return cls(obj)

  def get_obj(self):
    # type: () -> S
    return self.obj


class Data:
  def __init__(self, data_point):
    self.data_point = data_point


d = Data('a data point')
x = MyClass.create(d)
print(x.obj.data_point)

This fails, for a variety of reasons:

1. Higher kindedness isn't supported, so T[S] as a type signature fails. Returning T isn't correct, either, since T is a kind. (In particular, it fails with Unsupported type: Type["T"].)
2. Let's say I remove the generic class and accept that obj is Any. Then x.obj is of type Any, so mypy doesn't recognize data_point as a property of Any and throws an error.

One thing I could do is this:

d = Data('a data point')
x = MyClass(d)
obj = x.obj  # type: Data
print(obj.data_point)

However, it seems to me that having to override the type-inference system isn't ideal—I'd rather not have to do it manually and let the magic of mypy put everything in place for me :)

Out of curiosity, what makes adding kind support a hard problem?

@ilevkivskyi / others: if I should open a new issue for this use-case, please let me know.

[ilevkivskyi]

Returning T isn't correct [...]

I actually think this may be what you want. Could you please post a self-consistent example that at least uses create()? Do you want an alternative constructor?

For example (not 100% perfect, but it works, just tried on master):

T = TypeVar('T', bound=C[Any])
S = TypeVar('S')

class C(Generic[S]):
    def __init__(self, x: S) -> None:
        self.x = x
    @classmethod
    def create(cls: Type[T], x: S) -> T:
        return cls(x)

class D(C[int]):
    ...

reveal_type(C.create('yes'))  # Revealed type is '__main__.C[builtins.str]'
D.create('no')  # Argument 1 to "create" has incompatible type "str"; expected "int"
reveal_type(D.create(42))  # Revealed type is '__main__.D'

Although higher order kinds is a useful feature, I don't see that they are desperately needed in this particular example.

[rohan]

Ooops. Fixed. Hmm, let me try that out.

[bwo]

This may or may not be necessary for Rohan's use case, but since D here is not still generic (subclasses from C[int]), it seems a little inflexible—you might have to define in advance subclasses for all the concrete types with which you want them to be instantiated.

[ilevkivskyi]

@bwo I can make D generic and things still reasonably work:

class D(C[S]):
    ...

reveal_type(D.create('yes'))  # Revealed type is '__main__.D[builtins.str]'
D[int].create('no')  # Argument 1 to "create" has incompatible type "str"; expected "int"

Could you please post an exact example that currently doesn't work, but you want it to work?

[bwo]

oh! well, that's unexpected and nice.