Add interactions between Literal and Final #6081

Michael0x2a · 2018-12-18T04:06:37Z

This pull request adds logic to handle interactions between Literal and Final: for example, inferring that foo has type Literal[3] when doing foo: Final = 3.

A few additional notes:

This unfortunately had the side-effect of causing some of the existing tests for Final become noiser. I decided to mostly bias towards preserving the original error messages by modifying many of the existing variable assignments to explicitly use things like Final[int].

I left in the new error messages in a few cases -- mostly in cases where I was able to add them in a relatively tidy way.

Let me know if this needs to be handled differently.
Since mypy uses 'Final', this means that once this PR lands, mypy itself will actually be using Literal types (albeit somewhat indirectly) for the first time.

I'm not fully sure what the ramifications of this are. For example, do we need to detour and add support for literal types to mypyc?
Are there any major users of Final other then mypy? It didn't seem like we were really using it in our internal codebase at least, but I could be wrong about that.

If there are some people who have already started depending on 'Final', maybe we should defer landing this PR until Literal types are more stable to avoid unnecessarily disrupting them. I had to make a few changes to mypy's own source code to get it to type check under these new semantics, for example.

ilevkivskyi · 2018-12-21T03:33:42Z

Before doing the full review, I wanted to point out an important thing. A pattern like this (using a constant as a default value) is very common:

BORDER_SIZE: Final = 10
class Window:
    def __init__(self) -> None:
        self.border = BORDER_SIZE

w = Window()
w.border = 20

START: Final = 42
def count() -> int:
    counter = START
    while True:
        counter += 1
        ...

Currently this works, but with this PR it will emit (quite mysterious) errors. I think the logic of interactions between Literal and Final is not exactly how I imagined it. I think it should work like this:

NUM: Final = 42

x = NUM  # type of x is just int, because there is no literal context

def test(x: Literal[42]) -> None:
    ...
test(NUM)  # this works however

This can be summarized in short as everything should work as if final names are replaced with their values (including chaining).

Note that this cannot be achieved by simply "erasing" the literal type if there is no literal context, because explicit literal types should "propagate":

small: Literal[1, 2, 3] = 1
smalls = [small]  # type here is List[Literal[1, 2, 3]]

I think this should be implemented very roughly like this: final assignments still infer instance types, then analyze_ref_expr can construct a literal type (depending on context) using the final_value (if known).

This may be more tricky with "intelligent indexing", unless we set a literal context there (more like in your option 2, that we recently discussed).

ilevkivskyi · 2018-12-21T03:36:43Z

Also note that all your three questions may become solved/irrelevant if you implement this PR the way I propose.

Michael0x2a · 2018-12-21T05:05:31Z

Hmm, I'll try experimenting with that idea. I agree that preserving the use-case you pointed out is important: I wasn't very happy about having to make changes to mypy itself after breaking that use-case.

I do have some new questions about this though:

If we do x: Final = 4, what should the output of reveal_type(x) be? I think it would be misleading to print out either int or Literal[4] -- maybe we need to come up with some new representation? I guess maybe Final[Literal[4]], but that also seems misleading since that isn't actually a proper type.
What should happen if the user does x: Final[int] = 4 or x: Final[Literal[4]] = 4? I think the two reasonable options are to either disable the "replace final names with their values" behavior if the type is explicitly provided like this, or to disallow users from writing things like this at all. Neither solution feels satisfactory though.
With this change, there's no longer going to be a convenient short-hand way of declaring some variable to be exactly a Literal[...] type. Is it worth introducing back the x: Literal = "foobar" notation to address this gap?

ilevkivskyi · 2018-12-21T13:22:04Z

OK, here are my thoughts:

I think we shouldn't do anything special here, but just reveal the truth. For example:
```
MODE: Final = 'rb'

data_mode = reveal_type(MODE)  # str
open('/home/data', reveal_type(MODE))  # Literal['rb']
```
different revealed type depending on context can already happen e.g. for generic functions, reveal_type() just reveals what do we have in type map for a given expression.
I think nothing special should happen for x: Final[int] = 42, x: Final[object] = 42, or x: Final[Literal[42]] = 42. In a non-literal context we use the explicit or inferred type, in a literal context, we use the final_value (if known).
Final (and ClassVar but there is a bug) is a qualifier, while Literal (like Tuple) is a type constructor. I don't think this (presumably very rare) use case justifies breaking consistency here.
Just to clarify, union of literals still counts as a literal context, so this example will work:
```
NO_DICE: Final = 0

next_dice: Literal[0, 1, 2, 3, 4] = NO_DICE
```

As a general comment, I am not worried about interference between literals and explicit types, since their use cases are almost non-overlapping. A typical use case for explicit type is a "global container": REGISTRY: Final[Dict[str, Type[Model]]] = {}, which can't be used in literal context anyway.

This pull request adds logic to handle interactions between Literal and Final: for example, inferring that `foo` has type `Literal[3]` when doing `foo: Final = 3`. A few additional notes: 1. This unfortunately had the side-effect of causing some of the existing tests for `Final` become noiser. I decided to mostly bias towards preserving the original error messages by modifying many of the existing variable assignments to explicitly use things like `Final[int]`. I left in the new error messages in a few cases -- mostly in cases where I was able to add them in a relatively tidy way. Let me know if this needs to be handled differently. 2. Since mypy uses 'Final', this means that once this PR lands, mypy itself will actually be using Literal types (albeit somewhat indirectly) for the first time. I'm not fully sure what the ramifications of this are. For example, do we need to detour and add support for literal types to mypyc? 3. Are there any major users of `Final` other then mypy? It didn't seem like we were really using it in our internal codebase at least, but I could be wrong about that. If there *are* some people who have already started depending on 'Final', maybe we should defer landing this PR until Literal types are more stable to avoid disrupting them. I had to make a few changes to mypy's own source code to get it to type check under these new semantics, for example.

This commit modifies this PR to make selecting the type of final variables context-sensitive. Now, when we do: x: Final = 1 ...the variable `x` is normally inferred to be of type `int`. However, if that variable is used in a context which expects `Literal`, we infer the literal type. This commit also removes some of the hacks to mypy and the tests that the first iteration added.

# Conflicts: # test-data/unit/check-literal.test

Michael0x2a · 2019-01-04T06:26:56Z

Ok, this should be ready for a second look whenever.

@ilevkivskyi -- I basically went with the approach you suggested (and removed the hacks I added to mypy + the tests.)

ilevkivskyi

Thanks! Overall looks good, I have few minor comments.

mypy/checkexpr.py

ilevkivskyi · 2019-01-04T17:27:34Z

mypy/type_visitor.py

+        if t.final_value is not None:
+            raw_final_value = t.final_value.accept(self)
+            assert isinstance(raw_final_value, LiteralType)
+            final_value = raw_final_value


Why can't you just write:

final_value = t.final_value.accept(self) assert isinstance(final_value, LiteralType)

thus avoiding the extra variable.

It's because t.final_value.accept(self) has a return type of Type, which means we wouldn't be able to assign it to final_value without raising an error or without casting.

I can replace this entire if statement with a cast if you want, similar to what we're doing in visit_tuple_type or visit_typeddict_type below, but I wanted to add a runtime check mostly for peace of mind.

Will my version work if you remove the type comment on final_value?

(This is not important however.)

Alas, it doesn't: I get a 'Argument "final_value" to "Instance" has incompatible type "Optional[Type]"; expected "Optional[LiteralType]"` error a little later on when we use the variable.

ilevkivskyi · 2019-01-04T18:04:31Z

test-data/unit/diff.test

 __main__.C.z
+__main__.C.zi
+__main__.w
 __main__.x


Hm, I am not sure I totally understand why x: Final = 1 and x: Final[int] = 1 appear in the diff. Shouldn't we infer exactly the same type Instance('builtins.int', final_value=Literal[1]) for both of them?

No -- we infer Instance('builtins.int', final_value=None) for the latter. So, since those two declarations have subtly different behavior, I thought it would be important to test both.

(I also went ahead and expanded this test a little. I forgot to update it to test the new semantics earlier.)

What is the reason for inferring different type? Is this because of situations like x: Final[float] = 42? It looks like this is similar to #2008, so I am not going to argue here.

Basically, yeah.

In short, I thought it was more important to make sure that statements like x: Final[float] = 42 and x: float = 42 behaves in the same way as much as possible. This helps ensure that mypy's behavior stays consistent no matter what, even if we lose track of the original literal value (for example, if the user is messing around with generics).

This does mean that x: Final[int] = 9 and x: Final = 9 appear to behave "inconsistently", but I thought that would be less surprising then if x: Final[int] = 9 and x: int = 9 were to behave "inconsistently".

On second thought though, I wonder if it might be worth making x: Final[int] = 4 and x: Final = 4 behave in the same way only if the declared and inferred types are equivalent. That way, we don't run into any issues with expressions like x: Final[float] = 4 since we'd be disabling this "substitution" behavior in that case.

That might be a little too magical though, not sure. LMK if this is an idea you think might be worth exploring.

LMK if this is an idea you think might be worth exploring.

Maybe just open an issue about this. We can decide later when we will have more experience.

ilevkivskyi · 2019-01-04T18:12:09Z

test-data/unit/check-literal.test

+from typing_extensions import Final, Literal
+
+a: Final = 1
+b: Final = (1, 2)


In principle we can support this at some point, but not now. IIRC there is already an issue about this (for common fields in named tuples).

mypy/types.py

# Conflicts: # test-data/unit/check-literal.test

…2a/mypy into add-literal-final-interactions

Michael0x2a · 2019-01-04T19:53:48Z

@ilevkivskyi -- ok, this should be ready for a second look whenever! I made the changes you asked and also added in a few more tests.

ilevkivskyi

I think this LG now, just a couple of optional questions.

ilevkivskyi · 2019-01-07T16:23:41Z

mypy/type_visitor.py

+        if t.final_value is not None:
+            raw_final_value = t.final_value.accept(self)
+            assert isinstance(raw_final_value, LiteralType)
+            final_value = raw_final_value


Will my version work if you remove the type comment on final_value?

(This is not important however.)

ilevkivskyi · 2019-01-07T16:25:14Z

test-data/unit/diff.test

 __main__.C.z
+__main__.C.zi
+__main__.w
 __main__.x


What is the reason for inferring different type? Is this because of situations like x: Final[float] = 42? It looks like this is similar to #2008, so I am not going to argue here.

This was referenced Dec 31, 2018

Literal types tracking issue #5935

Closed

Release 0.660 planning #6130

Closed

Literal types Michael0x2a/peps#1

Closed

Michael0x2a added 2 commits January 3, 2019 11:41

Michael0x2a force-pushed the add-literal-final-interactions branch from 304f4c7 to 7335991 Compare January 4, 2019 01:13

Michael0x2a added 4 commits January 3, 2019 20:11

Revert overzealous Instance.copy_modified changes

85fab7d

Fix self check

dd68816

Remove one last lingering test change

55a38ce

Merge branch 'master' into add-literal-final-interactions

118ff58

# Conflicts: # test-data/unit/check-literal.test

ilevkivskyi reviewed Jan 4, 2019

View reviewed changes

Michael0x2a added 3 commits January 4, 2019 10:34

Merge branch 'master' into add-literal-final-interactions

d37079c

# Conflicts: # test-data/unit/check-literal.test

Merge branch 'add-literal-final-interactions' of github.com:Michael0x…

5969196

…2a/mypy into add-literal-final-interactions

Respond to code review; add a few more tests; add missing logic to fixup

da6807f

Michael0x2a mentioned this pull request Jan 6, 2019

Add docs for Literal types #6150

Merged

ilevkivskyi approved these changes Jan 7, 2019

View reviewed changes

Michael0x2a merged commit 94fe11c into python:master Jan 8, 2019

Michael0x2a deleted the add-literal-final-interactions branch January 8, 2019 05:44

msullivan added a commit that referenced this pull request Jan 8, 2019

Fix type error introduced in #6081

8f948b1

msullivan added a commit that referenced this pull request Jan 8, 2019

Fix type error introduced in #6081 (#6159)

3d248a3

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add interactions between Literal and Final #6081

Add interactions between Literal and Final #6081

Michael0x2a commented Dec 18, 2018

ilevkivskyi commented Dec 21, 2018

ilevkivskyi commented Dec 21, 2018

Michael0x2a commented Dec 21, 2018

ilevkivskyi commented Dec 21, 2018

Michael0x2a commented Jan 4, 2019

ilevkivskyi left a comment

ilevkivskyi Jan 4, 2019

Michael0x2a Jan 4, 2019

ilevkivskyi Jan 7, 2019

Michael0x2a Jan 8, 2019

ilevkivskyi Jan 4, 2019

Michael0x2a Jan 4, 2019

ilevkivskyi Jan 7, 2019

Michael0x2a Jan 8, 2019

Michael0x2a Jan 8, 2019

ilevkivskyi Jan 8, 2019

ilevkivskyi Jan 4, 2019

Michael0x2a commented Jan 4, 2019

ilevkivskyi left a comment

ilevkivskyi Jan 7, 2019

ilevkivskyi Jan 7, 2019

Add interactions between Literal and Final #6081

Add interactions between Literal and Final #6081

Conversation

Michael0x2a commented Dec 18, 2018

ilevkivskyi commented Dec 21, 2018

ilevkivskyi commented Dec 21, 2018

Michael0x2a commented Dec 21, 2018

ilevkivskyi commented Dec 21, 2018

Michael0x2a commented Jan 4, 2019

ilevkivskyi left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Michael0x2a commented Jan 4, 2019

ilevkivskyi left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment