Final names and attributes

[ilevkivskyi]

Fixes #1214
Fixes python/typing#286
Fixes python/typing#242 (partially, other part is out of scope)

This is a working implementation of final access qualifier briefly discussed at PyCon typing meeting. Final names/attributes can be used to have more static guarantees about semantics of some code and can be used by other tools like mypyc for optimizations.

We can play with this implementation before starting to write an actual PEP.

The basic idea is simple: once declared as final, a name/attribute can't be re-assigned, overridden, or redefined in any other way. For example:

from typing import Final

NO: Final = 0
YES: Final = 255

class BaseEngine:
    RATE: Final[float] = 3000

YES = 1  # Error!

class Engine(BaseEngine):
    RATE = 9000  # Also an error!

For more use cases, examples, and specification, see the docs patch.

Here are some comments on decisions made:

• What can be final? It is hard to say what semantic nodes are important, I started from just module and class constants, but quickly realized it is hard to draw the line without missing some use cases (in particular for mypyc). So I went ahead and implemented all of them, everything can be final: module constants, class-level and instance-level attributes, method, and also classes.
• Two names or one name? I currently use two names Final for assignments and @final for decorators. My PEP8-formatted mind just can't accept @Final :-)
• Should re-exported names keep they const-ness? I think yes, this is a very common pattern, so it looks like this is a sane default.
• What to do with instance-level vs class-level attributes? The point here is that mypy has a common namespace for class attributes. I didn't want to complicate things (including the mental model), so I just decided that one can't have, e.g., a name that is constant on class but assignable on instances, etc. Such use cases are relatively rare, and we can implement this later if there will be high demand for this.

...deferred features:

• I didn't implement any constant propagation in mypy yet. This can be done later on per use-case
  basis. For example:

  fields: Final = [('x', int), ('y', int)]
  NT = NamedTuple('NT', fields)

• Should final classes be like sealed in Scala? I think probably no. On one hand it could be be a nice feature, on other hand it complicates the mental model and is less useful for things like mypyc.
• I don't allow Final in function argument types. One argument is simplicity, another is I didn't see many bugs related to shadowing an argument in function bodies, finally people might have quite different expectations for this. If people will ask, this would be easy to implement.

...and implementation internals:

• There are two additional safety nets that I don't mention in the docs: (a) there can be no TypeVars in the type of class-level constant, (b) instance-level constant can't be accessed on the class object.
• I generate errors for re-definitions in all subclasses, not only in immediate children. I think this is what most people would want: turning something into a constant will flag most re-assignment points.
• It is hard to always get a single error sometimes (I have two error messages, one for re-assignment, another for overriding) because of how mypy creates new Vars. But I think having two errors occasionally is fine.
• We store the final_value for constants initialized with a simple literal, but we never use it. This exists only for tools like mypyc that may use it for optimizations.

The PR is open for questions, suggestions, and comments.

cc @ambv @rchen152 @vlasovskikh

[mvaled]

@med-merchise you were asking for this, I think.

[Michael0x2a]

This is super cool!

Just two comments -- first, it might be nice if there were some tests + docs about Final and control flow. For example, the following program typechecks without an error:

[case testFinalControlFlow]
from typing import Final

for val in [1, 2, 3]:
    x: Final = val

[builtins fixtures/list.pyi]

I found this a bit surprising since we are rebinding x several times/tools like mypyc can no longer assume that 'x' will have the same value within the loop. But then again, maybe this is ok? I think i'd be sort of hard to accidentally do something weird here, and any weirdness that does happen will be localized since x will be actually final after the loop is over...

Not sure, but either way, some clarification might be nice (unless you were planning on doing this in a different PR).

Second, I expect that this proposal will be rejected immediately for being too unpythonic + too messy to implemented, but I figure I might as well get it out of the way... How do we feel about adding a flag to mypy that makes all variables and attributes Final by default?

[ilevkivskyi]

@Michael0x2a

first, it might be nice if there were some tests + docs about Final and control flow

I would actually prohibit Final in loops. Moreover, I would propose to also prohibit Final in if statements, unless the truthiness of condition is known statically (like sys.version > 3.6).

In addition, I just noticed that I forgot about in-place assignments, will fix this soon.

Second, I expect that this proposal will be rejected immediately for being too unpythonic

I am not sure I understand this comment. Rejected immediately by whom? There were no objections during PyCon typing meeting, moreover many people liked the idea.

too messy to implemented

Same story here, sorry :-) Do you want to say that the implementation in this PR is messy?

How do we feel about adding a flag to mypy that makes all variables and attributes Final by default?

I am not sure there is any existing Python program larger than 1k lines that will type-check with such flag. Plus we already have dozens of flags. So I am against this unless people will ask often for this.

[Michael0x2a]

I am not sure I understand this comment. Rejected immediately by whom? There were no objections during PyCon typing meeting, moreover many people liked the idea.

Ah, I meant to say that I expected my proposal to add a flag to make final always-on by default would be rejected immediately by you.

(Which is what just happened -- but it was worth a shot 😄)

[ilevkivskyi]

@JukkaL I implemented only some comments in the docs. I will kill it in the next commit, then you or any one else can finish it whenever.

[JukkaL]

Thanks for the updates! Looks good now, just one minor thing I noticed.

[JukkaL]

Grammar: check -> checks

[ilevkivskyi]

OK, all tests passed, we are good to go. Thanks for review!

[yxliang01]

From python official documentation for Python3.7, I don't see Final being mentioned anywhere. While executing python file with from typing import Final under Python3.6 , it raised ImportError: cannot import name 'Final'. As a result, this is not really usable? Because this is only valid when we lint using "mypy" and we even need to remove all occurrences of Final before execution.

[ilevkivskyi]

@yxliang01 It will be available in typing starting from Python 3.8, on older versions you should use typing_extensions. There is a actually a big blue box saying this in the docs. Next time please read them first, if you have troubles understanding how to use mypy.

[yxliang01]

Thanks for clarification!
p.s. I don't think I have difficulty on using mypy(besides with this Final thing) and I was directly following the repos and found this PR, that's why I missed that doc :(