Ask PRs to annotate tests

[max-sixty]

• Passes pre-commit run --all-files
• User visible changes (including notable bug fixes) are documented in whats-new.rst

As discussed #5690 (comment)

[github-actions]

Unit Test Results

         6 files           6 suites   56m 28s ⏱️
16 210 tests 14 483 ✔️ 1 727 💤 0 ❌
90 456 runs  82 289 ✔️ 8 167 💤 0 ❌

Results for commit e353086.

[shoyer]

Let's discuss this more back in #5690, but I don't think I would understand what this means from the description here alone.

[mathause]

An alternative would be to use check_untyped_defs. However, this would be much more strict. (I tried that once but ran into mypy errors I could not fix, so I gave up.)

[max-sixty]

An alternative would be to use check_untyped_defs. However, this would be much more strict. (I tried that once but ran into mypy errors I could not fix, so I gave up.)

Ah nice! That would be better. I guess we want it for new code — it would be the herculean effort to fix the old code, most of which doesn't imply a true problem.

FWIW I think most of the false positives are when reassinging variables to different types; e.g. expected =. I guess someone could write a script to increment each new expected = to expected2 = etc, but quite the mission as well.

Is there a way of explaining the TODO in a way that's easy for new contributors to understand? That's probably the closest solution here

[max-sixty]

As discussed on the call — can we get mypy coverage?

[dcherian]

Here are the docs: https://mypy.readthedocs.io/en/stable/command_line.html#report-generation

[max-sixty]

Nice, thanks @dcherian .

Is anyone familiar with coveralls / whether this can be added? Is it possible to have a rule like "new functions need to be typed", by looking at the diff? That requires storing the previous results.

One other alternative is to do the work to get all the test files typed, and then enforce @mathause 's idea of check_untyped_defs on those files only.