Redesign import handling

[JukkaL]

The way mypy deals with imports is quite confusing. I'm proposing that we reconsider how various import-related features work. There are existing issues that cover some of these, but I want to discuss the changes in a central location here, since many of these are related.

I'd love to hear what everybody thinks about this proposal! My idea is that all (or most) of the changes below would be implemented as part of a big overhaul, though likely not in a single mypy release. I think that the current situation is sufficiently problematic that small incremental changes aren't enough.

1. Enable namespace packages by default

Mypy would support namespace packages by default. Currently --namespace-packages is needed, which is confusing for new users, in particular.

2. Generalize specifying namespace packages on the command line

Allow passing namespace packages using all the supported ways of specifying which files to type check on the command line. (See also #5759.)

3. Require package roots to be explicitly specified

To support namespace packages properly, the way mypy currently infers module name from a source file path by looking for __init__.py files can't be used, since namespace packages don't contain __init__.py files. Instead, all package roots would need to be explicitly defined using the mypy module search path. Some package roots will be enabled by default, such as the current directory and typeshed stub directories.

We may want to tweak how the module search path is specified, since many additional users might need to use it.

4. Use modular typeshed, and don't ship (most) third party package stubs with mypy

This is unrelated to namespace packages, but will be essential to allow typeshed to grow significantly. As third-party stubs would now be installed as packages, this would also make PEP-561 and non-PEP-561 packages behave closer to each other (for both you'll have to install some package first). Currently PEP 561 packages require a very different workflow, which is confusing.

This assumes that we'll actually switch to a modular typeshed. I'm hoping that we'll make progress in the next few months.

5. Allow selectively processing arbitrary installed packages

Make it possible to opt-in to type checking individual installed packages, even if they don't support PEP 561. This was proposed in #8545. This would be make it easier to get basic support for many third party packages that don't have stubs and don't support PEP 561.

This isn't enabled by default since some packages won't work properly or at all (C extensions, for example). One option would be disallow this option by default for popular packages with known issues.

6. Rethink import related error messages

Typical import related error messages should give explicit instructions about how to solve the issue. Ignoring missing imports should rarely be needed, and not recommended in the documentation. Examples of what might be suggested:

• Install stub package <package> (this will be particularly important once we move to modular typeshed)
• Install package <package> (in case it's known to support PEP 561)
• Give the command line options that allow using an installed package (in case stubs don't exist, but the package is importable)
• Update the module search path (if it doesn't look like a third-party library module)

7. Add flag to use the old module name inference behavior

Provide a backward compatibility option to use the old behavior of inferring module names, as migrating from the old behavior can be non-trivial for large projects. Still, the key of this proposal is that the new, backward incompatible behavior will be enabled by default, and should be used by most users.

[warsaw]

This may have gotten buried in the original thread, but see this comment about my PEP 420 change not working for nested namespace packages, which needs to be fixed.

[ashb]

This proposal gets a 👍 from me as a dev working on on Apache Airflow

[jnsnow]

PEP420 support by default would be quite nice: it's unintuitive that something that works perfectly fine in pylint and setuptools and at runtime doesn't seem to work in mypy.

It's also quite confusing how, if you were in a git tree root, these two commands are almost, but not quite, the same:

> mypy namespace/
> mypy -p namespace

Checking "as files" vs "as a package" has implications for the fully qualified module names that mypy uses to write configuration against. Both seem to work, but they diverge at a point. It would be nice, as in #5759, to unify this check to eliminate divergent behaviour that is in all likelihood being incurred only accidentally.

I have some questions about what requiring a package root to be specified looks like, but as long as it assumed your CWD as a root, that would probably be a good default. I am not sure what should happen if your CWD is inside a subpackage, though. Anecdotally, one of the wonkiest aspects of pylint is how it tries to "guess" your module structure in order to enable it to be run from CWD inside of a subpackage -- but it does do this guessing incorrectly at times, making the feature unreliable. I have generally settled from only ever running python tools from my package/namespace root and hoping for the best.

One thing I have struggled with as a newcomer to Python in general (I started writing seriously under Python 3.6) is the pipeline between "single python script with no entry point" to "python package with setup.py and one or more modules" all the way to "python package with multiple hierarchical subpackages". It isn't actually always clear in which contexts tools like mypy are expected to run and work properly. Which directory should I be in, how should my imports be written, which invocation of mypy should I use? Unfortunately, there's no consensus; setuptools, pylint, mypy, and other tools all have different behaviors.

At some point, I wonder if it's worth attempting to ratify a "good tool behavior" PEP that explains how various third party utilities should attempt to understand and process file structures, in a bid to try and standardize disparate behavior between popular tools. Maybe something like that would only work after a lot of the Pyproject/setuptools/toml tumult is finally over and done with.

Cheers from the QEMU upstream,
--js

[timstaley]

It's also quite confusing how, if you were in a git tree root, these two commands are almost, but not quite, the same:

> mypy namespace/
> mypy -p namespace

Just a +1 to add that this has bitten me repeatedly, depending on precise combination of .pyi, __init__.py and the two invocations above can all give different scanning behaviour potentially resulting in (mistakenly, silently) unchecked files.

[hauntsaninja]

@timstaley
Bad experiences with mypy's import handling are very much confounded by the presence of two bugs that were recently fixed on master (currently unreleased):

1. When you pass files/directories, mypy silently used to not recurse further if an __init__.py was missing in a subdirectory of a package, fixed in find_sources: find build sources recursively #9614
2. When you passed in packages and --namespace-packages, mypy wouldn't really work beyond a level in the absence of __init__.py, fixed in modulefinder: make -p handle namespace packages correctly #9616

Together, both these fixes should hopefully resolve most cases of mistakenly, silently, unchecked files.

@jnsnow
We're trying to figure out what exactly specifying explicit package roots looks like in #9632. Note that if passing in files/directories was exactly the same as passing in packages/modules, we could just drop one of those input formats. But eg, a lot of people want mypy . to just work. mypy does have some improvable behaviour when e.g. passed a file inside a subpackage.

[jnsnow]

We're trying to figure out what exactly specifying explicit package roots looks like in #9632. Note that if passing in files/directories was exactly the same as passing in packages/modules, we could just drop one of those input formats. But eg, a lot of people want mypy . to just work. mypy does have some improvable behaviour when e.g. passed a file inside a subpackage.

I get you, it's hard to know what the right behavior is. Finding a way to unify the treatment of loose files and packages might be a great thing, though.

I had been typing mypy namespace/ instead, which has the weird side effect of working and checking all of my files, but secretly was chopping off the namespace for how it modeled the tree internally. This made writing mypy.ini quite hard, because I hadn't realized mypy was not capturing the folder name as part of the fully qualified import path. If it had assumed the folder was a valid import root, it might have just worked.

With regards to the "silently didn't realize it was doing the wrong thing" problem, would it be possible to ask for a short report mode where it prints line by line what it checked?

As I recall the "0 errors in 13 files" message is also fairly modern (and fantastic), but maybe it could be augmented:

mypy found 0 errors in 4 files

foo.bar.__init__.py ...... 0 errors
foo.bar.baz.py ........... 0 errors
tests.simpletest.py ...... 0 errors
setup.py ................. 0 errors

It's a bit extreme, but such output allows you to run a diff-style test in CI to prove that your mypy CI is doing exactly what you thought it was, no more, no less. Showing the fully qualified paths might have clued me in to the fact that mypy was perceiving my layout structure wrong, too.

I'll subscribe to traffic on your PR, thank you for taking the usability of mypy configuration seriously! mypy has been a lifesaver for me, so thanks!

[hauntsaninja]

@jnsnow mypy -v can already do this a little. This is something I'd find useful too, so if accepted #9672 will add more information about what files mypy plans on checking to mypy -v. It's not quite as pretty as what you're suggesting (feel free to open a feature request), but it should serve well for debugging purposes.

[hauntsaninja]

Here's a list of potential things remaining. Note that of Jukka's original list, 2, 3, 4 should be completed. 6, 7 are mostly completed or mostly true. I've added an 8, 9, 10.

.1. Enable namespace packages by default (#9636)
.5. Allow selectively processing arbitrary installed packages (#8545)
.8. Separate out --ignore-missing-imports functionality, e.g. --ignore-untyped-imports or --ignore-missing-type-hints (#9789 / #10660 / #10456 etc), or do something like #10044.
.9. Improve stub suggestions (#10619). If ^ introduces new flags, could be a good time to introduce backward incompatible behaviour here if needed.
.10. Improve --exclude. Should also allow it to be specified multiple times (#10310). The name was probably a little too general, since users often expect it to also do the equivalent of follow_imports=skip (e.g. #10820) — maybe it should do that?

[hauntsaninja]

Update: now, 1 and 8 should be completed, 10 is half completed, leaving 5, 9 and half of 10.

If we want an 11, we could do #10600 (comment)

[hauntsaninja]

Closing since basically everything in the original issue is done! I included the items I mentioned as undone in #16472 (and of course they have their own issues, as linked above)