Document new source finding behaviour

This should cover the current state on master, as implemented
across python#9742, python#9683, python#9632, python#9616, python#9614, etc.

This will need to be changed if we can make `--namespace-packages` the
default (python#9636).

docs/source/command_line.rst

@@ -110,10 +110,17 @@ imports.
     passed on the command line, the ``MYPYPATH`` environment variable,
     and the :confval:`mypy_path` config option.
 
-    Note that this only affects import discovery -- for modules and
-    packages explicitly passed on the command line, mypy still
-    searches for ``__init__.py[i]`` files in order to determine the
-    fully-qualified module/package name.
+    This flag also affects how mypy determines fully qualified module names for
+    files, modules and packages explicitly passed on the command line. See
+    :ref:`Mapping file paths to modules <mapping-paths-to-modules>` for details.
+
+.. option:: --explicit-package-bases
+
+    This flag tells mypy to that the current directory, the ``MYPYPATH``
+    environment variable, and the :confval:`mypy_path` config option as the
+    directories in which top-level packages are located. This option is only
+    useful in conjunction with :option:`--namespace-packages`. See :ref:`Mapping
+    file paths to modules <mapping-paths-to-modules>` for details.
 
 .. option:: --ignore-missing-imports
 

docs/source/running_mypy.rst

@@ -24,8 +24,11 @@ actual way mypy type checks your code, see our
 Specifying code to be checked
 *****************************
 
-Mypy lets you specify what files it should type check in several
-different ways.
+Mypy lets you specify what files it should type check in several different ways.
+
+Note that if you use namespace packages (in particular, packages without
+``__init__.py``), you'll need to specify :option:`--namespace-packages <mypy
+--namespace-packages>`.
 
 1.  First, you can pass in paths to Python files and directories you
     want to type check. For example::
@@ -342,52 +345,69 @@ hard-to-debug errors.
 Mapping file paths to modules
 *****************************
 
-One of the main ways you can tell mypy what files to type check
-is by providing mypy the paths to those files. For example::
+One of the main ways you can tell mypy what to type check
+is by providing mypy a list of paths. For example::
 
     $ mypy file_1.py foo/file_2.py file_3.pyi some/directory
 
 This section describes how exactly mypy maps the provided paths
 to modules to type check.
 
-- Files ending in ``.py`` (and stub files ending in ``.pyi``) are
-  checked as Python modules.
-
-- Files not ending in ``.py`` or ``.pyi`` are assumed to be Python
-  scripts and checked as such.
-
-- Directories representing Python packages (i.e. containing a
-  ``__init__.py[i]`` file) are checked as Python packages; all
-  submodules and subpackages will be checked (subpackages must
-  themselves have a ``__init__.py[i]`` file).
-
-- Directories that don't represent Python packages (i.e. not directly
-  containing an ``__init__.py[i]`` file) are checked as follows:
-
-  - All ``*.py[i]`` files contained directly therein are checked as
-    toplevel Python modules;
-
-  - All packages contained directly therein (i.e. immediate
-    subdirectories with an ``__init__.py[i]`` file) are checked as
-    toplevel Python packages.
-
-One more thing about checking modules and packages: if the directory
-*containing* a module or package specified on the command line has an
-``__init__.py[i]`` file, mypy assigns these an absolute module name by
-crawling up the path until no ``__init__.py[i]`` file is found.
-
-For example, suppose we run the command ``mypy foo/bar/baz.py`` where
-``foo/bar/__init__.py`` exists but ``foo/__init__.py`` does not.  Then
-the module name assumed is ``bar.baz`` and the directory ``foo`` is
-added to mypy's module search path.
-
-On the other hand, if ``foo/bar/__init__.py`` did not exist, ``foo/bar``
-would be added to the module search path instead, and the module name
-assumed is just ``baz``.
-
-If a script (a file not ending in ``.py[i]``) is processed, the module
-name assumed is ``__main__`` (matching the behavior of the
-Python interpreter), unless :option:`--scripts-are-modules <mypy --scripts-are-modules>` is passed.
+- mypy will check all paths provided that correspond to files.
+
+- mypy will recursively discover all Python files ending in ``.py`` (and stub
+  files ending in ``.pyi``) in all directory paths provided.
+
+- For each file to be checked, mypy will attempt to associate the file (e.g.
+  ``project/foo/bar/baz.py``) with a fully qualified module name (e.g.
+  ``foo.bar.baz``). The directory the package is in (``project``) is then
+  add to mypy's module search paths.
+
+How mypy determines fully qualified module names depends on if the options
+:option:`--namespace-packages <mypy --namespace-packages>` and
+:option:`--explicit-package-bases <mypy --explicit-package-bases>` are set.
+
+First, if :option:`--namespace-packages <mypy --namespace-packages>` is off,
+mypy will rely solely upon the presence of ``__init__.py[i]`` files to determine
+the fully qualified module name. That is, mypy will crawl up the directory tree
+for as long as it continues to find ``__init__.py`` (or ``__init__.pyi``) files.
+
+Second, if :option:`--namespace-packages <mypy --namespace-packages>` is on, but
+:option:`--explicit-package-bases <mypy --explicit-package-bases>` is off, mypy
+will allow for the possibility that directories without ``__init__.py[i]`` are
+packages. Specifically, mypy will look at all parent directories of the file and
+use the location of the highest ``__init__.py[i]`` in the directory tree to
+determine the top-level package.
+
+For example, say your directory tree consists solely of ``pkg/__init__.py`` and
+``pkg/a/b/c/d/mod.py``. When determining ``mod.py``'s fully qualified module
+name, mypy will look at ``pkg/__init__.py`` and conclude that the associated
+module name is ``pkg.a.b.c.d.mod``.
+
+You'll notice that that method still relies on ``__init__.py``. If you can't put
+an ``__init__.py`` in your top-level package, but still wish to pass paths (as
+opposed to packages or modules using the ``-p`` or ``-m`` flags),
+:option:`--explicit-package-bases <mypy --explicit-package-bases>` provides a
+solution.
+
+Third, with :option:`--explicit-package-bases <mypy --explicit-package-bases>`,
+mypy will locate the nearest parent directory that is a member of the
+``MYPYPATH`` environment variable, the :confval:`mypy_path` config or is the
+current working directory. mypy will then use the relative path to determine the
+fully qualified module name.
+
+For example, say your directory tree consists solely of
+``src/namespace_pkg/mod.py``. If you run the command following command, mypy
+will correctly associate ``mod.py`` with ``namespace_pkg.mod``::
+
+    $ MYPYPATH=src mypy --namespace-packages --explicit-package-bases .
+
+If you pass a file not ending in ``.py[i]``, the module name assumed is
+``__main__`` (matching the behavior of the Python interpreter), unless
+:option:`--scripts-are-modules <mypy --scripts-are-modules>` is passed.
+
+Passing :option:`-v <mypy -v>` will show you the files and associated module
+names that mypy will check.
 
 
 .. _finding-imports: