Support namespace packages without namespace-packages setting

[charliermarsh]

Right now, our module discovery doesn't support namespace packages -- or rather, it requires that projects specify namespace packages explicitly via the namespace-packages setting.

I'd like to remove this setting, and ensure that namespace packages "just work".

One option is to take inspiration from Mypy, where they effectively mark a directory without a __init__.py[i] file as a namespace package if it's within a directory that contains an __init__.py[i] file, and provide escape hatches for explicitly specifying package bases.

Another option is to take inspiration Pyright, where AFAICT they allow you to specify multiple roots which are effectively your package bases.

[charliermarsh]

@hauntsaninja - no pressure to respond, but I'm curious if you'd have any advice here based on your experiences in Mypy. This issue is roughly referring to the behavior Mypy has in the SourceFinder. Right now, we also try to map every Python file to a (package root directory, dot-separated module path), but our logic relies on namespace packages being explicitly specified in our configuration file.

[charliermarsh]

It's also plausible that our current approach of requiring namespace packages to be explicitly enumerated is not that bad and fine to preserve for now.

[hauntsaninja]

mypy's default behaviour isn't very good for actual namespace packages... It's good for "accidental" namespace packages, where you have a large tree and people forget to litter __init__.py everywhere.

I'd probably recommend using something like mypy's behaviour with --explicit-package-bases as described in https://mypy.readthedocs.io/en/stable/running_mypy.html#mapping-file-paths-to-modules . mypy doesn't use --explicit-package-bases by default because it would be too breaking. Having this be cwd dependent is something that's worked reasonably well for mypy, but you may wish to think more carefully about it.

Seems potentially a little annoying to list all namespace packages out, seems preferable to list bases.

[charliermarsh]

Thank you, that's helpful.

[hauntsaninja]

An argument against using literally mypy's --explicit-package-bases behaviour is that ruff wants to support monorepos with potentially a bunch of nesting, which would cause listing all bases to be tedious. Most users of mypy I've seen are not trying to run mypy in a single invocation on a varied tree. You may wish to explore something that uses the location of pyproject.toml's instead and lets users mix and match settings a little more.

Honestly, maybe mypy should probably just use sys.path of the target interpreter to determine bases. This is very defensible behaviour and unlike ruff mypy does usually have a target interpreter that has the installs needed.

[charliermarsh]

👍 Yeah monorepo support is important for Ruff. We do currently treat each pyproject.toml as creating a new "source root" -- it's a bit complicated but I wrote it out in detail here (not expecting you to read it, only if curious): https://github.com/astral-sh/ruff/blob/main/CONTRIBUTING.md#import-categorization.

[hmc-cs-mdrissi]

pyright while not python does still call python for sys.path. Beyond helping with namespace packages it also deals helps a lot of import weirdness like handling .pth files/ways sys.path can be modified.

Pylint's namespace package detection is buggy and I haven't seen an approach that really works besides explicitness/sys.path. Here's a test repo that's small but is already enough to lead to issues for number of tools due to there being files with same name that are distinguished if you detect namespace packages correctly.