Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support namespace packages without namespace-packages setting #6114

Open
charliermarsh opened this issue Jul 27, 2023 · 7 comments
Open

Support namespace packages without namespace-packages setting #6114

charliermarsh opened this issue Jul 27, 2023 · 7 comments
Labels
core Related to core functionality

Comments

@charliermarsh
Copy link
Member

charliermarsh commented Jul 27, 2023

Right now, our module discovery doesn't support namespace packages -- or rather, it requires that projects specify namespace packages explicitly via the namespace-packages setting.

I'd like to remove this setting, and ensure that namespace packages "just work".

One option is to take inspiration from Mypy, where they effectively mark a directory without a __init__.py[i] file as a namespace package if it's within a directory that contains an __init__.py[i] file, and provide escape hatches for explicitly specifying package bases.

Another option is to take inspiration Pyright, where AFAICT they allow you to specify multiple roots which are effectively your package bases.

@charliermarsh
Copy link
Member Author

@hauntsaninja - no pressure to respond, but I'm curious if you'd have any advice here based on your experiences in Mypy. This issue is roughly referring to the behavior Mypy has in the SourceFinder. Right now, we also try to map every Python file to a (package root directory, dot-separated module path), but our logic relies on namespace packages being explicitly specified in our configuration file.

@charliermarsh charliermarsh added the core Related to core functionality label Jul 27, 2023
@charliermarsh
Copy link
Member Author

It's also plausible that our current approach of requiring namespace packages to be explicitly enumerated is not that bad and fine to preserve for now.

@hauntsaninja
Copy link
Contributor

mypy's default behaviour isn't very good for actual namespace packages... It's good for "accidental" namespace packages, where you have a large tree and people forget to litter __init__.py everywhere.

I'd probably recommend using something like mypy's behaviour with --explicit-package-bases as described in https://mypy.readthedocs.io/en/stable/running_mypy.html#mapping-file-paths-to-modules . mypy doesn't use --explicit-package-bases by default because it would be too breaking. Having this be cwd dependent is something that's worked reasonably well for mypy, but you may wish to think more carefully about it.

Seems potentially a little annoying to list all namespace packages out, seems preferable to list bases.

@charliermarsh
Copy link
Member Author

Thank you, that's helpful.

@hauntsaninja
Copy link
Contributor

hauntsaninja commented Jul 27, 2023

An argument against using literally mypy's --explicit-package-bases behaviour is that ruff wants to support monorepos with potentially a bunch of nesting, which would cause listing all bases to be tedious. Most users of mypy I've seen are not trying to run mypy in a single invocation on a varied tree. You may wish to explore something that uses the location of pyproject.toml's instead and lets users mix and match settings a little more.

Honestly, maybe mypy should probably just use sys.path of the target interpreter to determine bases. This is very defensible behaviour and unlike ruff mypy does usually have a target interpreter that has the installs needed.

@charliermarsh
Copy link
Member Author

👍 Yeah monorepo support is important for Ruff. We do currently treat each pyproject.toml as creating a new "source root" -- it's a bit complicated but I wrote it out in detail here (not expecting you to read it, only if curious): https://github.com/astral-sh/ruff/blob/main/CONTRIBUTING.md#import-categorization.

@hmc-cs-mdrissi
Copy link

pyright while not python does still call python for sys.path. Beyond helping with namespace packages it also deals helps a lot of import weirdness like handling .pth files/ways sys.path can be modified.

Pylint's namespace package detection is buggy and I haven't seen an approach that really works besides explicitness/sys.path. Here's a test repo that's small but is already enough to lead to issues for number of tools due to there being files with same name that are distinguished if you detect namespace packages correctly.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
core Related to core functionality
Projects
None yet
Development

No branches or pull requests

3 participants