Support implicit/native namespace packages (PEP 420)

[bamarsha]

I have set of projects that share the same Python namespace package (e.g., foo.project1.module1 and foo.project2.module2 are both under the foo namespace). My directory structure looks like this:

project1/
  foo/
    project1/
      __init__.py
      module1.py
  src/...
  Cargo.toml
  pyproject.toml

project2/
  foo/
    project2/
      __init__.py
      module2.py
  src/...
  Cargo.toml
  pyproject.toml

foo is a native namespace package (PEP 420), so it should not have an __init__.py. I set the Maturin module name to match:

[package.metadata.maturin]
name = "foo.project1._native"

However, maturin complains that there is no __init__.py:

Found a directory with the module name (foo) next to Cargo.toml, which indicates a mixed python/rust project, but the directory didn't contain an __init__.py file.

As a workaround, I have to use the pre-Python 3.3 kind of namespace packages by adding this __init__.py to foo:

__path__ = __import__("pkgutil").extend_path(__path__, __name__)

But I would prefer to use the newer kind of namespace packages. This is related to #792, but I created a new issue since it feels different - I don't have a pure stub package, but instead my __init__.py is in a nested folder. (Feel free to close as a duplicate if I'm mistaken though.)

[konstin]

While I'm aware that namespace packages are a python standard, they are also a massive footgun and break some assumption around installation and removal, so I'd rather not support them in maturin (and rather lint against them). I'd recommend using packages foo-project1-module1 and foo-project2-module2 and if desired a foo wrapper package which installs them through extras or reexports them

[bamarsha]

While I'm aware that namespace packages are a python standard, they are also a massive footgun and break some assumption around installation and removal, ...

Could you elaborate on this? We can reconsider using namespace packages, but we weren't aware of any major issues (other than needing to make sure all projects with the namespace use the same type of namespace package).

[konstin]

Normally, a package foo has a folder .venv/lib/python3.8/site-packages/foo and .venv/lib/python3.8/site-packages/foo-1.2.3.dist-info. Only foo will modify the foo folder, and you can (ignoring data and entrypoints) remove foo with rm -r .venv/lib/python3.8/site-packages/{foo,foo-1.2.3.dist-info}. If you have any backtrace with .venv/lib/python3.8/site-packages/foo you know the package is responsible is foo.

Namespace packages break those assumption: Suddenly installing a new package can modify an already existing package. To figure out the responsible package of a line of code you have to check the list of installed package instead of just assuming it's the top-level. Also I'm not sure if there's any mechanism to ensure that the different foo.* versions are compatible with each other. For users, you'll create situations where they are "I have installed foo-something and I can import foo, so why doesn't foo.otherthing work? imo the slightly nicer api is not worth this confusion

[bamarsha]

Thanks for the explanation. I agree that it adds some complexity to the package management.

For my project, we are leaning towards continuing to use namespace packages despite the disadvantages. Ideally, we would like the option to use namespace packages or not based on their own merits, rather than based on Maturin not supporting them. I hope that you could reconsider since they do have official support in Python and seem to be fairly popular. However, if Maturin doesn't add support, we'll probably continue to use namespace packages via a workaround.

[konstin]

In that case I recommend switching to setuptools-rust, which supports all the python standards and additionally gives you nearly arbitrary configuraribility. It'll likely match your needs much better

[ileixe]

Hi,

Although @konstin explained his opinion about the namespace package, but I'd like to ask for any chance to revisit the issue. I think the main confusion about the assumption described is already there regardless of maturin and should be known for those who are using namespace package. As it has been commonly used for many organization and definitely helps for ourself, if there's no critical conflict that maturin itself have, I'd like to contribute the feature if possible. (maturin has more features than setuptools-rust we'd love to, so it may not an option for us to replace it)

Can you share your thought?

[krpatter-intc]

Just do add to this: namespace packages are supported due to how we can do the __init__.py, but it would extremely helpful to fully support the PEP420 by allowing the __init__.py to not be present.

We currently could use this because many of our python packages get re-packaged in to linux packaging, They have issues if files like the __init__.py overlap with other python packages.