Add support for namespace-packages without __init__.py (PEP420)

[jstriebel]

In Python 3 modules don't require an __init__.py anymore, they are referred to as namespace packages then (see PEP 420 for details). It seems that those are not supported by import-linter at the moment, I'm getting Could not find package.initless.module when scanning …. This may be due to a missing __init__.py file in the parent package warnings and Module 'package.initless' does not exist. errors. In my case the structure is roughly:

package
  __init__.py
  initless
    module.py

It would be great if this could be supported.

[jstriebel]

Just found the relevant issue in the project that is used for the module-graph generation: seddonym/grimp#80

[seddonym]

Thanks Jonathan - as mentioned on the issue you've linked to, I see this as low priority as it's quite difficult to fix. Would welcome insight as to what you find useful about namespace packages - to me it feels like an edge case that won't bring much benefit to most users, but happy to be proved wrong!

[jstriebel]

Simply put, there's not much value in empty __init__.py files. If you're learning python now, you would only use them if you need the specific level in the module-hierarchy. Since namespace-modules are supported at runtime and with many important analysis tools (pylint, mypy, black, …), not supporting it in import-linter means rather not to use import-linter for us right now instead of adding empty files. There is quite a lot of discussion about this topic, e.g. in this mypy issue about it: python/mypy#1645
I've just looked briefly how the imports are aggregated. Probably it might be enough to add empty modules which only import their parent module (as always done implicitly), if there is no __init__.py next to other python files. Unfortunately I won't have time to dig deeper into it.

[seddonym]

there's not much value in empty init.py files.

Interesting - this surprises me! But you may have a point.

My understanding of namespace packages is that they are focused specifically on "splitting a single Python package across multiple directories on disk" (from the PEP). I'm not aware of an intention to drop them except for this use case.

I think it's possible that they're not really intended to be optional in regular Python packages. It's possible, for example, that not including them might have performance implications - but I don't know enough about it.

I wasn't able to see anything that addressed that in the Mypy link you directed me too, but do direct me to any other discussion on the subject if you come across it.

[jstriebel]

My understanding of namespace packages is that they are focused specifically on "splitting a single Python package across multiple directories on disk" (from the PEP). I'm not aware of an intention to drop them except for this use case.

Exactly, that's the original motivation. However, it means that also for a single package in a single directory, one can simply omit the __init__.py. For example, I use an __init__.py in the root of a package usually, since it has content, as some classes and functions should be available on the package level. For sub-modules, I tend to omit them, since I use the full import paths anyways. So the structure for a package would be as described above:

package
  __init__.py
  initless
    module.py

It does not matter, that this was not the intended usage of the PEP, but since empty __init__.py file can simply be omitted in most of the cases, I don't care to add them anymore.

A comment from the linked thread that resonates with this (python/mypy#1645 (comment)):

Just a philosophical note here that might influence the way mypy handles this: in my experience in a Py3-only world, packages without init.py are not a special edge case for namespace packages, they are the default case. It's pretty unreasonable to expect a new Py3 developer to remember to drop empty init.py turds everywhere when their code demonstrably runs perfectly fine without them. It's hard to even make a sensible case in code review for why they should add init.py (other than "it's the way we've always done things in the old world, get off my lawn" or "the typechecker is buggy and won't find your code without it.") The presence of init.py becomes the special case, used only when you need some package-level code or imports.

This reversal may not have been an intended consequence of PEP 420, but AFAICS it's an inevitable one.

From a similar thread (python/mypy#8584 (comment)):

PEP420 support by default would be quite nice: it's unintuitive that something that works perfectly fine in pylint and setuptools and at runtime doesn't seem to work in mypy.
[…]
At some point, I wonder if it's worth attempting to ratify a "good tool behavior" PEP that explains how various third party utilities should attempt to understand and process file structures, in a bid to try and standardize disparate behavior between popular tools.

[jstriebel]

PS:

I think it's possible that they're not really intended to be optional in regular Python packages. It's possible, for example, that not including them might have performance implications - but I don't know enough about it.

Same for me! Performance implications on import speed aren't what I care about usually. But this consideration shouldn't prevent one from using static analysis tools, such as import-linter.

[seddonym]

Thanks for your input, it's challenging my thinking! I'll have another look at how hard this would be to do in import-linter. And of course always open to pull requests!

[antoniivanov]

We also use namespace packages. We have separate distribution packages using common root package.

Another benefit:
Namespace packages make it easier to develop when you have multiple modules with the asme common prefix e.g
I have a project "cool" - with package "companyname.cool" and project "hot" that depends on cool.

While developing cold we would use pip install -e . to install it in editable form and thus companyname.hot would be from local folder while companyname.cool would be from site-packages of python. Without namespace packages only one would be found.

import-linter looked really promising for us but we need namespace packages so we cannot use it.

[Incognito]

For anyone in the use-case "I just don't use the file in py3" and not in the use-case "I specifically need to namespace things differently", I run this command before linting to pre-populate anything missing (but do not commit it). A bit of a hack but it solves 50% of the use-cases I'm aware of.

find . -type d -exec touch {}/__init__.py \

If you have some time to fork a bit, I found you can also just directly add the implied ancestors to the grimp use-case (for example, in app.foo.bar, you can split on . to add app and app.foo to the graph). You'd want to adjust that here https://github.com/seddonym/grimp/blob/master/src/grimp/application/usecases.py#L73 .

If you really really want to do fancy namespace things I don't have advice for your use-case. I assume 90% of python devs just want to make a folder without an empty init file.

[seddonym]

I'm pleased to say there is now a PR in place which offers support for namespace packages: #127

@tozka This should now meet your needs. @jstriebel, I have a feeling this won't exactly fulfil your needs I'm afraid but you might want to take a look just in case.

Comments and testing welcome.

[seddonym]

I'm going to close this ticket as Import Linter now supports namespace packages.