Rely on importlib2 and PEP 420 for namespace packages

[ghost]

Originally reported by: jaraco (Bitbucket: jaraco, GitHub: jaraco)

---

In this comment, @ncoghlan indicates that it may be possible to get PEP-420 namespace packages on Python 2.7. @rbtcollins indicates that he's worked on this as well with some issues.

I'd like to see setuptools support (and prefer) the PEP 420 model for namespaces. I don't yet have an idea quite what the solution looks like, so I'm opening this discussion here.

Is it possible for setuptools to detect whether the importlib2 hook has been installed on older Pythons? Would that be a sufficient condition in practice to infer that packages could be installed with that model (and eliminate the .pth and other init.py techniques)?

As a solution for this issue would likely address issues with namespace packages between different installation models, I'm marking it as higher priority.

---

• Bitbucket: https://bitbucket.org/pypa/setuptools/issue/406

[ghost]

Original comment by embray (Bitbucket: embray, GitHub: embray):

---

Another question I have is if it would be possible to somehow sneak this in to setup.py invocation. If nothing else it would be good practice to just manually add import importlib2.hook; importlib2.hook.install() (perhaps within a try/except) at the top of a setup.py that needs it. Or is it possible to do this later on, after setuptools has already been imported?

[ghost]

Original comment by jaraco (Bitbucket: jaraco, GitHub: jaraco):

---

It would be insufficient to hook in setup.py because the hook is needed not only at setup/install time but also at runtime.

[ghost]

Original comment by embray (Bitbucket: embray, GitHub: embray):

---

Ah, of course. So that raises my other question, which is if it's okay to install the hook later. I don't see why not--it just adds a sys.meta_path hook, right? So as long as it's added before you import a package that depends on it that's fine. Maybe some kind of .pth hack could take care of that, if there are any packages on the path that need it.

[ghost]

Original comment by ncoghlan (Bitbucket: ncoghlan, GitHub: ncoghlan):

---

I'd strongly prefer we don't encourage anyone to install the Python 3 import system implicitly at startup or as a side effect of import, as it does change the import semantics in various important ways: you get all the Python 3 import semantics, including pycache directories, listdir caching, native namespace packages. It's OK for particular applications to opt in to it, and for setuptools to recommend that folks needing certain behaviours enable it, but enabling it by default would run into problems due to other utilities that still assume the convention Python 2 import system (e.g. as far as I am aware, installing the importlib2 hooks doesn't alter the behaviour of compileall).

Detecting it at runtime would be a matter of looking at sys.meta_path to see if any of the importlib2 importers were installed.

Also tagging @ericsnowcurrently for any insight he can provide on potential challenges folks face in using importlib2 to enable Python 3 import semantics in 2.7.

[ghost]

Original comment by embray (Bitbucket: embray, GitHub: embray):

---

If nothing else it would be cool if importlib2 were better advertised and blessed as something you should care about, so that the occasional code that does muck with such things can be aware of it and check for it. For example, we still have some legacy code that messes around with __builtins__.__import__ (which I've tried to kill a couple times to avail). But it would be good if it at least knew that importlib2 being installed is a possibility.

[ghost]

Original comment by embray (Bitbucket: embray, GitHub: embray):

---

As for detecting importlib2 installation, looking at the source it seems like some variation of:
if 'importlib2.hook' in sys.modules and sys.modules['importlib2.hook'].__installed__ should do it.

[ghost]

Original comment by ericsnowcurrently (Bitbucket: ericsnowcurrently, GitHub: ericsnowcurrently):

---

(This is, essentially, a status report on importlib2.)

FWIW, the origin and current status behind importlib2 are a typical open-source story: I had an itch to scratch (pymigrate) and no longer have the itch (thanks to @brettcannon and pylint), so I haven't spent a lot of time on it.

That said, importlib2 doesn't need a lot of attention once the port is complete, which it is; I did make sure importlib2 worked before life intervened. At this point the only persistent attention it needs is updating every time importlib is updated in Python3, which is not very often. Most of the time I spent on importlib2 was automating the porting process, so keeping importlib2 up-to-date should be relatively trivial if I did the job well. :)

Not long after I finished work on importlib2, a couple different people expressed interest in making use of importlib2. However, in each case they eventually decided it would be better to work on adopting Python 3 instead. Needless to say, I was happy with their conclusion though I was quite ready and willing to help them with any problems they encountered with importlib2. In fact, one consequence of their interest was several bugs they helped identify and we were able to resolve (or at least triage).

Since then I haven't really spent any time on importlib2. If it were to get broader use (and exposure) then I'd likely steal away some time to work on it. I will mention that I've also received some offers of help in maintaining it, so that is something to consider.

As to where things stand right now, here are the outstanding tasks that need to be sorted out before I would feel comfortable broadly advertising importlib2:

• several outstanding bugs need to be addressed
• importlib2 is out-of-date with Python 3's importlib (most notably PEP 489)

PEP 489 ("Multi-phase extension module initialization") demonstrates in important point. While importlib2 provides a port of the import machinery from Python 3, it is strictly limited to what is in the importlib package. That means code in the imp module or pkgutil or zipimport or runpy or compileall (as Nick mentioned) is not ported. Likewise most of the changes introduced by PEP 489 won't make their way into importlib2 because they were made to C code (which is exposed via the _imp module). importlib2 does provide just about all the import machinery that matters, but I hope noone operates under the illusion that it ports everything or that it works exactly the same.

Finally, I also have reservations about making a big deal out of importlib2. I don't want it to slow down efforts to move to Python 3. importlib2 could be considered a tool to help with such efforts, but I'm not convinced it will be more help than hinderance.

[ghost]

Original comment by ericsnowcurrently (Bitbucket: ericsnowcurrently, GitHub: ericsnowcurrently):

---

As to challenges, I'm not exactly sure. The main trickiness I can think of is the gap between when the interpreter starts up and when importlib2.hook.install() is called. importlib2 does make an effort to sort that out but it isn't perfect.

As to detecting if importlib2 is installed, importlib2.hook.__installed__ indicates that, as Erik indicated.

[ghost]

Original comment by embray (Bitbucket: embray, GitHub: embray):

---

Thanks for the info, and I can understand your hesitancy about promoting importlib2 too heavily.

I'm doing what I can to get my user community to move on to Python 3, but it's a slow process, especially thanks to institutional inertia :)

[ghost]

Original comment by ncoghlan (Bitbucket: ncoghlan, GitHub: ncoghlan):

---

There are three specific cases where I think importlib2 is potentially worth promoting:

• improvements to native namespace package support in the PyPA tooling (hence this issue). For that, we do need to have a Python 2.7 story, and I think it makes more sense to promote importlib2 for that task than introducing further Python 2 specific hacks into the PyPA ecosystem.
• import performance in a HPC context. On local disks, the Python 3 redesign to reduce the number of stat calls was mainly aimed at regaining some of the speed lost through the move from a primarily C implementation to a Python one. However, when importing modules from a shared filesystem accessed over NFS, the significant reduction in the number of network round trips means that the Python 3/importlib2 approach absolutely destroys the legacy Python 2 approach.
• sharing packages between multiple interpreters (including between CPython 2 and CPython 3). The pycache design introduced in Python 3 lets the bytecode caching for different interpreters (including different versions of the same interpreter) avoid treading on each others' toes. This is becoming more important as the barriers to adoption for the various major interpreter implementations start getting lower. The change in Python 3.5 to also cover optimisation levels means that using "-OO" to strip docstrings (and hence reduce memory consumption) for non-interactive use cases becomes a more viable option.

Python 2.7 support is going to be with us upstream for another 5 years, and commercial redistributor support will extend well beyond that, so I consider it potentially worthwhile to bring these benefits into the "common Python 2/3 subset", rather than viewing them as "Python 3 only" enhancements. As Eric notes, there will still be improvements that necessarily fall into the latter category (like multi-phase initialisation for extension modules), but we can still backport a lot of nice meta_path based improvements.

As far as long term maintainability goes, it might be worth talking to the http://python-future.org/ folks about bringing importlib2 under their umbrella as an independently installable project, as I think it fits squarely into their "Make programming in Python 2 as much like programming in Python 3 as we reasonably can" objectives.

[asmodehn]

In case someone ends up here, looking for PEP 420 support in python 2, and importing namespace packages, here is a gist that might help you to do just that : https://gist.github.com/asmodehn/d4d69374213be20002ec64ed7ab77dad.
A Finder and a Loader, extracted from importlib2.