Use importlib.metadata

[jaraco]

Working on #339, I discovered this use of the deprecated pkg_resources module. This change adds a dependency on importlib_metadata for python 3.7 and earlier and uses the stdlib version on Python 3.8 and later.

I haven't tested this behavior, so I could use some help validating that it does what's expected.

[jaraco]

Between #348 and the fact that the metadata inspector swallows all exceptions, I'm struggling to determine where the error is in the code.

[jaraco]

Patching the venv inspector and manually invoking it, I see the error:

pipx feature/importlib-metadata $ .nox/tests-3-8/bin/python                                                                                                               
Python 3.8.1 (v3.8.1:1b293b6006, Dec 18 2019, 14:08:53) 
[Clang 6.0 (clang-600.0.57)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import pathlib
>>> import pipx.venv
>>> env.get_venv_metadata_for_package('tox')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NameError: name 'env' is not defined
>>> env = pipx.venv.Venv(pathlib.Path('~/.local/pipx/venvs/tox').expanduser())
>>> env.get_venv_metadata_for_package('tox')
Traceback (most recent call last):
  File "<string>", line 157, in <module>
  File "<string>", line 118, in main
  File "<string>", line 54, in get_apps
AttributeError: 'PackagePath' object has no attribute 'samefile'
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/jaraco/code/public/pipx/src/pipx/venv.py", line 243, in get_venv_metadata_for_package
    data = json.loads(
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/json/__init__.py", line 357, in loads
    return _default_decoder.decode(s)
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

[jaraco]

I've fixed two issues, but another still remains - when attempting to resolve the dependency of tox, one dependency is importlib-metadata, but importlib.metadata doesn't recognize any distribution by that name, because the dist name is importlib_metadata.

[jaraco]

Correction - the issue isn't the naming convention. distribution('importlib-metadata') works if importlib_metadata is installed. The problem is that importlib_metadata isn't installed because it's not selected on Python 3.8.

[jaraco]

I'm stumped. I don't know what's causing the changes to fail. The venv inspector script runs on my local machine. The fact that it's apparently emitting nothing suggests it's failing to run to if __name__..., but I can't see how.

[layday]

It appears you need to install packaging in the virtual environment of the metadata inspector:

----------------------------- Captured stderr call -----------------------------
Traceback (most recent call last):
  File "<string>", line 13, in <module>
ModuleNotFoundError: No module named 'packaging'

I believe this reuses the pipx 'shared' virtual environment which - incidentally - is not isolated from the user installation during testing. My /tmp/pytest-of-layday/pytest-1/test_simple_run0/subdir/pipxhome/.cache/b63af28d1e99e25/lib/python3.8/site-packages/pipx_shared.pth contains: /Users/layday/.local/pipx/shared/lib/python3.7/site-packages.

[layday]

The tests on Travis will pass making the following change:

diff --git a/src/pipx/shared_libs.py b/src/pipx/shared_libs.py
index 9a375c2..b2efcbd 100644
--- a/src/pipx/shared_libs.py
+++ b/src/pipx/shared_libs.py
@@ -79,6 +79,7 @@ class _SharedLibs:
                         "pip",
                         "setuptools",
                         "wheel",
+                        "packaging",
                     ]
                 )
                 self.has_been_updated_this_run = True

The shared library would also have to be updated when upgrading pipx to install packaging if this PR is merged.

[layday]

Whoops, for 3.7 and lower you will also need to add importlib-metadata.

[itsayellow]

I still don't understand why we need to eliminate pkg_resources. Can somebody explain that? I've googled a lot and can't find anything saying it or setuptools is deprecated.

Also, the use of pkg_resource in venv_metadata_inspector.py does NOT cause a dependency on setuptools by pipx. It is only used with the venvs, and we install setuptools in a separate shared venv ourselves.

I'm not in favor of this PR without a really good reason. venv_metadata_inspector.py can easily be a source of bugs in pipx (and has been many times in the past.) It's a critical part of pipx that I'm frankly just happy that it seems to be working.

[jaraco]

I'm the maintainer of setuptools and author of importlib.metadata. In coordination with Barry Warsaw, we created importlib.metadata to compliment importlib.resources in order to replace pkg_resources. I can say with authority that pkg_resources is deprecated. It has intractable bugs and undesirable behaviors that were easier to sidestep than fix. I haven't yet gotten around to documenting the deprecation, and it has no deadline, but the intention is to move use-cases like this one to those tools. The welcome paragraph alludes to the replacement, implying deprecation.

I submit this change here with the expectation that it would be a preferable change. And at first, this change does simplify the code, but then it became more complicated when adding support for environment markers (which pkg_resources has integrated). Probably there should be some code that draws these behaviors together with clean interfaces.

If the adoption of this change is objectionable for reasons other than YAGNI, I want to address those objections in packaging, importlib_metadata, etc. If the sole objection is YAGNI, I'd like to encourage you to work with me to make the shift as a demonstration of what moving off of pkg_resources (and possibly setuptools) looks like.

[itsayellow]

@jaraco, thanks for the explanation how it is deprecated. I would hear you and @gaborbernat referring to it, but I could not find anything publicly available that would say the same thing. Going forward it might be nice if there was a reference online that talked about deprecating pkg_resources so more people know about it (besides the maintainer of setuptools himself! 😄).

I agree there's a lot of goofiness in pkg_resources. My caution is not YAGNI as far as I understand it, I just didn't want us to rush into something that has a pervasive influence on how pipx operates and could cause bugs--when what we have works, and there was no hint anywhere online that pkg_resources is going away or deprecated.

It's nice now to see the rationale, from your above comment.

If @cs01 is fine with this I'm not going to object. But we should be careful that we cover existing users, have a scheme to force an update of the shared libs, and basically don't require a reinstall-all to work. We just had one of those changes to the code...

[cs01]

I also like treading carefully when modifying venv_metadata_inspector. I don't have time to carefully review these changes right now and absorb all the conversation though. I am not opposed to changes in it, but just want to make sure they are necessary, as minimal as possible, and are not breaking to existing users. I'll review again when I get the time. Sorry I can't do it immediately.

[itsayellow]

One of your failing tests seems to be installing a package in python3.5, which we currently support in venvs. Is importlib_metadata backported to python3.5?

[itsayellow]

One question I have about importlib.metadata: will it work for local, editable packages (i.e. packages installed with pipx -e/pipx -e from a local directory)? I'm just looking at the quote from the importlib.metadata docs:

By “installed package” we generally mean a third-party package installed into Python’s site-packages directory via tools such as pip.

Historically, editable packages from a local filepath have been the edge cases for pipx's install system, and have been the source of errors when other installs work. I don't think we currently have a local editable install in our tests.

[uranusjr]

@itsayellow It does. An editable install puts an .egg-link in site-packages, so importlib-metadata is still able to locate the correct information from the distribution.

[jaraco]

Is importlib_metadata backported to python3.5?

Yes.

will it work for local, editable packages (i.e. packages installed with pipx -e/pipx -e from a local directory)?

Yes. Out of the box, importlib_metadata supports packages on sys.path with egg-info or dist-info metadata. Currently, pip install -e invokes python setup.py develop and adds a .pth file to put the metadata and creates egg-info metadata at that path. I use it extensively for dozens of projects that rely on importlib_metadata to resolve versions during tox-orchestrated editable installs.

[jaraco]

Even after getting tests passing on master, I'm struggling to replicate the test failures for this branch as observed on Travis in my local environment. Instead, the tests fail early in test_inject with an error about ModuleNotFoundError: No module named 'packaging'. Why is the travis environment apparently getting this package, but I'm not (on the same commit)? Are the tests perhaps sensitive to the presence of pipx being installed in ~?

[jaraco]

Indeed, it seems the monkeypatch in the pipx_temp_env fixture is insufficient to isolate the test runs from the user's environment because the pipx.shared_libs.PIPX_SHARED_LIBS and pipx.shared_libs.shared_libs.root are already computed by the time the monkeypatch is run, so monkeypatching the constant has no effect. Adding this patch provides better isolation:

diff --git a/tests/conftest.py b/tests/conftest.py
index 4542215..c486836 100644
--- a/tests/conftest.py
+++ b/tests/conftest.py
@@ -2,7 +2,7 @@ from pathlib import Path
 
 import pytest  # type: ignore
 
-from pipx import constants
+from pipx import constants, shared_libs
 
 
 @pytest.fixture
@@ -17,6 +17,8 @@ def pipx_temp_env(tmp_path, monkeypatch):
     bin_dir = Path(tmp_path) / "otherdir" / "pipxbindir"
 
     monkeypatch.setattr(constants, "PIPX_SHARED_LIBS", shared_dir)
+    shared_libs.PIPX_SHARED_LIBS = shared_dir
+    shared_libs.shared_libs.__init__()
     monkeypatch.setattr(constants, "PIPX_HOME", home_dir)
     monkeypatch.setattr(constants, "LOCAL_BIN_DIR", bin_dir)
     monkeypatch.setattr(constants, "PIPX_LOCAL_VENVS", home_dir / "venvs")

[layday]

Even after getting tests passing on master, I'm struggling to replicate the test failures for this branch as observed on Travis in my local environment. Instead, the tests fail early in test_inject with an error about ModuleNotFoundError: No module named 'packaging'. Why is the travis environment apparently getting this package, but I'm not (on the same commit)? Are the tests perhaps sensitive to the presence of pipx being installed in ~?

Yes, like I said above:

I believe this reuses the pipx 'shared' virtual environment which - incidentally - is not isolated from the user installation during testing. My /tmp/pytest-of-layday/pytest-1/test_simple_run0/subdir/pipxhome/.cache/b63af28d1e99e25/lib/python3.8/site-packages/pipx_shared.pth contains: /Users/layday/.local/pipx/shared/lib/python3.7/site-packages.

[jaraco]

One of your failing tests seems to be installing a package in python3.5, which we currently support in venvs.

I'm seeing this issue too. I think the problem is that 'importlib_metadata' is being installed in 'shared_libs', but for Python 3.8 (e.g. $shared_libs/lib/python3.8/site-packages/importlib_metadata), which isn't relevant for Python 3.5.

I don't understand how 'shared_libs' as a single venv makes sense in a world of multiple Python versions. My guess is that the only reason the tests aren't failing now is because the existing dependencies (setuptools) are installed implicitly in the main, non-shared environment.

[itsayellow]

Actually I always wondered how the same shared libs dir was servicing possibly multiple versions of python too... Do we need a separate shared libs dir for each python major.minor?

[layday]

My guess is that the only reason the tests aren't failing now is because the existing dependencies (setuptools) are installed implicitly in the main, non-shared environment.

The tests aren't failing because pipx creates a pth file pointing to the original shared library irrespective of Python version. This is why I have pipx running under Python 3.8 with a pth to a shared library created under 3.7 (see above). It matters naught to the running Python - which is explicitly not the shared venv's Python - if the venv was created and setuptools was installed under 3.5 or .8.

[jaraco]

All of the tests are passing except for the Windows tests. I ran the tests in my Windows environment and they were passing on Python 3.8, but failing on Python 3.7 (perhaps similar to the Travis failures; I'm still awaiting the output).

[jaraco]

I've spent too many hours of my life on this. I'm giving up.

[layday]

This is a shot in the dark but it looks like Windows might be truncating the path to 180 characters which so happens to result in:

C:\Users\travis\AppData\Local\Temp\pip-uninstall-q5cuv0kx\users\travis\appdata\local\temp\pytest-of-travis\pytest-0\test_something_that_pushes_it_to_180\shareddir\lib\site-packages\pip

... which also might be why it can't find a __main__.py to load.

[layday]

The default MAX_PATH value is 260 apparently and C:\Users\travis\AppData\Local\Temp\pip-uninstall-q5cuv0kx\users\travis\appdata\local\temp\pytest-of-travis\pytest-0\test_install_same_package_twic0\shareddir\lib\site-packages\pip\_vendor\urllib3\packages\ssl_match_hostname\__pycache__\_implementation.cpython-37.pyc measures at 266 characters. The error message is definitely unhelpful.

[cs01]

Actually I always wondered how the same shared libs dir was servicing possibly multiple versions of python too... Do we need a separate shared libs dir for each python major.minor?

I asked this when @pfmoore added it. It was designed assuming the shared libs work across all python versions.

[pfmoore]

It was designed assuming the shared libs work across all python versions.

(Or at least, all versions that pipx supports)

It's worth noting that setuptools has now dropped support for Python < 3.5. As pipx is Python >= 3.6 only, this isn't a problem yet, but it bears watching.

When I designed the feature, I was assuming that the various core packaging tools would be more cautious than average over dropping older Python versions. While that's true for pip so far, it hasn't turned out that way for the other tools.