Editable installs do not satisfy dependencies when using pip>=20.2 and Python 2.7

[alanbriolat]

Environment

• pip version: 20.2.1
• Python version: 2.7.17
• OS: Ubuntu 18.04 64-bit

Description

Regression in dependency resolution (not the new resolver) where already installed "editable" packages are no longer taken into account when satisfying dependencies. Only happens with the combination of pip>=20.2 and Python 2.7; older pip or using Python 3.x doesn't give the same issue. Happens with both path and VCS editable installs.

Expected behavior

Already installed editable packages should satisfy dependencies when installing another package, consistent with behaviour when using older pip versions or newer Python versions.

How to Reproduce

1. Create two basic packages: package-a which has no dependencies, and package-b which depends only on package-a
2. Create and activate a Python 2.7 virtualenv
3. Perform an editable install of package-a, see a successful installation
4. Perform an editable install of package-b, see Could not find a version that satisfies the requirement package-a
5. Observe that pip freeze definitely contains package-a
6. Observe that pkg_resources.get_distribution("package-a") gives a valid result
7. Downgrade pip to an earlier version, repeat editable install of package-b, see a successful installation

Output

$ mkdir -p package_a package_b

$ cat > package_a/setup.py <<EOF
from setuptools import setup, find_packages

setup(
    name="package-a",
    version="0.1",
    packages=find_packages(),
)
EOF

$ cat > package_b/setup.py <<EOF
from setuptools import setup, find_packages

setup(
    name="package-b",
    version="0.1",
    packages=find_packages(),
    install_requires=["package-a"],
)
EOF

$ virtualenv -p python2.7 venv
Running virtualenv with interpreter /usr/bin/python2.7
Already using interpreter /usr/bin/python2.7
New python executable in /home/alanb/tmp/venv/bin/python2.7
Also creating executable in /home/alanb/tmp/venv/bin/python
Installing setuptools, pip, wheel...
done.

$ source venv/bin/activate

$ pip --version
pip 20.2.1 from /home/alanb/tmp/venv/local/lib/python2.7/site-packages/pip (python 2.7)

$ pip install -e package_a/
DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. pip 21.0 will drop support for Python 2.7 in January 2021. More details about Python 2 support in pip can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support
Obtaining file:///home/alanb/tmp/package_a
Installing collected packages: package-a
  Running setup.py develop for package-a
Successfully installed package-a

$ pip install -e package_b/
DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. pip 21.0 will drop support for Python 2.7 in January 2021. More details about Python 2 support in pip can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support
Obtaining file:///home/alanb/tmp/package_b
ERROR: Could not find a version that satisfies the requirement package-a (from package-b==0.1) (from versions: none)
ERROR: No matching distribution found for package-a (from package-b==0.1)

$ pip freeze
DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. pip 21.0 will drop support for Python 2.7 in January 2021. More details about Python 2 support in pip can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support
# Editable install with no version control (package-a==0.1)
-e /home/alanb/tmp/package_a

$ python -c 'import pkg_resources; print(pkg_resources.get_distribution("package-a"))'
package-a 0.1

$ pip install 'pip<20.2'
DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. pip 21.0 will drop support for Python 2.7 in January 2021. More details about Python 2 support in pip can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support
Collecting pip<20.2
  Using cached pip-20.1.1-py2.py3-none-any.whl (1.5 MB)
Installing collected packages: pip
  Attempting uninstall: pip
    Found existing installation: pip 20.2.1
    Uninstalling pip-20.2.1:
      Successfully uninstalled pip-20.2.1
Successfully installed pip-20.1.1

$ pip install -e package_b/
DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. pip 21.0 will drop support for Python 2.7 in January 2021. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support
Obtaining file:///home/alanb/tmp/package_b
Requirement already satisfied: package-a in ./package_a (from package-b==0.1) (0.1)
Installing collected packages: package-b
  Running setup.py develop for package-b
Successfully installed package-b
WARNING: You are using pip version 20.1.1; however, version 20.2.1 is available.
You should consider upgrading via the '/home/alanb/tmp/venv/bin/python2.7 -m pip install --upgrade pip' command.

[sbidoul]

Thank you for the detailed report. I can reproduce and confirm the regression is for python 2 only.

[sbidoul]

Hm, I can reproduce on Ubuntu 20, but not on CentOS 8.

I think we are hitting #5193. Indeed dist_is_local(pkg_resources.get_distribution("package-a")) returns False in a Debian virtualenv.

Fortunately #8702 should work around that. @alanbriolat could you test with that branch, which you can install with pip install -U "pip @ git+https://github.com/pypa/pip@refs/pull/8702/head".

[alanbriolat]

@sbidoul Thanks, #8702 seems to fix the issue and give the expected behaviour:

$ pip install -U "pip @ git+https://github.com/pypa/pip@refs/pull/8702/head"
[...]
Successfully installed pip-20.3.dev0

$ pip install  -e package_a/
[...]
Obtaining file:///home/alanb/tmp/pip-bug/package_a
Installing collected packages: package-a
  Running setup.py develop for package-a
Successfully installed package-a

$ pip install  -e package_b/
[...]
Obtaining file:///home/alanb/tmp/pip-bug/package_b
Requirement already satisfied: package-a in ./package_a (from package-b==0.1) (0.1)
Installing collected packages: package-b
  Running setup.py develop for package-b
Successfully installed package-b

$ pip freeze
[...]
# Editable install with no version control (package-a==0.1)
-e /home/alanb/tmp/pip-bug/package_a
# Editable install with no version control (package-b==0.1)
-e /home/alanb/tmp/pip-bug/package_b

[uranusjr]

Why does #8702 resolve the issue, if the root cause is in sysconfig? Is it because the change means we’d switch back to use distutils.sysconfig?

[sbidoul]

@uranusjr you are right, I was confusing dist_is_local with dist_in_site_packages. So #8702 does address the root cause of this issue and this has nothing to do with #5193.

[uranusjr]

Thanks for the investigation. I was worried that if we switched back to distutils.sysconfig it may just opened the possibility of Debian breaking sysconfig even for Python 3 in the future. It’s relieving to know that won’t be the case.

[sbidoul]

@uranusjr still, the fact that this bug seems to apply to debian derivatives only is disturbing. So after a good night sleep I reconfirm that dist_is_local(pkg_resources.get_distribution("package-a")) does return False in a Debian virtualenv and not on centos. And that is because egg_link_path() works differently due to the same cause as #5193 (i.e. the site_package variable pointing to the wrong place).

#8702 fixes it by not calling dist_is_local, but nevertheless I still think the patch I offered in #6918 would leave the last pip-that-supports-python-2 in a better state.

[uranusjr]

I think you’re right. It would be a good split at this point with importlib.metadata. Python 2 can continue to use pkg_resources and distutils.sysconfig, while newer Pythons get importlob.metadata plus sysconfig.

[pradyunsg]

Removing from 20.2.2 since #8702 does contain the required fix/workaround but IDK if anyone wants a more proper fix here.

It's a Py2-only bug so I personally don't care. :)

[sbidoul]

I think we can close this issue as the root cause is tracked in #5193.