With pip (>20.1.1), User package cannot install ruamel.yaml (silently ignored) in the Conda base environment that has a dependency on ruamel.yaml, but an error occurs while running the user package: ModuleNotFoundError: No module named 'ruamel'

[harneetvirk]

Environment

• pip version: 20.2.2
• Python version: 3.6.2, 3.7 (didn't checked on 3.8)
• OS: Linux
  conda 4.8.4

Description
This issue is encountered in the conda base environment only. Major impact is in on the Docker images where the packages are installed in the base environment. Conda base environment comes up with ruamel_yaml 0.15.87. Installing a package that has dependency on ruamel.yaml fails with an exception on the entry point of the setup.py: The 'ruamel.yaml>=0.15.35' distribution was not found and is required by the application. This dependency: 'ruamel.yaml>=0.15.35' is defined in the setup.py of the user package.

ruamel.yaml installation is IGNORED on conda base environment as ruamel_yaml is already installed from conda base environment. Dot, underscore, and dash are treated as equivalents when comparing package names during the installation of the dependencies but not during the entry point of package.

Expected behavior
The ruamel.yaml package should get installed in conda base environment when defined in setup.py of the user package.

How to Reproduce
Run the inline command in the conda base environment. You may not be able to install ruamel.yaml as ruamel_yaml comes up conda in the base environment. User package cannot install ruamel.yaml (silently ignored) in the Conda base environment that has a dependency on ruamel.yaml, but an error occurs while running the user package: ModuleNotFoundError: No module named 'ruamel'.

Commands to run:
Create conda base environment:
python --version;
pip --version;
pip install ruamel.yaml

Output

root@b9f5c2385871:/# python --version
Python 3.6.2 :: Anaconda, Inc.
root@b9f5c2385871:/# pip --version
pip 20.2.2 from /opt/miniconda/lib/python3.6/site-packages/pip (python 3.6)
root@b9f5c2385871:/# pip install ruamel.yaml
Requirement already satisfied: ruamel.yaml in /opt/miniconda/lib/python3.6/site-packages (0.15.87)
root@b9f5c2385871:/# pip list | grep -i ruamel
ruamel-yaml            0.15.87

[uranusjr]

ruamel.yaml is the same as ruamel_yaml, and pip 20.2 actually fixed a bug that is was not correctly detecting its identify. It is correct to not reinstall the package since it is already installed (by Conda). If this causes errors, whatever did not work would be the problem, not the fact that pip does not reinstall ruamel.yaml.

[harneetvirk]

yes, ruamel.yaml is the same as ruamel_yaml. With latest pip (20.2.2), when we try to install a package in conda base environment that has dependency on the ruamel.yaml, installation of the python package work without any issue, but we get an error on the entry point. The python cannot find the required list of packages in the environment as the installed package is ruamel_yaml, but setup.py is configured to look for ruamel.yaml.

Error message:
import ruamel.yaml
ModuleNotFoundError: No module named 'ruamel'

exit()
root@931f2fb9d7fb:/# conda list | grep -i ruamel
ruamel_yaml 0.15.87 py36h7b6447c_1

[uranusjr]

Can you provide steps to reproduce the error?

[harneetvirk]

Perform these operations in conda base environment, where ruamel_yaml is already installed.

Repro from a publicly available package:

1. Install nbreport
2. Try this test command to check if nbreport work:
   nbreport test https://github.com/lsst-sqre/nbreport --git-subdir tests/TESTR-000 -c title "My first report"

root@931f2fb9d7fb:/# pip install nbreport
Collecting nbreport
Downloading nbreport-0.7.4-py3-none-any.whl (25 kB)
Requirement already satisfied: ruamel.yaml<0.16.0,>=0.15.0 in /opt/miniconda/lib/python3.6/site-packages (from nbreport) (0.15.87)
Requirement already satisfied: requests>=2.0 in /opt/miniconda/lib/python3.6/site-packages (from nbreport) (2.24.0)
Collecting GitPython
Downloading GitPython-3.1.7-py3-none-any.whl (158 kB)
|████████████████████████████████| 158 kB 8.2 MB/s
...
...
root@931f2fb9d7fb:/# nbreport test https://github.com/lsst-sqre/nbreport --git-subdir tests/TESTR-000 -c title "My first report"
Traceback (most recent call last):
File "/opt/miniconda/bin/nbreport", line 5, in
from nbreport.cli.main import main
File "/opt/miniconda/lib/python3.6/site-packages/nbreport/cli/main.py", line 11, in
from ..userconfig import read_config, get_config_path, create_empty_config
File "/opt/miniconda/lib/python3.6/site-packages/nbreport/userconfig.py", line 10, in
import ruamel.yaml
ModuleNotFoundError: No module named 'ruamel'
root@931f2fb9d7fb:/#

root@931f2fb9d7fb:/# conda list | grep -i ruamel
ruamel_yaml 0.15.87 py36h7b6447c_1

[harneetvirk]

We can repro this issue in any conda environment by installing ruamel_yaml before installing the user package:

--- Create conda environment
conda create --name mytest36 python=3.6
--- Activate conda environment
conda activate mytest36
--- install ruamel_yaml as a prerequisite, to repro this failure.
conda install ruamel_yaml
--- install nbreport which tries to bring ruamel.yaml
pip install nbreport
--- run test
nbreport test https://github.com/lsst-sqre/nbreport --git-subdir tests/TESTR-000 -c title "My first report"

(mytest36) root@931f2fb9d7fb:/# conda list | grep -i ruamel
ruamel_yaml 0.15.87 py36h7b6447c_1
(mytest36) root@931f2fb9d7fb:/# nbreport test https://github.com/lsst-sqre/nbreport --git-subdir tests/TESTR-000 -c title "My first report"
Traceback (most recent call last):
File "/opt/miniconda/envs/mytest36/bin/nbreport", line 5, in
from nbreport.cli.main import main
File "/opt/miniconda/envs/mytest36/lib/python3.6/site-packages/nbreport/cli/main.py", line 11, in
from ..userconfig import read_config, get_config_path, create_empty_config
File "/opt/miniconda/envs/mytest36/lib/python3.6/site-packages/nbreport/userconfig.py", line 10, in
import ruamel.yaml
ModuleNotFoundError: No module named 'ruamel'

[uranusjr]

The problem seems to be that the Conda ruamel_yaml is actually imported under a different name, despite having an equivalent package name to the PyPI ruamel.yaml:

$ conda create -p test ruamel_yaml
...
$ ls -d test/lib/python3.8/site-packages/ruamel*
test/lib/python3.8/site-packages/ruamel_yaml
test/lib/python3.8/site-packages/ruamel_yaml-0.15.87-py3.8.egg-info
$ test/bin/python -c 'import ruamel.yaml'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'ruamel'
$ test/bin/python -c 'import ruamel_yaml'
$

But this problem was accidentally masked by pip’s inability to detect ruamel_yaml as a satisfying installation on pip install ruamel.yaml.

Conda packager will need to fix the conflicting ruamel_yaml to either correctly identify itself as a different package, or make the import path work (put the module in ruamel/yaml instead) as the pip-distributed one. I don’t think there’s much pip can do here.

[uranusjr]

One workaround would be to delete the .egg-info file that identifies the package. pip relies on it (and its successor .dist-info) to determine whether a package is installed, so removing it would trick pip into installing ruamel.yaml from PyPI. I don’t know whether this would cause consequences to Conda, however.

[harneetvirk]

Opened an issue for conda: conda/conda#10178