ModuleNotFoundError: No module named 'ruamel'

[svenvanderburg]

With certain new versions of pip and conda the ruamel.yaml installation fails. This is a problem with conda: conda/conda#10178.

[svenvanderburg]

A workaround is to install ruamel.yaml using conda: conda install ruamel.yaml

[raar1]

Is this something we should have in the documentation somewhere? Is there any way to check if the conda installation is broken? It's annoying that it currently just looks like our tool is the one failing (with a pretty obscure error) when it's a conda problem.

[LourensVeen]

You could work around it in your own code by catching the exception:

try:
    import ruamel_yaml as yaml
except ModuleNotFoundError:
    import ruamel.yaml as yaml

Assuming that this mysterious ruamel_yaml module is actually a renamed ruamel.yaml. Although I guess the work-around would have to be applied to YAtiML in this case?

[raar1]

Hmm that's a good idea. Would this workaround cause problems elsewhere in YAtiML if it were in there?

[LourensVeen]

Oh joy. Digging into this a bit, it seems that ruamel_yaml is essentially a fork of an obsolete version of ruamel.yaml. The fork is implemented in the form of a conda-forge recipe which (trivially) patches the source and renames the module. There is in fact a ruamel.yaml package on conda-forge as well, which looks to contain an up-to-date version of the original.

So the problem is that there are two different-but-similar packages whose names differ only by punctuation, and now pip considers them identical but Python's importlib doesn't. So maybe the correct solution would be for PyPA to make sure that their installer and their programming language agree on which package names are and are not equivalent, and for PyPI and Conda-forge to update their rules to disallow equivalent (not just identical) package names.

Alternatively, just removing ruamel_yaml from conda-forge would also work, especially since the only thing it seems to add is confusion. Or they could make it a proper fork with a more distinguishing name, like ruyaml does.

So, I could add a work-around to YAtiML to make it import ruamel.yaml, then fall back to ruamel_yaml if it cannot find it (YAtiML is actually compatible with ruamel.yaml 0.15.80, because I'm trying to avoid version conflicts for my users and papered over the incompatible changes that happened somewhere between 0.15.76 and the current version). But I'd want to test with ruamel_yaml as well then, so that at least I'll find out if they change something more and break my code. It could be made to work, but it sounds like official support for something that shouldn't exist, and that doesn't feel good to be honest.

Of course the only real good solution would be to invent a new programming language with a packaging solution that isn't a horridly bodged together contraption reminiscent of Simone Giertz's shitty robots, a new markup language that isn't an overcomplicated jungle gym, and a library for the markup language written in the programming language that is maintained by someone who knows how to talk to people. But that sounds like work, I'm a lazy Millennial, and it's Friday afternoon, so...

[raar1]

Haha, that is a much deeper rabbit hole than I expected... but given the horrors I've seen with conda/pypi packaging perhaps I should not be so surprised.

I'm a lazy Millennial, and it's Friday afternoon, so...

Avocados have ruined this generation ;)

[LourensVeen]

Actually, the rabbit hole goes even deeper...

When looking up packages, pip follows PEP 503 which specifies that the package name must be normalised before it is appended to a base URL and resolved. This normalisation process replaces each run of periods, dashes and underscores with a single dash. As a result, pip considers ruamel_yaml and ruamel.yaml to be the same name, at least when it comes to looking up packages online.

As a result of that, conda install ruamel_yaml creates an environment in which pip install <package> breaks if <package> depends on ruamel.yaml, because pip will consider the normalised ruamel-yaml dependency already resolved and fail to install ruamel.yaml. This results in import ruamel.yaml giving ModuleNotFoundError. And that ruamel_yaml is installed by default in the conda base environment, which the installation instructions helpfully point out can be made to be active by default everywhere on your system.

It seems that it was Continuum itself which made the ruamel_yaml package, because they didn't like ruamel.yaml being a namespace package. So they forked it and repackaged it. To confuse things a bit further, there's a version of ruamel_yaml on conda-forge as well, which was fixed here, renaming the ruamel_yaml package to ruamel_yaml_conda. This was merged on September 1st, and the package was rebuilt, but since it isn't what's installed by Anaconda by default, it doesn't help.

So I've asked Continuum to patch that version as well. I don't know if that will automatically upgrade everyone's base environment or if they'll have to upgrade manually, but we can figure that out when this is resolved and add a note in the documentation.

[Innanov]

Go to anaconda3\lib\site-packages\rpcq_base.py and change line #22 :
from ruamel import yaml
to
from ruamel_yaml as yaml

[vishxl]

pip install ruamel_yaml

[LourensVeen]

Hi! Thank you for commenting.

Could you please explain how your suggestions help fix this problem? I don't see how changing the source code of rpcq_base will fix yatiml, and the problem is that Conda comes with ruamel_yaml rather that ruamel.yaml, so installing ruamel_yaml sounds like it will make things worse...

[gurubhandari]

Just changing the from ruamel import yaml to import yaml to "/miniconda/lib/python3.8/site-packages/drf_yasg/codecs.py" and installing pip install PyYaml solved the problem for me.

[LourensVeen]

Just a note for anyone reading the above: these people have issues with different Python packages that are hit by the same bug in Anaconda. The fixes posted above will not fix nanopub, because it doesn't use any of them. Also, editing an already installed package like that is a bad idea, because the edit will be reverted on every upgrade.

If you find this issue and you are not using nanopub, please do not post here! Instead, file an issue with the package you are actually using, or go to ContinuumIO/anaconda-issues#12231 and comment there to try to get the underlying issue in Anaconda addressed.

[neelamari]

Below are steps which worked after spending some time on issue

i.e running this " conda install ruamel_yaml" before running the conda update anaconda

curl -O https://repo.anaconda.com/archive/Anaconda3-5.3.1-Linux-x86_64.sh
bash Anaconda3-5.3.1-Linux-x86_64.sh
source .bashrc
conda
conda update conda
conda install ruamel_yaml
conda update anaconda