pip install virtualenv broken / missing zipp package

[hyperknot]

Thanks for submitting an issue!

If submitting a BUG please provide:

• Minimal reproducible example or detailed descriptions
• the output of the virtual environment creation with the -vvv --with-traceback flags included
• pip list of the environment where virtualenv is installed into if not using the zipapp

I've created a minimal reproducible Dockerfile, it's self explanatory:

FROM ubuntu:16.04

RUN apt-get update && apt-get install -y --no-install-recommends \
    python python-pip python-pkg-resources python-setuptools python-wheel python-dev libxslt1-dev
RUN pip install virtualenv --disable-pip-version-check --no-cache-dir


WORKDIR /root
RUN virtualenv venv

Full install log:

UN pip install virtualenv --disable-pip-version-check --no-cache-dir
 ---> Running in 92fa3d30b758
Collecting virtualenv
  Downloading https://files.pythonhosted.org/packages/a3/dd/e43866ac0d25cd1ecbde0814f6cd7372ea257fbdf443dc9f0a01740da93d/virtualenv-20.0.4-py2.py3-none-any.whl (4.6MB)
Collecting six<2,>=1.9.0 (from virtualenv)
  Downloading https://files.pythonhosted.org/packages/65/eb/1f97cb97bfc2390a276969c6fae16075da282f5058082d4cb10c6c5c1dba/six-1.14.0-py2.py3-none-any.whl
Collecting distlib<1,>=0.3.0 (from virtualenv)
  Downloading https://files.pythonhosted.org/packages/7d/29/694a3a4d7c0e1aef76092e9167fbe372e0f7da055f5dcf4e1313ec21d96a/distlib-0.3.0.zip (571kB)
Collecting importlib-metadata<2,>=0.12; python_version < "3.8" (from virtualenv)
  Downloading https://files.pythonhosted.org/packages/8b/03/a00d504808808912751e64ccf414be53c29cad620e3de2421135fcae3025/importlib_metadata-1.5.0-py2.py3-none-any.whl
Collecting appdirs<2,>=1.4.3 (from virtualenv)
  Downloading https://files.pythonhosted.org/packages/56/eb/810e700ed1349edde4cbdc1b2a21e28cdf115f9faf263f6bbf8447c1abf3/appdirs-1.4.3-py2.py3-none-any.whl
Collecting importlib-resources<2,>=1.0; python_version < "3.7" (from virtualenv)
  Downloading https://files.pythonhosted.org/packages/2f/f7/b4aa02cdd3ee7ebba375969d77c00826aa15c5db84247d23c89522dccbfa/importlib_resources-1.0.2-py2.py3-none-any.whl
Collecting filelock<4,>=3.0.0 (from virtualenv)
  Downloading https://files.pythonhosted.org/packages/14/ec/6ee2168387ce0154632f856d5cc5592328e9cf93127c5c9aeca92c8c16cb/filelock-3.0.12.tar.gz
Collecting contextlib2<1,>=0.6.0; python_version < "3.3" (from virtualenv)
  Downloading https://files.pythonhosted.org/packages/85/60/370352f7ef6aa96c52fb001831622f50f923c1d575427d021b8ab3311236/contextlib2-0.6.0.post1-py2.py3-none-any.whl
Collecting pathlib2<3,>=2.3.3; python_version < "3.4" and sys_platform != "win32" (from virtualenv)
  Downloading https://files.pythonhosted.org/packages/e9/45/9c82d3666af4ef9f221cbb954e1d77ddbb513faf552aea6df5f37f1a4859/pathlib2-2.3.5-py2.py3-none-any.whl
Collecting zipp>=0.5 (from importlib-metadata<2,>=0.12; python_version < "3.8"->virtualenv)
  Downloading https://files.pythonhosted.org/packages/60/85/668bca4a9ef474ca634c993e768f12bd99af1f06bb90bb2655bc538a967e/zipp-2.2.0.tar.gz
  Running setup.py (path:/tmp/pip-build-pW7K_Y/zipp/setup.py) egg_info for package zipp produced metadata for project name unknown. Fix your #egg=zipp fragments.
Collecting configparser>=3.5; python_version < "3" (from importlib-metadata<2,>=0.12; python_version < "3.8"->virtualenv)
  Downloading https://files.pythonhosted.org/packages/7a/2a/95ed0501cf5d8709490b1d3a3f9b5cf340da6c433f896bbe9ce08dbe6785/configparser-4.0.2-py2.py3-none-any.whl
Collecting typing; python_version < "3.5" (from importlib-resources<2,>=1.0; python_version < "3.7"->virtualenv)
  Downloading https://files.pythonhosted.org/packages/22/30/64ca29543375759dc589ade14a6cd36382abf2bec17d67de8481bc9814d7/typing-3.7.4.1-py2-none-any.whl
Collecting scandir; python_version < "3.5" (from pathlib2<3,>=2.3.3; python_version < "3.4" and sys_platform != "win32"->virtualenv)
  Downloading https://files.pythonhosted.org/packages/df/f5/9c052db7bd54d0cbf1bc0bb6554362bba1012d03e5888950a4f5c5dadc4e/scandir-1.10.0.tar.gz
Installing collected packages: six, distlib, contextlib2, unknown, scandir, pathlib2, configparser, importlib-metadata, appdirs, typing, importlib-resources, filelock, virtualenv
  Running setup.py install for distlib: started
    Running setup.py install for distlib: finished with status 'done'
  Running setup.py install for unknown: started
    Running setup.py install for unknown: finished with status 'done'
  Running setup.py install for scandir: started
    Running setup.py install for scandir: finished with status 'done'
  Running setup.py install for filelock: started
    Running setup.py install for filelock: finished with status 'done'
Successfully installed appdirs-1.4.3 configparser-4.0.2 contextlib2-0.6.0.post1 distlib-0.3.0 filelock-3.0.12 importlib-metadata-1.5.0 importlib-resources-1.0.2 pathlib2-2.3.5 scandir-1.10.0 six-1.14.0 typing-3.7.4.1 unknown-0.0.0 virtualenv-20.0.4
Removing intermediate container 92fa3d30b758
 ---> 6127ebfdaa58
Step 5/6 : WORKDIR /root
 ---> Running in 3998fd059d5e
Removing intermediate container 3998fd059d5e
 ---> 62a1db87d19c
Step 6/6 : RUN virtualenv venv
 ---> Running in c40b1478666a
Traceback (most recent call last):
  File "/usr/local/bin/virtualenv", line 7, in <module>
    from virtualenv.__main__ import run_with_catch
  File "/usr/local/lib/python2.7/dist-packages/virtualenv/__init__.py", line 3, in <module>
    from .run import cli_run
  File "/usr/local/lib/python2.7/dist-packages/virtualenv/run/__init__.py", line 9, in <module>
    from .plugin.activators import ActivationSelector
  File "/usr/local/lib/python2.7/dist-packages/virtualenv/run/plugin/activators.py", line 6, in <module>
    from .base import ComponentBuilder
  File "/usr/local/lib/python2.7/dist-packages/virtualenv/run/plugin/base.py", line 9, in <module>
    from importlib_metadata import entry_points
  File "/usr/local/lib/python2.7/dist-packages/importlib_metadata/__init__.py", line 9, in <module>
    import zipp
ImportError: No module named zipp
The command '/bin/sh -c virtualenv venv' returned a non-zero code: 1

[hyperknot]

The same commands work if I change Ubuntu's version number to 18.04

[hyperknot]

So I got some help on IRC, pip install -U pip fixed this issue.

Still, this would be great to either:

• fixed or
• terminate the installation with an error message if the system's pip version is too old to support recent versions

[gaborbernat]

We've just today improved on this with #1628; though just trying it seems this gets you to:

Traceback (most recent call last):
  File "/usr/local/bin/virtualenv", line 6, in <module>
    from pkg_resources import load_entry_point
  File "/usr/local/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 3241, in <module>
    @_call_aside
  File "/usr/local/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 3225, in _call_aside
    f(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 3254, in _initialize_master_working_set
    working_set = WorkingSet._build_master()
  File "/usr/local/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 585, in _build_master
    return cls._build_from_requirements(__requires__)
  File "/usr/local/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 598, in _build_from_requirements
    dists = ws.resolve(reqs, Environment())
  File "/usr/local/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 786, in resolve
    raise DistributionNotFound(req, requirers)
pkg_resources.DistributionNotFound: The 'zipp>=0.5' distribution was not found and is required by importlib-metadata

As the zipp packages can't work with setuptools 41.0.0...

[gaborbernat]

setuptools 42.0.0 works though in this case 👍

[asottile]

this isn't a bug in virtualenv, if you pip install zipp you'll see the same issue

it's a combination of two things:

1. pip<9 will ignore python_requires selecting the latest version
2. python2.x pip cannot utilize the py36-none-any wheel so it must build from source
3. setuptools<30.3 cannot build declarative metadata packages from source

so you end up with:

• a broken installation of python3.6+ zipp on python2.x named "unknown==0.0.0"

the easiest fix is to use a newer version of pip to install virtualenv, or to utilize the zipapp

[asottile]

@gaborbernat I don't think this is virtualenv itself, since a py2.py3-none-any wheel is provided there, and that will be used all the way back to pip==1.5

[gaborbernat]

Yeah, it's not itself; but our users are hit with this; I'm tempted to bump https://github.com/pypa/virtualenv/blob/master/setup.py#L4 to 42... which seems to solve it.

[asottile]

that code doesn't run at all when installed from a wheel

[hyperknot]

From a user point of view, the best would be an error message saying something like: "Unsupported pip version, please run 'pip install -U pip' first".

[gaborbernat]

In case of a wheel installation, we can't do anything, the issue is already past us. If someone is installing from the source than they don't necessarily need newer pip, just newer setuptools. If they have newer setuptools we can fail with something more helpful. This is the best we can do here I believe.

[hyperknot]

You could even display that at runtime. It should simply display an error message when run on very old pip versions.

[asottile]

@gaborbernat that doesn't help though, pip will still install a version of zipp that's incompatible with the running python (1.1 is the last version supporting python2.x):

$ virtualenv venv -ppython2
Running virtualenv with interpreter /usr/bin/python2
Already using interpreter /usr/bin/python2
New python executable in /tmp/y/venv/bin/python2
Also creating executable in /tmp/y/venv/bin/python
Installing setuptools, pip, wheel...
done.
$ . venv/bin/activate
$ pip install 'pip<9' 'setuptools>=42'
DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support
Collecting pip<9
  Using cached pip-8.1.2-py2.py3-none-any.whl (1.2 MB)
Requirement already satisfied: setuptools>=42 in ./venv/lib/python2.7/site-packages (44.0.0)
Installing collected packages: pip
  Attempting uninstall: pip
    Found existing installation: pip 20.0.2
    Uninstalling pip-20.0.2:
      Successfully uninstalled pip-20.0.2
Successfully installed pip-8.1.2
$ pip install zipp
Collecting zipp
  Downloading https://files.pythonhosted.org/packages/60/85/668bca4a9ef474ca634c993e768f12bd99af1f06bb90bb2655bc538a967e/zipp-2.2.0.tar.gz
Building wheels for collected packages: zipp
  Running setup.py bdist_wheel for zipp ... done
  Stored in directory: /home/asottile/.cache/pip/wheels/a0/1f/9d/d132b3fbc38b33322681e3a7830c0a29fce7f55c3a06b85c65
Successfully built zipp
Installing collected packages: zipp
Successfully installed zipp-2.2.0
You are using pip version 8.1.2, however version 20.0.2 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

[gaborbernat]

Thinking this through I don't think we can do anything here. If it's already installed the only way we would know if our dependencies are correct is to manually check... and that's very expensive.

[gaborbernat]

All we need to do here is to document what @asottile said above as per #1618

[pfmoore]

From a user point of view, the best would be an error message saying something like: "Unsupported pip version, please run 'pip install -U pip' first".

Given that pip already produces a warning (which I notice you disabled in the docker script) when you use an older version, suggesting that you upgrade, would adding another message telling you that you are on an out of date version actually help?

(That's not a rhetorical or sarcastic question - one of the big problems with Python packaging is that whenever we introduce new standards, older versions of the tools don't support them, and we need to find the best way to encourage users to keep up to date. So knowing what is and isn't working in how we try to get the message out to users is really important to us).

[hyperknot]

which I notice you disabled in the docker script

Where? I see all the warnings, Python 2.7 EOL, etc.

Anyway, maybe the best thing we could do is to have this here for Google, at least now it finds this issue for "ImportError: No module named zipp" so people can find the pip update fix.

[asottile]

--disable-pip-version-check

[hyperknot]

That's needed in Dockerfile to avoid caching. I was thinking of some kind of Warning like the 2.7 EOL red lines, those are very visible.

[hyperknot]

So I tried without it, and it just shows that meaningless.

You are using pip version 8.1.1, however version 20.0.2 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

line. Of course everyone disables this in Dockerfiles as this is always displayed, even as a soon as 20.0.3 will be released.

[scubbo]

Still getting this issue, despite being on the latest version of pip:

$ pip3 --version
WARNING: pip is being invoked by an old script wrapper. This will fail in a future version of pip.
Please see https://github.com/pypa/pip/issues/5599 for advice on fixing the underlying issue.
To avoid this problem you can invoke Python with '-m pip' instead of running pip directly.
pip 20.2.3 from /Users/jackjack/Library/Python/3.7/lib/python/site-packages/pip (python 3.7)
$ pip3 install -U pip
# removing repeated "pip is being invoked by an old script wrapper" messages
Defaulting to user installation because normal site-packages is not writeable
Requirement already up-to-date: pip in /Users/jackjack/Library/Python/3.7/lib/python/site-packages (20.2.3)
$ pip3 install -U pip3
Defaulting to user installation because normal site-packages is not writeable
ERROR: Could not find a version that satisfies the requirement pip3 (from versions: none)
ERROR: No matching distribution found for pip3
$ pip3 install -U virtualenv
Defaulting to user installation because normal site-packages is not writeable
Requirement already up-to-date: virtualenv in /Users/jackjack/Library/Python/3.7/lib/python/site-packages (20.0.33)
Requirement already satisfied, skipping upgrade: distlib<1,>=0.3.1 in /Users/jackjack/Library/Python/3.7/lib/python/site-packages (from virtualenv) (0.3.1)
Requirement already satisfied, skipping upgrade: appdirs<2,>=1.4.3 in /Users/jackjack/Library/Python/3.7/lib/python/site-packages (from virtualenv) (1.4.4)
Requirement already satisfied, skipping upgrade: importlib-metadata<3,>=0.12; python_version < "3.8" in /Users/jackjack/Library/Python/3.7/lib/python/site-packages (from virtualenv) (2.0.0)
Requirement already satisfied, skipping upgrade: six<2,>=1.9.0 in /Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/site-packages (from virtualenv) (1.12.0)
Requirement already satisfied, skipping upgrade: filelock<4,>=3.0.0 in /Users/jackjack/Library/Python/3.7/lib/python/site-packages (from virtualenv) (3.0.12)
Requirement already satisfied, skipping upgrade: zipp>=0.5 in /Users/jackjack/Library/Python/3.7/lib/python/site-packages (from importlib-metadata<3,>=0.12; python_version < "3.8"->virtualenv) (3.3.0)
$ sudo /usr/bin/easy_install virtualenv
Searching for virtualenv
Best match: virtualenv 20.0.33
Processing virtualenv-20.0.33-py2.7.egg
virtualenv 20.0.33 is already the active version in easy-install.pth
Installing virtualenv script to /usr/local/bin

Using /Library/Python/2.7/site-packages/virtualenv-20.0.33-py2.7.egg
Processing dependencies for virtualenv
Searching for zipp>=0.4
Reading https://pypi.org/simple/zipp/
Downloading https://files.pythonhosted.org/packages/7b/50/8a41d217ae898bb7a422a55a2a2afc22dfaab42b7bce5fe771a3892fe5e7/zipp-3.3.0.tar.gz#sha256=64ad89efee774d1897a58607895d80789c59778ea02185dd846ac38394a8642b
Best match: zipp 3.3.0
Processing zipp-3.3.0.tar.gz
Writing /tmp/easy_install-QuPyHI/zipp-3.3.0/setup.cfg
Running zipp-3.3.0/setup.py -q bdist_egg --dist-dir /tmp/easy_install-QuPyHI/zipp-3.3.0/egg-dist-tmp-2lSPpU
  File "build/bdist.macosx-10.15-x86_64/egg/zipp.py", line 240
    def open(self, mode='r', *args, pwd=None, **kwargs):
                                      ^
SyntaxError: invalid syntax

zip_safe flag not set; analyzing archive contents...
Removing /Library/Python/2.7/site-packages/zipp-0.0.0-py2.7.egg
Copying zipp-0.0.0-py2.7.egg to /Library/Python/2.7/site-packages
zipp 0.0.0 is already the active version in easy-install.pth

Installed /Library/Python/2.7/site-packages/zipp-0.0.0-py2.7.egg
error: The 'zipp>=0.4' distribution was not found and is required by importlib-resources
$ /usr/bin/easy_install virtualenv
error: can't create or remove files in install directory

The following error occurred while trying to add or remove files in the
installation directory:

    [Errno 13] Permission denied: '/Library/Python/2.7/site-packages/test-easy-install-20543.write-test'

The installation directory you specified (via --install-dir, --prefix, or
the distutils default setting) was:

    /Library/Python/2.7/site-packages/

Perhaps your account does not have write access to this directory?  If the
installation directory is a system-owned directory, you may need to sign in
as the administrator or "root" account.  If you do not have administrative
access to this machine, you may wish to choose a different installation
directory, preferably one that is listed in your PYTHONPATH environment
variable.

For information on other options, you may wish to consult the
documentation at:

  https://setuptools.readthedocs.io/en/latest/easy_install.html

Please make the appropriate changes for your system and try again.

Similar errors even if I've sudo pip3 install'd virtualenv

[asottile]

easy_install is not pip, don't use it it's never what you want

[scubbo]

Various guides (including this one) claim that it's a fix to this problem (whereby virtualenv is installed via pip, but not findable on PATH). What would you suggest as an alternative?

[asottile]

yikes, definitely don't follow their advice

in your case you're doing a --user installation, which means you need to put the right bin directory on the PATH -- probably /Users/jackjack/Library/Python/3.7/bin if I had to guess (though I'm not familiar with macos)

the first installation should have warned you about this (you can probably get the messaging again with pip3 uninstall virtualenv && pip3 install virtualenv

[scubbo]

Hmmm, interesting. So it turns out there's already a file a /usr/local/bin/virtualenv with the following contents:

#!/System/Library/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS/Python
# EASY-INSTALL-ENTRY-SCRIPT: 'virtualenv==20.0.33','console_scripts','virtualenv'
__requires__ = 'virtualenv==20.0.33'
import re
import sys
from pkg_resources import load_entry_point

if __name__ == '__main__':
    sys.argv[0] = re.sub(r'(-script\.pyw?|\.exe)?$', '', sys.argv[0])
    sys.exit(
        load_entry_point('virtualenv==20.0.33', 'console_scripts', 'virtualenv')()
    )

Note it's hard-coding to 2.7 in the shebang, which is probably the source of some of the issues, since I'm install with pip3.

I had to make sure to add that path (which you guessed correctly - and which was also output from pip3 install virtualenv when I reinstalled!) before /usr/local/bin to make sure it took precedence. Working now - thanks!

[asottile]

I'd suggest uninstalling that one, but lol, easy_install doesn't have an uninstall (that was one of the big features pip brought to the table when replacing easy_install)