ImportError in system pip wrappers after an upgrade

[pfmoore]

Overview

After upgrading to pip 10 or higher, many users are encountering error like

• ImportError: cannot import name 'main'
• TypeError: 'module' object is not callable
• ImportError: No module named _internal
• ModuleNotFoundError: No module named 'pip._internal'

These are caused by an incorrect attempt to upgrade pip, which has typically resulted in (parts of) multiple versions of pip being installed in the same Python installation, and those parts being incompatible.

It should be noted that these issues are invariably not problems with pip itself, but are due to incorrect use of pip, or unexpected interactions with system scripts that are not controlled by pip. So while we'll try to help you solve your issue, this is not the "fault" of pip, and you will have to be prepared to do at least some of the debugging and fixes on your own.

You can reach out to your Python provider (eg: Linux Distro, Cloud Provider, etc) for help with this issue.

General Advice

First, some general advice. It is assumed that anyone encountering issues will have failed to follow some part of this advice. Listing these items here is not intended to imply that "it's your fault and we won't help", but rather to help users to identify what went wrong, so that they can work out what to do next more easily.

1. Only ever use your system package manager to upgrade the system pip. The system installed pip is owned by the distribution, and if you don't use distribution-supplied tools to manage it, you will hit problems. Yes, we know pip says "you should upgrade with pip install -U pip" - that's true in a pip-managed installation, ideally distributions should patch this message to give appropriate instructions in the system pip, but they don't. We're working with them on this, but it's not going to happen soon (remember, we're looking at cases where people are upgrading old versions of pip here, so patches to new versions won't help).
2. Never use sudo with pip. This follows on from the first point. If you think you need to use sudo, you're probably trying to modify a distribution-owned file. See point 1.
3. Prefer to use --user. By doing this, you only ever install packages in your personal directories, and so you avoid interfering with the system copy of pip. But there are PATH issues you need to be aware of here. We'll cover these later. Put simply, it's possible to follow this advice, and still hit problems, because you're not actually running the wrapper you installed as --user.

Debugging the Issue

Before trying to work out what's going on, it's critically important that you understand precisely what scripts you are running and what versions of pip the relevant Python interpreter is using.

First, identify the full absolute path of the executable script that you are running. That's often in the traceback you get, but if it isn't, you can use OS tools like which, or Python's shutil.which to identify the right script. Watch out for your shell confusing the issue, with aliases or potentially stale hashed commands.

Once you have identified the script, make sure you can reproduce the problem using the absolute path to that script. If you can't, you probably found the wrong script, so check again.

Second, work out which version of Python the script is using to run pip. You'll often be able to get that from the shebang line of the script. This can often be the trickiest problem, as wrapper scripts can take many forms depending on what tool created them. In the worst case, you can simply make an intelligent guess at this point.

Once you know which Python is running pip, you can run python -m pip to invoke pip. In most cases, this will work fine, as it's the wrapper causing the issue, and not pip itself. Running pip via python -m in this way is often a perfectly acceptable workaround for the issue (at least in the short term).

Now run python -m pip --version. This will give you the exact version and location of the installation of pip that your Python is seeing.

At this point, you're usually done - the fundamental cause of all these problems is running a wrapper script which is written expecting to see a version of pip older than pip 10 (that's why it imports pip.main) under a Python interpreter that sees a copy of pip that's version 10 or later.

Fixing the Issue

The problem, of course, is fixing the issue. And that's where you really are on your own. If you have changed your system installation, you really need to put it back the way that the distribution installed it. That may well require an uninstall and reinstall of your distribution's pip package. Reinstalling is easy, of course - but uninstalling may require manually removing incorrect files. Or you may be able to force-reinstall with your distribution package manager, simply overwriting the incorrect files. Of course, this reverts you to the system-supplied version of pip. If you need a newer version, you should ask your distribution vendor, or use something like virtualenv to install it independently of your system packages.

It may be that you're simply running the "wrong" wrapper script. Maybe you did a --user install of a new version of pip, but your PATH is set to run the system version of the wrapper rather than the user-local one installed with pip. In that case, you can simply fix your PATH. That's usually the issue for people who do pip install --user --upgrade pip and get the pip.main error.

As already noted, python -m pip is a reliable workaround, at the cost of using a more verbose command to invoke pip.

Community Advice

The comments section of this issue is open for people to discuss specific problems, and to share potential solutions. Just to be clear, it's quite likely with problems of this nature that you'll need to modify system-supplied files or settings. You do so at your own risk. If you're not comfortable modifying your OS, or running as root, you should seek expert advice. To put it another way, if by following suggestions given here, you break your system, you get to keep the pieces. There's only so much that can be achieved remotely.

Also, the pip developers don't provide any guarantees that advice in the comments on this issue is correct, or that it won't damage your system. Again, use at your own risk.

Related Issues

The following issues have been reported which are related to this issue:
#5240, #5221, #5588, #5495, #5493, #5487, #5447, #5432, #5373, #5326, #5318, #5253

[denisglotov]

The bug happens when not using virtualenv, using virtualenv fixes the problem on my side.
To fix the duplicate pip problem (described above) and downgrade to original pip version, I just rm -rf ~/.local/lib/python2.7/site-packages/ (beware!).

[randomdudethathelpsyou]

I am pretty new to coding, however I managed to fix it in a kind of odd way.
My problem was that there was nothing in my /usr/bin/ folder regarding pip.
to fix this I had my buddy upload the get-pip.py program to raspberry pi, from which I downloaded the code to my computer and ran it. It then over-rode the old pip program and installed the new one.

Here's the code we used:
working computer to pi: scp /whatever_your_directory_is/get-pip.py pi@pi's_address:/pi's_directory
pi to my computer (the last period is important): scp pi@10.200.130.65:/pi's_directoryi/get-pip.py .
to download pip: sudo python get-pip.py

Sorry if it's weird, but that's what got it working for me

[veneman]

After upgrading pip I ran into the error ImportError: cannot import name 'py31compat' when trying to import pyarrow. I have a conda installation and after rolling back pip to version 9 on all of my environments I still ran into the error.

The fix for me was to change line 72 in site-packages/pkg_resources/__init__.py from from . import py31compat to from tempfile import TemporaryDirectory.

setuptools/py31compat.py is a workaround for older versions of Python that don't have TemporaryDirectory. Since I'm running Python 3.6 this change eliminates the need to use py31compat.

Things seem to be working after this change but I'm not that confident it'll keep working down the road.

[ErichBSchulz]

The advice above:

  Only ever use your system package manager to upgrade pip.

conflicts with the cheerful siren call:

You are using pip version 8.1.1, however version 18.0 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

This seems to be some kind of tarpit :-/

[pfmoore]

@ErichBSchulz We're aware of that - see #5346.

Maybe I should have more explicitly said "Only ever use your system package manager to upgrade your system pip". The more detailed comment in that bullet point makes it clearer, but I've edited the post to make it explicit.

[MannyC]

I followed some instructions elsewhere and ran

sudo apt install python-pip
pip install --upgrade pip

and hit upon this issue.

python -m pip uninstall pip

fixed it, returning me to the system pip

I have to say, although I don't fully understand the complications involved, it doesn't feel like pip should self-immolate like this.

[PieterGit]

I would add this advice of not upgrading your system pip in the documentation.
I just followed these instructions and killed my raspbian pip and pip3 installs.

On the page:
https://packaging.python.org/tutorials/installing-packages/#ensure-pip-setuptools-and-wheel-are-up-to-date it lists python -m pip install --upgrade pip setuptools wheel , which is in conflict with recommentation 1 and 3 in this issue.

I think I restored that with:

python3 -m pip uninstall pip setuptools wheel
sudo apt-get --reinstall install  python3-setuptools python3-wheel python3-pip

python -m pip uninstall pip setuptools wheel
sudo apt-get --reinstall install python-setuptools python-wheel python-pip 

After that my raspbian PI zero pip and pip3 report the ancient 9.0.1 versions, which is the latests available according to https://tracker.debian.org/pkg/python-pip
The request to update pip in debian repo can be found here:
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=901393

[borismo]

pip3 install --user pip seems to break my system pip.

Isn't it a good idea to install the latest version of pip in the user's home directory?

[abhiTronix]

The Ubuntu's python-pip-9.0.1 installed(via apt get python-pip) is conflicting with Latest pip-18.0 provided by pypa. So sudo rm -rf ~/.local/lib/python2.7/site-packages/(Beware!) removes the ubuntu's python-pip and resolves the problem. Only one pip can exist on the system. Thanks @denisglotov

Also Upgrading pip doesn't help as it does'nt overwrite the preexisting installed python-pip but instead create another copy of pip in different directory causing the conflict.

[Ark-kun]

My problem was installing pip in ~/.local/ and having at the end of $PATH.

[ddombrowsky]

I wonder how npm and cpan solve this problem? In any case, I just uninstalled system pip3 and created my own:

#!/bin/sh

exec python3 -m pip "$@"

I'm the only user running it on this system anyway. Victory!

[pradyunsg]

If your environment is breaking, please take that up with whoever provides the Python installation to you. The first message in this issue goes into great lengths describing this situation.

You will only hit this situation when using a launcher script for pip that is not managed by pip's maintainers, has been modified by a downstream redistributor (like a Linux distro or cloud provider) and the pip installation has been modified by the user to update it to a newer version.

The situation where this happens is when you use pip in a way that, whomsoever is providing the Python installation you're using, didn't intend for you to.

For Debian-derivatives like Ubuntu, I'll note that #5599 (comment) is the highest voted comment on this issue. And, well...