Packages installed by pip in lib64 are not recognized by pip

[paulcyr]

• Pip version: 9.0.1
• Python version: 3.5
• Operating system: Amazon Linux AMI release 2017.03

Description:

I am working in a virtual environment created with virtualenv. There are a few packages that I am trying to install, however two of them, psycopg2 and pillow, are not working correctly.

When I run pip install <package>, it works as expected. However, neither of the two modules are shown using pip show <package> or pip list. pip install <package> can be run again with the same result.

It turns out that the two packages were being installed in env/lib64/python3.5/dist-packages. All the other packages were being installed in env/lib/python3.5/dist-packages

For some reason, pip is not seeing the packages it has installed in the lib64 folder. Packages installed in the lib folder are recognized correctly.

What I've run:

(env)[ec2-user@ws4 test]$ pip install psycopg2
Collecting psycopg2
  Using cached psycopg2-2.7.1-cp35-cp35m-manylinux1_x86_64.whl
Installing collected packages: psycopg2
Successfully installed psycopg2-2.7.1

(env)[ec2-user@ws4 test]$ pip show psycopg2
(env)[ec2-user@ws4 test]$

(env)[ec2-user@ws4 test]$ pip install pillow
Collecting pillow
  Using cached Pillow-4.1.1-cp35-cp35m-manylinux1_x86_64.whl
Requirement already satisfied: olefile in /var/www/domains/ndp/associations/env/lib/python3.5/dist-packages (from pillow)
Installing collected packages: pillow
Successfully installed pillow-4.1.1

(env)[ec2-user@ws4 test]$ pip show pillow
(env)[ec2-user@ws4 test]$

[nchammas]

I am seeing this exact issue as well on Amazon Linux. pip is installing everything to lib/ except for a couple of libraries, which it installs to lib64/. In my case, those libraries are jellyfish and regex. My current workaround is to use pip --target to force pip to install to lib/ rather than lib64/.

So my questions are:

• Why does pip install some libraries to lib/ vs. lib64/?
• Why isn't pip seeing the stuff it's installing itself to lib64/?

[nchammas]

Probably a long shot, but maybe @dstufft can ping someone at Amazon to take a look at this since this seems to be limited to the combination of pip + Amazon Linux.

[xavfernandez]

This is likely an issue with the python install provided by Amazon Linux as pip "asks" (via distutils) the Python installation where it should install packages, cf https://github.com/pypa/pip/blob/9.0.1/pip/locations.py#L124-L182

I'd love to see the output of :

$ python -c "from pip.locations import distutils_scheme; from pprint import pprint;pprint(distutils_scheme('test'))"
{'data': '/home/xfernandez/.virtualenvs/tmp-76998a82a537a0',
 'headers': '/home/xfernandez/.virtualenvs/tmp-76998a82a537a0/include/site/python3.6/test',
 'platlib': '/home/xfernandez/.virtualenvs/tmp-76998a82a537a0/lib/python3.6/site-packages',
 'purelib': '/home/xfernandez/.virtualenvs/tmp-76998a82a537a0/lib/python3.6/site-packages',
 'scripts': '/home/xfernandez/.virtualenvs/tmp-76998a82a537a0/bin'}

[jpotterm]

@xavfernandez Here's the output on my EC2 instance:

$ /home/ec2-user/spc/venv/bin/python -c "from pip.locations import distutils_scheme; from pprint import pprint;pprint(distutils_scheme('test'))"
{'data': '/home/ec2-user/spc/venv',
 'headers': '/home/ec2-user/spc/venv/include/site/python2.7/test',
 'platlib': '/home/ec2-user/spc/venv/lib64/python2.7/dist-packages',
 'purelib': '/home/ec2-user/spc/venv/lib/python2.7/dist-packages',
 'scripts': '/home/ec2-user/spc/venv/bin'}

[morrone]

Same problem here with pip 9.0.1 and python 3.6. My information for what @xavfernandez asked for is substantially similar to what @jpotterm supplied. I too was able to work around the problem by using "pip install --upgrade --target /path/to/my/sandbox/lib/python3.6/dist-packages". That seems to force everything into lib that was once isntalled in lib64. Now pip and python can find those packages.

[morrone]

I think that this python issue is probably related:

https://bugs.python.org/issue1294959

And perhaps the Amazon image includes a version of the fedora patch that they mention, since platlib for us is pointing to /lib64/:

http://pkgs.fedoraproject.org/cgit/rpms/python3.git/plain/00102-lib64.patch

[data-wrangler]

I was having this same issue, and initially worked around it by forcing everything into the /lib/ directory as @morrone suggested, but one application in particular started having namespace conflicts when I did that.

I was able to resolve that by adding the /lib64/Python3.5/dist-packages directory to my $PYTHONPATH in the .bash_profile, and that seems to have fixed everything. Just wish I had come to this solution earlier.

I'll be testing upgrading our AMI to the 2017.09 release in the near future (as described here) -- hopefully that includes the fedora patch, will update if it resolves this.

[axper]

@data-wrangler I am in the same place as you - I upgraded Amazon Linux to latest version and now it installs a couple of packages into /opt/python/run/venv/lib64/python2.7/site-packages/ in ElasticBeanstalk.

How did you solve this issue?

[data-wrangler]

@axper I didn't find subsequent AMI updates affected the issue, I fixed it with adding the missing directory to the $PYTHONPATH env var in my bash profile.

[chasetb]

I was able to solve the issue in Lambda by running unset PYTHON_INSTALL_LAYOUT which turns off the Amazon specific layout for Python packages. (Shameless plug: https://amoreopensource.wordpress.com/2018/05/25/problems-with-aws-linux-and-pip/) Now they get installed where I expect them. I imagine you could still run this same command in ELB using the commands key in a custom .config file. https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/customize-containers-ec2.html#linux-commands

[nchammas]

@chasetb:

unset PYTHON_INSTALL_LAYOUT

That's a neat solution. Seems to be the most lightweight one proposed here.

Something else for people to consider: I just tried Amazon Linux 2 and it appears to solve this problem. The Amazon Linux 2 AMI IDs are listed here.

[ec2-user@ip-10-97-56-131 ~]$ python3 -m venv venv
[ec2-user@ip-10-97-56-131 ~]$ source venv/bin/activate
(venv) [ec2-user@ip-10-97-56-131 ~]$ pip install -U pip
<snipped>
(venv) [ec2-user@ip-10-97-56-131 ~]$ pip --version
pip 10.0.1 from /home/ec2-user/venv/lib64/python3.7/site-packages/pip (python 3.7)
(venv) [ec2-user@ip-10-97-56-131 ~]$ pip install numpy jellyfish psycopg2 requests sqlparse
<snipped>
(venv) [ec2-user@ip-10-97-56-131 ~]$ pip list
Package    Version  
---------- ---------
certifi    2018.4.16
chardet    3.0.4    
idna       2.7      
jellyfish  0.6.1    
numpy      1.14.5   
pip        10.0.1   
psycopg2   2.7.5    
regex      2018.6.21
requests   2.19.1   
setuptools 38.4.0   
sqlparse   0.2.4    
urllib3    1.23     
(venv) [ec2-user@ip-10-97-56-131 ~]$ pip show psycopg2
Name: psycopg2
Version: 2.7.5
Summary: psycopg2 - Python-PostgreSQL Database Adapter
Home-page: http://initd.org/psycopg/
Author: Federico Di Gregorio
Author-email: fog@initd.org
License: LGPL with exceptions or ZPL
Location: /home/ec2-user/venv/lib/python3.7/site-packages
Requires: 
Required-by: 
(venv) [ec2-user@ip-10-97-56-131 ~]$ pip show jellyfish
Name: jellyfish
Version: 0.6.1
Summary: a library for doing approximate and phonetic matching of strings.
Home-page: http://github.com/jamesturk/jellyfish
Author: UNKNOWN
Author-email: UNKNOWN
License: UNKNOWN
Location: /home/ec2-user/venv/lib/python3.7/site-packages
Requires: 
Required-by: 

So it looks like everything is now being installed to lib/, including the problematic packages that were previously going to lib64/.

[nchammas]

To test whether the fix was from Python 3.7 or from Amazon Linux, I tried the above test with Python 3.4, Amazon Linux 2, pip 10.0.1, and psycopg2, which @paulcyr had reported in the initial post as problematic. It still worked, which suggests to me that the fix is coming specifically from Amazon Linux 2.

[xavfernandez]

It looks like the issue was coming from Amazon Linux and is now fixed.
I'm closing this issue but feel free to reopen one.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.