-
Notifications
You must be signed in to change notification settings - Fork 6
Docker and Python Quirks on PowerPC
-
pip install
on non-x86_64 often requires compilation which takes minutes -
upstream bugs might necessitate you to manually install dependencies first
We are using an IBM Power9 machine called bu-21-9. Specs:
-
Red Hat Enterprise Linux 7.6
-
Docker version 1.13.1
System software is out-of-date, which is not a good starting point.
tl;dr it’s slow to install numpy.
Typically (on Intel x86_64, aka amd64) it’s a breeze to do a
pip install numpy
Installation takes a few seconds because PyPi houses precompiled "wheels" (binary packages) for amd64.
These are not available for ppc64le
(PowerPC 64 Little-Endian) and you have no choice but to compile from source.
On bu-21-9 the command
time docker run --rm python:buster pip install numpy
takes 2.6 minutes.
tl;dr upstream bug, do pip install numpy && pip install scikit-image
docker run --rm python:buster pip install scikit-image
The command above actually fails on both amd64
and ppc64le
.
Collecting scikit-image Downloading scikit-image-0.17.2.tar.gz (29.8 MB) ERROR: Command errored out with exit status 1: command: /usr/local/bin/python -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-_c7n1tuv/scikit-image/setup.py'"'"'; __file__='"'"'/tmp/pip-install-_c7n1tuv/scikit-image/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-pip-egg-info-eud0npsh cwd: /tmp/pip-install-_c7n1tuv/scikit-image/ Complete output (7 lines): Traceback (most recent call last): File "<string>", line 1, in <module> File "/tmp/pip-install-_c7n1tuv/scikit-image/setup.py", line 234, in <module> 'build_ext': openmp_build_ext(), File "/tmp/pip-install-_c7n1tuv/scikit-image/setup.py", line 58, in openmp_build_ext from numpy.distutils.command.build_ext import build_ext ModuleNotFoundError: No module named 'numpy' ---------------------------------------- ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
scikit-image
does not provide the precompiled binary "wheel" for any platform, so every installation must compile it from source.
The problem lies with a circular dependency in its setup.py which imports numpy
before listing numpy
as a requirement.
There is nothing we can do about upstream bugs but to manually install numpy
in our Dockerfile
pip install numpy && pip install scikit-image
With some versions of pip (known issue with pypip 9.0.1, known working with pip 20.2.4) you get this error:
Traceback (most recent call last): File "/tmp/pip-build-v7lch0r1/numpy/tools/cythonize.py", line 59, in process_pyx from Cython.Compiler.Version import version as cython_version ModuleNotFoundError: No module named 'Cython'
Similar to the problem with scikit-image
, here Cython
is an unspecified build dependency.
You don’t get this error on amd64 since a wheel is available for amd64.
Again we don’t have a choice but to manually install build dependencies.
pip3 install Cython && pip3 install numpy
Or updating pip might work
pip3 install --upgrade pip && pip install numpy