broken installation of 'cffi' extension in Python 2 easyconfig when running EasyBuild with Python 3 and using --rpath

[boegel]

I ran into this weird problem when installing the cffi extension in Python-2.7.18-GCCcore-9.3.0.eb:

    building '_cffi_backend' extension
  creating build/temp.linux-x86_64-2.7
  creating build/temp.linux-x86_64-2.7/c
  gcc -fno-strict-aliasing -O2 -ftree-vectorize -march=native -fno-math-errno -fPIC -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -O2 -ftree-vectorize -march=native -fno-math-errno -fPIC -fPIC -I/software/Python/2.7.18-GCCcore-9.3.0/include/python2.7 -c c/_cffi_backend.c -o build/temp.linux-x86_64-2.7/c/_cffi_backend.o
  Fatal Python error: initsite: Failed to import the site module
  Traceback (most recent call last):
    File "/software/Python/2.7.18-GCCcore-9.3.0/lib/python2.7/site-packages/site.py", line 73, in <module>
      __boot()
    File "/software/Python/2.7.18-GCCcore-9.3.0/lib/python2.7/site-packages/site.py", line 26, in __boot
      import imp  # Avoid import loop in Python 3
    File "/usr/lib/python-exec/python3.7/../../../lib/python3.7/imp.py", line 27, in <module>
      import tokenize
    File "/usr/lib/python-exec/python3.7/../../../lib/python3.7/tokenize.py", line 33, in <module>
      import re
    File "/usr/lib/python-exec/python3.7/../../../lib/python3.7/re.py", line 145, in <module>
      class RegexFlag(enum.IntFlag):
  AttributeError: module 'enum' has no attribute 'IntFlag'
  error: command 'gcc' failed with exit status 1

(relevant) details about the environment:

• EasyBuild is being run with python3 (by setting $EB_PYTHON to python3)
• EasyBuild is configured to use RPATH (via eb --rpath)
• both setuptools and enum34 are also installed as extensions in Python-2.7.18-GCCcore-9.3.0.eb, before cffi
• the default python command is actually Python 3 (but I don't think that's relevant here)

It took me a looong time to figure out what is actually going wrong here, so I'm documenting this for eternity...

When installing Python extensions, PythonPackage.install_step makes sure that the installation prefix is prepended to the $PYTHONPATH environment variable; in this case, that's $EBROOTPREFIXlib/python2.7/site-packages.

Installing cffi from source (with pip install) kicks of a Cython installation, which involves compiling stuff from source using gcc.

The first gcc in $PATH is our RPATH wrapper script, which is a bash script that runs the rpath_args.py Python script:

easybuild-framework/easybuild/scripts/rpath_wrapper_template.sh.in

Line 50 in 0da2122

rpath_args_out=$(%(python)s -O %(rpath_args_py)s $CMD '%(rpath_filter)s' '%(rpath_include)s' "$@")

)

rpath_args.py is run using the same python command as used by EasyBuild itself (so python3). But $PYTHONPATH is set up for Python 2 (since we're installing Python packages as extensions for Python 2), and that's where the problem lies:

• pip install triggers a a compilation with gcc;
• gcc is our RPATH wrapper script, which runs python3 rpath_args.py ...;
• the rpath_args.py script triggers an import site;
• site is imported from $EBROOTPREFIXlib/python2.7/site-packages/site.py, since that location is listed first in $PYTHONPATH (so it wins over the standard library locations for python3, uh-oh Typos #1);
  • this site.py was installed by our dear friend setuptools (which is installed as an extension before cffi)
• site.py does an import imp, in a try-catch:

   try:
       import imp
       ...
   except ImportError:
       continue

  This code is used in both Python 2 & 3 it seems, but actually only makes sense in Python 3 (since there's no imp module in Python 2).
  In this case however the import imp actually works, since it's being done in a Python 3 environment (since the rpath_args.py script is being run with python3).
• the import imp leads to import tokenize, which leads to import re, which are all imported from the Python 3 standard library;
• the import re triggers an import enum, which is resolved by the enum34 extension that is installed in $EBROOTPREFIXlib/python2.7/site-packages (uh-oh 1226 mr bayes #2);
• this triggers the AttributeError because the enum package provided by enum34 is not what Python 3 expects (it expects the enum included in Python 3 standard library), and so things go boom 💥 ...

And this is why friends don't let friends combine Python 2 and 3 in the same environment.

This is easy to fix: we just need to make sure that the rpaths_args.py script is run using python -E (to ignore $PYTHONPATH) or python -S (to skip the import site). Doing both is probably the best way forward, to make the RPATH wrapper scripts more robust... The rpaths_args.py script is pretty basic, doesn't need anything special (Python 2/3 standard library is sufficient), so it's fine to limit what it has access to.

edit: side note: pypa/pip#8214 looks very similar, but it really a different problem (this bug has nothing to do with pip perse)

[casparvl]

I am 95% sure this will also solve easybuilders/easybuild-easyconfigs#6113 (after 2.5 years XD).

I agree that -E is needed. If someone unknowingly in the future adds an import to rpaths_args.py (god knows why you'd need one), a -S would not be enough. Whether a -S is needed in addition, I don't dare to say. I don't know enough about the implications of not importing site at initializiation...

[casparvl]

Ah, no wait... it won't solve easybuilders/easybuild-easyconfigs#6113... there, the issue is with rpath_args.py picking up the python lib from LD_LIBRARY_PATH (instead of the one from /lib64). That can only be solved by changing the shebang into

#!/usr/bin/env -u LD_LIBRARY_PATH python

Or something... But that might have other consequences (e.g. what if EB is not being run with system Python...). Probably best we discuss that part in #6113 and not here, the issues are complicated enough without mixing them...

[casparvl]

Hm, same question applies here: what happens if we add -E -S, but the Python being invoked is NOT the system python? (and maybe relies on it's own entries in PYTHONPATH)

I guess there is no way really to solve that scenario, since if PYTHONPATH contains both python2 and python3 stuff, and you want it to somehow magically pick up the correct stuff, there really is no solution...

[boegel]

@casparvl The rpaths_args.py script only requires the Python (2 or 3) standard library, and any python command will find its standard library installation without having $PYTHONPATH set, so using -E is fine.

Let's follow up on the $LD_LIBRARY_PATH issue in easybuilders/easybuild-easyconfigs#6113.

One option could be trying to run python -E -S -s -O rpath_args.py after undefining $LD_LIBRARY_PATH. If that works, we're happy. If not, we try again after restoring $LD_LIBRARY_PATH. Or something like that.