{lang}[GCCcore/6.4.0] Python v2.7.13 #4962

JackPerdue · 2017-07-30T16:13:57Z

(created using eb --new-pr)

JackPerdue · 2017-07-30T16:15:09Z

was #PR 4914

JackPerdue · 2017-07-30T16:15:26Z

PR #4914

JackPerdue · 2017-07-30T16:37:56Z

Travis liked this one... #4914 closed

verdurin · 2017-07-30T20:36:13Z

Test report by @verdurin
SUCCESS
Build succeeded for 2 out of 2 (1 easyconfigs in this PR)
ca004.camp.thecrick.org - Linux centos linux 7.3.1611, Intel(R) Xeon(R) CPU E5-2640 v3 @ 2.60GHz, Python 2.7.5
See https://gist.github.com/c433f81cfce8b3f046edd36b578ba082 for a full test report.

verdurin

Looks good

verdurin · 2017-07-30T20:38:40Z

Merging, thanks @JackPerdue

boegel · 2017-08-07T13:04:34Z

easybuild/easyconfigs/p/Python/Python-2.7.13-GCCcore-6.4.0-bare.eb

+preconfigopts = """
+    sed -e\
+      's:$(LINKCC) $(LDFLAGS) $(LINKFORSHARED):$(LINKCC) $(LDFLAGS) $(LINKFORSHARED) -Wl,-rpath=%(installdir)s/lib:'\
+      -i.eb Makefile.pre.in


Bleh, not too happy with this part...

At the very list, this should be done properly via an rpath toolchain option, and it should be clearly explained why the RPATH is really needed.

This is just another sign that using a bare Python as a build dep later to be replaced by a 'full' Python installation higher up the hierarchy is a bad idea imho...

Thoughts @damianam?

To be honest, I am not sure why is this even needed. Why can't this be used without rpathing?

@boegel well, AFAIK, there is no way to tell Travis to use --expirmental. Why should I be FORCED to make three different versions (iomkl/intel/foss) for a module when the GCCcore one will suffice as a builddep that can be used with any of those later?

@damianam because pythonpackage.py doesn't set LD_LIBRARY_PATH that includes builddeps when it does a "python -c 'import 'foo'" check.. So the check fails because python can't find libpythonX.Y.so. The alternative is disable that check in everything that uses Python-bare, which seems much worse than any concern expressed thus far.

Y'all could fix this by updating pythonpackage.py to set the LD_LIBRARY_PATH for the Python-bare that was used as a builddep. Not sure if that's the best solution. I just know this one works).

Like I said, I have some 50+ modules using Python bare. I"ve spent a lot of time working through the issues. Multiply that by 2 versions of Python. Y'all are basically saying I should have to multiply that by 3 (iomkl/intel/foss). So I end up with 300 modules when 100 would have sufficed. That's just a waste of time and space.

In our setup, we have Python in GCCcore and a bunch of packages on top. On that sense, it is similar to what you say. Some, like numpy, scipy, etc are in the appropriate toolchain (top level with math libraries), which needed to rebuild the interpreter to get fast exponential and trigonometric functions. Others like mpi4py, were simply built on top of the python in GCCcore. In the latter case, I haven't run into the issues you described. The only difference I see is that I've listed Python in dependencies, and not builddependencies. In cases were python is just used for building, without any particular package, we simply rely on the system python. I guess that explains why that hasn't been a problem for us. In our case the setup you are proposing (python-bare for builddependencies of some packages) is not really necessary.

Certainly you have to step up to a MPI toolchain if you need MPI or a toolchain with math for numpy/scipy. eg we have these

Boost-1.61.0-gompi-2017A-Python-2.7.12-bare.eb
Boost-1.61.0-iimpi-2017A-Python-2.7.12-bare.eb
Boost-1.61.0-iimpi-2017A-Python-3.5.2-bare.eb
Boost-1.61.0-iompi-2017A-Python-2.7.12-bare.eb
Boost-1.61.0-iompi-2017A-Python-3.5.2-bare.eb

because Boost needs MPI for some parts.

How does a system Python (e.g. 2.6.6 on RHEL6) provide you with usuable packages for 2.7.13 and 3.6.2? Somebody tried that trick with X11 and it doesn't work in the long run (causes all kinds of weird problems later on).
.

How does a system Python (e.g. 2.6.6 on RHEL6) provide you with usuable packages for 2.7.13 and 3.6.2? Somebody tried that trick with X11 and it doesn't work in the long run (causes all kinds of weird problems later on).

Well, it doesn't. What I meant is that if python (as in the python interpreter to be used to run a simple python script) is needed for building, we use the system python. Anything more convoluted than that will use our Python-2.7.13-GCCcore-5.4.0.eb or Python-3.6.1-GCCcore-5.4.0.eb, which are listed as dependencies, not builddependencies. I haven't found any case where python should be in builddependencies, but the system python isn't enough for it. X11 needs python just for xcb-proto, as stated in the easyconfig file:

# Python is required for xcb-proto allow_system_deps = [('Python', SYS_PYTHON_VERSION)]

so we simply use that and don't introduce any Python dependency.

JackPerdue · 2017-08-07T14:02:17Z

Note: Here's a quick list of things I have working with Python-bare and GCCcore....

NTLR ASE asyncoro ATK cairo cairocffi cffi colorama Cycler Cython dbus-glib dispy Docutils FFmpeg FIAT FLTK FOX freeglut FTGL GConf Gdk-Pixbuf GEOS gettext Ghostscript GLEW GLib GObject-Introspection GST-plugins-base GStreamer GTK+ HarfBuzz ImageMagick jemalloc json2html libdap libdrm libepoxy libglade libGLU libICE libspatialite libxml2 libxslt LLVM Mako MarkupSafe matplotlib Mesa mpmath myriad Pango ply PostgreSQL psycopg2 PyCairo pycparser PyGObject PyGTK PyOpenGL PyQt PyQt5 PySlurm Python PyYAML PyZMQ QCA QJson QScintilla QScintilla5 Qt Qt5 Qwt QwtPolar rebuild.FLTK rebuild.PyGObject requests setuptools simplejson SIP six SWIG sympy Tk VTK wxPython X11 xprop

I see no reason to have to build three versions for each of those when one will do.

boegel · 2017-08-07T14:13:52Z

@JackPerdue I'm not entirely convinced that your approach is 'safe' in a general sense...

What you are doing is installing Python packages with one particular Python installation (built with GCCcore), so it can be used with multiple other Python installations (of the same version), built with another toolchain (intel, foss, etc.).

How sure are you that's actually (always) OK to do? I would expect it's possible to run into ABI compatibility issues? Are the .pyc binaries you obtain with each of the different Python builds (again, sticking to a particular Python version) 100% identical?
If they are, can we get confirmation on that somewhere (probably from a Python core developer or so), and is this approach future-proof? If not, we're in trouble already (at least in theory).

Next to the concerns above, the module names being produced are misleading. The -Python-<version> in them indicates a runtime dependency on that particular version of Python, but when Python is listed as a build dep, it's not loaded as a dependency for that module, and hence the module doesn't work 'out-of-the-box'. In contrast to before, where modules with -Python-<version> in there name do work as expected because that Python version gets loaded as a runtime dependency.

I understand your motivations for avoiding duplicate installations, I'm just concerned whether we are confident that this is a proper solution...

JackPerdue · 2017-08-07T14:37:01Z

Why would there be ABI issues? They are all using the same GCCcore and binutils.

JackPerdue · 2017-08-09T02:36:44Z

re: ABI questions..... currently trying to find some answers using a tool called abi-compliance-checker.. rather tedious getting all the builds/modules right.... so far no problems re: API/ABI

JackPerdue · 2017-08-09T02:43:19Z

Maybe I should just change the name of this module from Python-bare to just pybuilddep and then have all modules have a "-pybuilddep-2.7.13" (or whatever) versionsuffix to make it clear Python isn't included. I think that would solve some issues, especially since Python would be loaded later and thus take precedent in PATHs.

JackPerdue · 2017-08-09T02:45:26Z

Y'all will have to update pythonpackage.py to set LD_LIBRARY_PATH w/builddeps OR implement rpath OR take my rpath hack that works.... or I'm taking my ball home..... :)

boegel · 2017-08-09T12:44:03Z

Support for hard enabling use of RPATH is being worked on in easybuilders/easybuild-framework#2283...

After giving this some thought, I'm wondering why we don't create a -core easyconfig for Python instead that uses GCCcore as toolchain and includes a couple of (compiler-only) Python packages as extensions, e.g. setuptools, Cython, six, PyYAML, etc.

This apprach has a couple of advantages over the current approach:

distinction between -bare (no extensions) and -core (only compiler-only extensions)
no additional easyconfigs/modules required for setuptools & co on top of a Python installed with GCCcore
no problems with modules for Python packages in which the required Python runtime is missing

I think this also eliminates the need for RPATH linking, but I'm not entirely sure about that.

Also, I think this approach is fairly close to what @damianam & co are doing at JSC.

Thoughts @JackPerdue, @damianam?

boegel · 2017-08-09T12:55:13Z

W.r.t. using one Python build (e.g. with intel toolchain) as a drop-in replacement of another (e.g. with GCCcore toolchain, same Python version)...

Here's what I get when doing an import numpy using a python (2.7.13) installed with foss/2017a, when importing a numpy package that was installing with python installed with intel/2017a:

>>> import numpy
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/user/home/gent/vsc400/vsc40023/eb_phanpyscratch/CO7/sandybridge/software/Python/2.7.13-intel-2017a/lib/python2.7/site-packages/numpy-1.12.1-py2.7-linux-x86_64.egg/numpy/__init__.py", line 142, in <module>
    from . import add_newdocs
  File "/user/home/gent/vsc400/vsc40023/eb_phanpyscratch/CO7/sandybridge/software/Python/2.7.13-intel-2017a/lib/python2.7/site-packages/numpy-1.12.1-py2.7-linux-x86_64.egg/numpy/add_newdocs.py", line 13, in <module>
    from numpy.lib import add_newdoc
  File "/user/home/gent/vsc400/vsc40023/eb_phanpyscratch/CO7/sandybridge/software/Python/2.7.13-intel-2017a/lib/python2.7/site-packages/numpy-1.12.1-py2.7-linux-x86_64.egg/numpy/lib/__init__.py", line 8, in <module>
    from .type_check import *
  File "/user/home/gent/vsc400/vsc40023/eb_phanpyscratch/CO7/sandybridge/software/Python/2.7.13-intel-2017a/lib/python2.7/site-packages/numpy-1.12.1-py2.7-linux-x86_64.egg/numpy/lib/type_check.py", line 11, in <module>
    import numpy.core.numeric as _nx
  File "/user/home/gent/vsc400/vsc40023/eb_phanpyscratch/CO7/sandybridge/software/Python/2.7.13-intel-2017a/lib/python2.7/site-packages/numpy-1.12.1-py2.7-linux-x86_64.egg/numpy/core/__init__.py", line 24, in <module>
    raise ImportError(msg)
ImportError:
Importing the multiarray numpy extension module failed.  Most
likely you are trying to import a failed build of numpy.
If you're working with a numpy git repo, try `git clean -xdf` (removes all
files not under version control).  Otherwise reinstall numpy.

numpy may not be the best example here, but it seems to confirm my concern that swapping out one Python build for another for the same Python version is not a good idea...

JackPerdue · 2017-08-10T20:44:46Z

I'm quite sure mixing toolchains (foss/intel) won't work due to different MPI and maths. And numpy needs maths so it always has to be built with a full toolchain. So numpy can never be a -core package.

-core works for me.

Will the Python in the -core package be a builddep? If not, what happens when I want to use a Python-GCCcore-core module with, say, other modules (e.g. numpy) that use the full toolchain?

JackPerdue · 2017-08-22T19:22:40Z

As mentioned on Slack/IRC, but not here, I did some ABI testing to try to find issues with my approach. I'm welcome to ideas on nomenclature but I'm fairly sure this approach works pretty well.

ABI test results at: http://www.siliconslick.com/easybuild/ABIPython/

I recommend starting with 00_README

I couldn't find any problems (I tried).

boegel · 2017-08-30T10:08:58Z

I fleshed out my Python-core proposal in #5072

adding easyconfigs: Python-2.7.13-GCCcore-6.4.0-bare.eb

23625fc

JackPerdue mentioned this pull request Jul 30, 2017

{lang}[GCCcore/6.4.0] Python v2.7.13 #4914

Closed

verdurin added this to the 3.4.0 milestone Jul 30, 2017

verdurin approved these changes Jul 30, 2017

View reviewed changes

verdurin merged commit 0d02fa8 into easybuilders:develop Jul 30, 2017

verdurin mentioned this pull request Jul 31, 2017

{lib}[GCCcore/6.4.0] PyYAML v3.12 #4915

Merged

boegel reviewed Aug 7, 2017

View reviewed changes

JackPerdue mentioned this pull request Aug 7, 2017

{lang}[GCCcore/6.4.0] Cython v0.26 #4966

Merged

JackPerdue mentioned this pull request Aug 9, 2017

{lang}[GCCcore/6.4.0] pybuilddep v2.7.13 #4995

Closed

boegel mentioned this pull request Aug 9, 2017

add 'rpath' easyconfig parameter to enable use of RPATH regardless of EasyBuild configuration easybuilders/easybuild-framework#2283

Open

boegel mentioned this pull request Aug 30, 2017

add easyconfig for 'Python-core' built with GCCcore which can be used as a build dep #5072

Closed

boegel mentioned this pull request Aug 30, 2017

remove bare Python easyconfig using GCCcore/6.4.0 toolchain + easyconfigs that require it #5075

Merged

Micket mentioned this pull request Jan 21, 2019

Python @ GCCcore #7463

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

{lang}[GCCcore/6.4.0] Python v2.7.13 #4962

{lang}[GCCcore/6.4.0] Python v2.7.13 #4962

JackPerdue commented Jul 30, 2017

JackPerdue commented Jul 30, 2017

JackPerdue commented Jul 30, 2017

JackPerdue commented Jul 30, 2017

verdurin commented Jul 30, 2017

verdurin left a comment

verdurin commented Jul 30, 2017

boegel Aug 7, 2017

damianam Aug 7, 2017

JackPerdue Aug 7, 2017

damianam Aug 7, 2017

JackPerdue Aug 7, 2017

damianam Aug 8, 2017

JackPerdue commented Aug 7, 2017

boegel commented Aug 7, 2017

JackPerdue commented Aug 7, 2017

JackPerdue commented Aug 9, 2017

JackPerdue commented Aug 9, 2017

JackPerdue commented Aug 9, 2017

boegel commented Aug 9, 2017

boegel commented Aug 9, 2017

JackPerdue commented Aug 10, 2017

JackPerdue commented Aug 22, 2017

boegel commented Aug 30, 2017

{lang}[GCCcore/6.4.0] Python v2.7.13 #4962

{lang}[GCCcore/6.4.0] Python v2.7.13 #4962

Conversation

JackPerdue commented Jul 30, 2017

JackPerdue commented Jul 30, 2017

JackPerdue commented Jul 30, 2017

JackPerdue commented Jul 30, 2017

verdurin commented Jul 30, 2017

verdurin left a comment

Choose a reason for hiding this comment

verdurin commented Jul 30, 2017

boegel Aug 7, 2017

Choose a reason for hiding this comment

damianam Aug 7, 2017

Choose a reason for hiding this comment

JackPerdue Aug 7, 2017

Choose a reason for hiding this comment

damianam Aug 7, 2017

Choose a reason for hiding this comment

JackPerdue Aug 7, 2017

Choose a reason for hiding this comment

damianam Aug 8, 2017

Choose a reason for hiding this comment

JackPerdue commented Aug 7, 2017

boegel commented Aug 7, 2017

JackPerdue commented Aug 7, 2017

JackPerdue commented Aug 9, 2017

JackPerdue commented Aug 9, 2017

JackPerdue commented Aug 9, 2017

boegel commented Aug 9, 2017

boegel commented Aug 9, 2017

JackPerdue commented Aug 10, 2017

JackPerdue commented Aug 22, 2017

boegel commented Aug 30, 2017