-
Notifications
You must be signed in to change notification settings - Fork 145
Conference call notes 20170329
Kenneth Hoste edited this page Apr 26, 2017
·
9 revisions
(back to Conference calls)
Notes on the 73rd EasyBuild conference call, Wednesday March 29th 2017 (5pm - 6pm CET)
Alphabetical list of attendees (??):
- Damian Alvarez (JSC, Germany)
- Davide Vanzo (Vanderbilt University)
- Markus Geimer (JSC, Germany)
- Kenneth Hoste (HPC-UGent, Belgium)
- Bart Oldeman (McGill University, Canada)
- report on benchmarking of Python/numpy built with GCC/Intel compilers + Intel MKL by Damian
- boegelbot reporting broken unit tests in PRs
- moderator for EB conf calls on April 12th & 26th?
- Q&A
- JSC is testing the performance of a Python installation in GCCcore, but a numpy library in the top level of the toolchain hierarchy (e.g. with Intel compilers, in a bundle with other scipy-stack libraries).
- For most Python related things, it didn't seem to matter whether Python was built with GCC or other compilers
- PGI doesn't built Python properly...
- so (minimal version of) Python was installed with GCCcore to provide it everywhere
- The benchmarking was done using https://github.com/serge-sans-paille/numpy-benchmarks
- Most benchmarks perform satisfactorily
- Significant slowdown in functions that rely on symbols defined by
libm
- Significant slowdown in functions that rely on symbols defined by
- What happened before (interpreter compiled with icc):
- Everything was compiled with icc, so
libimf
was used instead oflibm
- Everything was compiled with icc, so
- What happened after having a base python in
GCCcore
:- Even though numpy itself was compiled with icc, when launching the interpreter (compiled with gcc),
libm
was loaded, and theexp
and related symbols are picked from there, instead of the fasterlibimf
- can be worked out/verified by preloading libimf
- Even though numpy itself was compiled with icc, when launching the interpreter (compiled with gcc),
- Workaround at the moment:
- The SciPy-Stack bundle installs a
python
interpreter compiled with icc, that gets precedence over the one loaded inGCCcore
- 2 (well 3)
python
s available via$PATH
...
- 2 (well 3)
- The Python module defines
PYTHONPATH
(to use all the modules installed there) since differentpython
is being used...
- The SciPy-Stack bundle installs a
- some differences may be explained by comparing different numpy versions
- largest differences (~2x slowdown) explained by libm vs libimf
- see vibr_energy, log_likelihood and arc_distance
- See https://github.com/numpy/numpy/issues/8823
- Is statically linking
libimf
intonumpy
a solution? - Bart: will also check whether they are hitting this with Python provided via Nix (together with wrapper module) + scipy stack compiled with EasyBuild with Intel compiler
- related to https://github.com/hpcugent/easybuild-easyblocks/issues/1136?
- this needed to be fixed, but not related to performance issues
- For most Python related things, it didn't seem to matter whether Python was built with GCC or other compilers
- e.g. https://github.com/hpcugent/easybuild-easyconfigs/pull/4346#issuecomment-287622541
- done with script available at https://github.com/boegel/eb-scripts/blob/master/pr_check.py
pr_check.py --travis
- keep an eye on https://travis-ci.org/hpcugent/easybuild-easyconfigs/pull_requests
- reports back broken tests in PRs
-
Davide: ncurses
--with-termlib
issue: https://github.com/hpcugent/easybuild-easyconfigs/issues/4049- mainly triggered for Python 3 which requires XZ, to dance around cyclic dep between XZ-gettext-ncurses
- why was
--with-termlib
added to ncurses? because oflibreadline
? cfr. https://github.com/hpcugent/easybuild-easyconfigs/pull/3545 - should figure out why ncurses is being picked up from
Core/
in the first place...-
--minimal-toolchains
is enabled, but--add-dummy-to-minimal-toolchains
was not - Lmod is loading
ncurses/6.0
fromCore
while it shouldn't? wrong$MODULEPATH
?
-
- possible solutions:
- add another 'configure' loop iteration without
--with-termlib
to installed => libncurses will not have termlib symbols stripped out, butlibtinfo
will be available... - give ncurses with
dummy
a specificversionsuffix
to avoid it being picked up asncurses/6.0
- add another 'configure' loop iteration without
-
Davide: plan to enhance MATLAB easyblock to fix problems with recent versions
-
Markus: feedback on extra parameters related to documentation?
- https://github.com/hpcugent/easybuild-framework/pull/2113
- should be possible to get this in soon, in time for v3.2.0 (ETA mid/end-April)
-
Markus: Clang as toolchain compiler without Fortran support
- is Fortran support strictly required by the EasyBuild toolchain mechanism?
- if it is, then we could make it more flexible
- only for tools/apps that don't require Fortran support
- unclear error message blocked progress on this, Markus will come back to it
- is Fortran support strictly required by the EasyBuild toolchain mechanism?
-
Markus: update on EasyBuild tutorial proposal for SC17?
- deadline is April 12th 2017, cfr. http://sc17.supercomputing.org/submitters/tutorials/
- Kenneth can help out with the proposal, but will most likely not be attending SC17...
- Markus will check with Damian/Alan