enhance systemtools.py to also support POWER systems #1044

boegel · 2014-09-12T17:11:27Z

fixes issues reported in easybuilders/easybuild#32 by @chwilk

…WER, and clean things up

boegel · 2014-09-12T18:06:21Z

@wpoely86: please review?

stdweird · 2014-09-13T09:16:49Z

easybuild/tools/systemtools.py

@@ -219,30 +222,40 @@ def get_cpu_speed():
        try:


seeing this block made me think of http://www.isitfriday.org/ on a thursday....

why is this whole block wrapped in a try/except?

please recheck, cleaned up significantly now

The affinity mask is a POSIX thing right? So why not try the BSD thing and then fall back to the affinity mask?

I don't think the BSD one takes into account cpuset and all that crap, the affinity one does, so that should be tried first (and it should always work on Linux)

yeah agreed. But now, it's only tried on Linux. Why not try it always?

because it won't work on non-Linux systems :)

JackPerdue · 2014-12-17T21:05:54Z

I wholeheartedly endorse this change (given my need to get building on PPC64). 👍

JackPerdue · 2015-01-30T15:39:34Z

Note: If you could add "PPC = 'PPC'" somewhere around line 55 that would make it easier to add the following to toolchains/compiler/gcc.py:

COMPILER_OPTIMAL_ARCHITECTURE_OPTION = {
...
systemtools.PPC : 'mcpu=native' # no support for march=native on PPC

And make it easier for me to try to figure out how to make this bit of code in tools/toolchain/compiler.py work:

    if build_option('optarch') is not None:
        optarch = build_option('optarch')
    elif self.arch in (self.COMPILER_OPTIMAL_ARCHITECTURE_OPTION or []):
        optarch = self.COMPILER_OPTIMAL_ARCHITECTURE_OPTION[self.arch]

(my python Kung Fu is the worst)

But this code works on Power7 running a lpar kernel with a /proc/cpuinfo that looks like the below. So far I have (gcc only):

$ find /software/easybuild/modulefiles/ -type f | grep -c .
294
$ module spider |& grep -c Homepage:
184
on:
$ uname -a
Linux p7-5229-lpar 2.6.32-431.29.2.el6.ppc64 #1 SMP Sun Jul 27 15:53:50 EDT 2014 ppc64 ppc64 ppc64 GNU/Linux
with /proc/cpuinfo similar to:
processor : 0
cpu : POWER7 (architected), altivec supported
clock : 4228.000000MHz
revision : 2.1 (pvr 004a 0201)
[repeat for processor=4 to 60 step 4]
timebase : 512000000
platform : pSeries
model : IBM,8246-L2T
machine : CHRP IBM,8246-L2T

Anyway... I strongly endorse this update (since I'm under a lot of pressure to get EasyBuild working on P7 and am hoping to at least get GCC stable/happy while trying to create ibmxl.py and friends for the IBM compilers/etc.).

…functions

… flag

wpoely86 · 2015-02-25T12:40:37Z

easybuild/tools/systemtools.py

        except ValueError:
            pass

-    raise SystemToolsException('Can not determine number of cores on this system')
+    if core_cnt is None:
+        raise SystemToolsException('Can not determine number of cores on this system')


and POWER?

nothing specific for POWER here, we only need to differentiate between Linux and Darwin

hpcugentbot · 2015-02-25T12:44:39Z

Refer to this link for build results (access rights to CI server needed):
https://jenkins1.ugent.be/job/easybuild-framework-pr-builder/1421/
Test PASSed.

wpoely86 · 2015-02-25T12:45:00Z

easybuild/tools/systemtools.py

+            model = out.strip()
+            _log.debug("Determined CPU model on Darwin using cmd '%s': %s" % (cmd, model))
+
+    if model is None:


nothing for power?

sure, see the comment right above where model_regex is defined

hpcugentbot · 2015-02-25T16:27:14Z

Refer to this link for build results (access rights to CI server needed):
https://jenkins1.ugent.be/job/easybuild-framework-pr-builder/1425/
Test PASSed.

…n True for specified path

hpcugentbot · 2015-02-25T17:43:23Z

Refer to this link for build results (access rights to CI server needed):
https://jenkins1.ugent.be/job/easybuild-framework-pr-builder/1428/
Test PASSed.

stdweird · 2015-02-25T18:56:08Z

easybuild/tools/systemtools.py

-    from vsc.utils.affinity import sched_getaffinity
-except ImportError:
-    pass
+from vsc.utils.affinity import sched_getaffinity


oooh, this uses ctypes and assumes stuff about glibc. careful with this one on non-linux.

only used on Linux

stdweird · 2015-02-25T19:19:40Z

@boegel looks ok, regexps are unnecessarily complicated though

hpcugentbot · 2015-02-26T10:53:11Z

Refer to this link for build results (access rights to CI server needed):
https://jenkins1.ugent.be/job/easybuild-framework-pr-builder/1429/
Test PASSed.

hpcugentbot · 2015-02-26T12:20:49Z

Refer to this link for build results (access rights to CI server needed):
https://jenkins1.ugent.be/job/easybuild-framework-pr-builder/1430/
Test PASSed.

boegel · 2015-02-26T12:28:35Z

remarks fixed, unit tests are happy, so (finally!) good to go in, in time for EasyBuild v2.0

thanks to @ocaisa, I've been able to confirm that this basically passes all tests on Linux/POWER too (there's a small issue w.r.t. a hanging test (or test setup) in toolchain.py on the system where this was tested on, but is likely more related to modulecmd.tcl being used, I'll figure that out)

thanks for the thorough review, @wpoely86 and @stdweird, and @JackPerdue for the feedback (there's a dedicated POWER constant available now, and the requested changes were made in the GCC compiler support)

enhance systemtools.py to also support POWER systems

boegel added 2 commits September 12, 2014 17:34

fix get_cpu_model in case /proc/cpuinfo doesn't list a model name

f3db58e

enhance get_cpu_speed to also support determining CPU frequency on PO…

b2ab456

…WER, and clean things up

boegel mentioned this pull request Sep 13, 2014

Problem using eb on POWER architecture easybuilders/easybuild#32

Closed

stdweird reviewed Sep 13, 2014
View reviewed changes

boegel force-pushed the develop branch from fe70342 to b2278f0 Compare December 18, 2014 08:49

Merge branch 'develop' into systemtools_fixes_power

b97f84f

boegel mentioned this pull request Feb 11, 2015

EasyBuild 2.0 #1000

Closed

27 tasks

boegel added 12 commits February 25, 2015 09:06

clean up implementation of get_cpu_speed

00bfdb5

thoroughly test possible code paths in get_cpu_speed by mocking used …

6f900a4

…functions

clean up get_avail_core_count, enhance testing for it

82f3b4a

clean up get_cpu_vendor and enhance dedicated test for it

526c4fe

clean up get_cpu_model and enhance dedicated test for it

793af55

enhance error message in get_glibc_version

d6b33a0

moar systemtools testing via mocking used functions

708de3e

add get_cpu_family() function in systemtools.py + dedicated test for it

1a9f9a3

use get_cpu_family() rather than get_cpu_vendor() for picking optarch…

b6f57c3

… flag

use -mcpu=native for GCC/Clang on POWER

c977b3e

split up systemtools tests into smaller ones (native vs mocked)

9bde497

Merge branch 'develop' into systemtools_fixes_power

900b660

wpoely86 reviewed Feb 25, 2015
View reviewed changes

boegel added 2 commits February 25, 2015 16:22

fix small remark

11d199b

add support for letting get_cpu_vendor return IBM + code cleanup

eab3153

boegel added 2 commits February 25, 2015 17:31

fix test_cpu_vendors test

cdc875d

fix mocked version of os.path.exists in systemtools tests, only retur…

403acba

…n True for specified path

stdweird reviewed Feb 25, 2015
View reviewed changes

fix remarks in systemtools.py

06ec32a

use a single regex for obtaining model name

7ff5511

boegel added a commit that referenced this pull request Feb 26, 2015

Merge pull request #1044 from boegel/systemtools_fixes_power

3c0f1d1

enhance systemtools.py to also support POWER systems

boegel merged commit 3c0f1d1 into easybuilders:develop Feb 26, 2015

boegel deleted the systemtools_fixes_power branch February 26, 2015 12:28

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

enhance systemtools.py to also support POWER systems #1044

enhance systemtools.py to also support POWER systems #1044

boegel commented Sep 12, 2014

boegel commented Sep 12, 2014

stdweird Sep 13, 2014

stdweird Sep 13, 2014

boegel Feb 25, 2015

wpoely86 Feb 25, 2015

boegel Feb 25, 2015

wpoely86 Feb 25, 2015

boegel Feb 25, 2015

JackPerdue commented Dec 17, 2014

JackPerdue commented Jan 30, 2015

wpoely86 Feb 25, 2015

boegel Feb 25, 2015

hpcugentbot commented Feb 25, 2015

wpoely86 Feb 25, 2015

boegel Feb 25, 2015

hpcugentbot commented Feb 25, 2015

hpcugentbot commented Feb 25, 2015

stdweird Feb 25, 2015

boegel Feb 26, 2015

stdweird commented Feb 25, 2015

hpcugentbot commented Feb 26, 2015

hpcugentbot commented Feb 26, 2015

boegel commented Feb 26, 2015

enhance systemtools.py to also support POWER systems #1044

enhance systemtools.py to also support POWER systems #1044

Conversation

boegel commented Sep 12, 2014

boegel commented Sep 12, 2014

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

JackPerdue commented Dec 17, 2014

JackPerdue commented Jan 30, 2015

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hpcugentbot commented Feb 25, 2015

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hpcugentbot commented Feb 25, 2015

hpcugentbot commented Feb 25, 2015

Choose a reason for hiding this comment

Choose a reason for hiding this comment

stdweird commented Feb 25, 2015

hpcugentbot commented Feb 26, 2015

hpcugentbot commented Feb 26, 2015

boegel commented Feb 26, 2015