update custom easyblock for Boost to always build single and multi threaded versions #2456

akesandgren · 2021-06-02T14:44:47Z

(created using eb --new-pr)

Fixes Boost end of old problem with using CMake and Boost.Python, still need change in cmakemake.py to complete the fix.

akesandgren · 2021-06-02T14:57:02Z

Still not warning about ignoring boost_multi_thread... Not sure it's really needed...
esp if we drop it from 1.76, since it's only used in 1.74

akesandgren · 2021-06-02T15:30:36Z

This of course need more testing...

Flamefire · 2021-06-02T17:29:50Z

easybuild/easyblocks/b/boost.py


        if self.cfg['boost_mpi']:
            self.log.info("Building boost_mpi library")
-            self.build_boost_variant(bjamoptions + " --with-mpi", paracmd)
+            mpi_bjamoptions = " --with-mpi"
+            self.build_boost_variant(self.bjamoptions + mpi_bjamoptions, self.paracmd)


The goal should be to do a single invocation of b2 with everything. To use mpi you can add using mpi ; to the user-config.jam file. I'll double check if that is the same

Yeah, didn't get that to work with my first tries, so I decided to PR it as it is to start with...

I'm not so sure about that. Think Boost.MPI the same way we have Boost.Python, then we would in reality only want it to build the mpi parts, i.e. using --with-mpi, but it would then be the only build/install step it does, just like for the python part.

Yes, but that is not what it currently is: boost_mpi = True in the EC additionally installs Boost.MPI and we can't change that until EB 5.x
However it makes sense providing a (new) knob to build only Boost.MPI (same as Boost.Python) because we moved Boost to GCCcore, i.e. no MPI there.
We might even have boost_mpi = None being the default and the EasyBlock decides whether to build with or without MPI based on the currently loaded modules/dependencies.

We could simply check if name = Boost.MPI do --with-mpi and only that (and verify that usempi = True first)
Else if either boost_mpi=True (requires usempi=True) or just usempi = True then add "using mpi" to config and build everything, perhaps verifying that we have MPI in the toolchain in all cases.

usempi=True/False should be the determining factor for MPI since that's what is mostly used in other easyblocks and for simpler easyconfigs.

Nah, I'd prefer not to change this any more, it does the "same" thing as the old one, i.e. multiple b2 invocations when boost_mpi=True, producing all the libs.
It's better to deal with this MPI stuff in a separate PR.

I.e. my Boost-1.74.0-gompi-2020b.eb which has boost_mpi=True produces all the libs of standard boost + the mpi ones and creates the correct links when using this boost.py.

Hm, but multiple invocations of b2 was the source of the problem that lead to this. So IMO we should fix that together. I can offer to tackle the remaining issues on top of this. But yeah, maybe in a separate PR, I guess doesn't matter much

No multiple invocations was not really the problem, it was doing it with layout=system once and layout=tagged once. Now I always do layout=tagged so there is no problem with multiple invocations.

I mean: if we do a single invocation, then there is no room for such mistakes at all.

I agree that this can be changed in a follow up PR, let's get this one merged as is to avoid dragging it along...

Flamefire · 2021-06-03T07:36:48Z

easybuild/easyblocks/b/boost.py

+                lib_mt_suffix += '-x64'
+
+        shlib_ext = get_shared_lib_ext()
+        lib_glob = 'lib*%s.%s.%s' % (lib_mt_suffix, shlib_ext, self.version)


This replace and glob stuff here gives me headaches. Not sure if all is correct.
Currently we have the following:

libboost_type_erasure.a

libboost_type_erasure-mt-x64.a

libboost_type_erasure-mt-x64.so

libboost_type_erasure-mt-x64.so.1.74.0

libboost_type_erasure.so

libboost_type_erasure.so.1.74.0

So I would say the glob should be lib*.* and the replacement simply: "use current name but remove everything from first '-' until first dot".
This removes the need for the lib_mt_suffix and the arch stuff above and makes it way simpler to understand: "Remove the tag"

When doing threading=single,multi --layout=tagged you get:

libboost_type_erasure-mt-x64.a libboost_type_erasure-mt-x64.so@ libboost_type_erasure-mt-x64.so.1.74.0* libboost_type_erasure-x64.a libboost_type_erasure-x64.so@ libboost_type_erasure-x64.so.1.74.0*

and nothing else, and that's what the glob/replace deals with.

and nothing else

The "currently" was referring to current EB installations of Boost.

Ah, forgot the single-threaded. Then glob for lib*-mt*.* instead. Then 2 comments like "Glob for the mt libs" and "remove the tags from the name" would be enough and the code much easier to understand. Maybe even include those names of yours as examples in a comment

yeah, testing...

Better now I hope...

easybuild/easyblocks/b/boost.py

Simplyfy lib globbing and replacement for default libs Co-authored-by: Alexander Grund <Flamefire@users.noreply.github.com>

…ning.

… by Alexander Grund <Flamefire@users.noreply.github.com>

akesandgren · 2021-06-03T13:40:51Z

There seem to be problems with older Boost versions that need to be looked at a bit more...

akesandgren · 2021-06-03T13:53:24Z

@Flamefire Boost 1.71 (at least) doesn't protect itself correctly in the multiple-variants case:

-- Found Boost 1.71.0 at /home/a/ake/Public/easybuild/software/Boost/1.71.0-gompi-2019b/lib/cmake/Boost-1.71.0
--   Requested configuration: QUIET REQUIRED COMPONENTS system;filesystem;python;thread
-- BoostConfig: find_package(boost_headers 1.71.0 EXACT CONFIG REQUIRED QUIET HINTS /home/a/ake/Public/easybuild/software/Boost/1.71.0-gompi-2019b/lib/cmake)
-- Found boost_headers 1.71.0 at /home/a/ake/Public/easybuild/software/Boost.Python/1.71.0-gompi-2019b/lib/cmake/boost_headers-1.71.0
-- BoostConfig: find_package(boost_system 1.71.0 EXACT CONFIG REQUIRED QUIET HINTS /home/a/ake/Public/easybuild/software/Boost/1.71.0-gompi-2019b/lib/cmake)
-- Found boost_system 1.71.0 at /home/a/ake/Public/easybuild/software/Boost/1.71.0-gompi-2019b/lib/cmake/boost_system-1.71.0
-- Boost toolset is gcc8 (GNU 8.3.0)
-- Scanning /home/a/ake/Public/easybuild/software/Boost/1.71.0-gompi-2019b/lib/cmake/boost_system-1.71.0/libboost_system-variant*.cmake
--   Including /home/a/ake/Public/easybuild/software/Boost/1.71.0-gompi-2019b/lib/cmake/boost_system-1.71.0/libboost_system-variant-mt-x64-shared.cmake
--   [x] libboost_system-mt-x64.so.1.71.0
--   Including /home/a/ake/Public/easybuild/software/Boost/1.71.0-gompi-2019b/lib/cmake/boost_system-1.71.0/libboost_system-variant-mt-x64-static.cmake
--   [ ] libboost_system-mt-x64.a
--   Including /home/a/ake/Public/easybuild/software/Boost/1.71.0-gompi-2019b/lib/cmake/boost_system-1.71.0/libboost_system-variant-x64-shared.cmake
--   [x] libboost_system-x64.so.1.71.0
CMake Warning at /home/a/ake/Public/easybuild/software/Boost/1.71.0-gompi-2019b/lib/cmake/boost_system-1.71.0/libboost_system-variant-x64-shared.cmake:59 (message):
  Target Boost::system already has an imported location
  '/home/a/ake/Public/easybuild/software/Boost/1.71.0-gompi-2019b/lib/libboost_system-mt-x64.so.1.71.0',
  which will be overwritten with
  '/home/a/ake/Public/easybuild/software/Boost/1.71.0-gompi-2019b/lib/libboost_system-x64.so.1.71.0'
Call Stack (most recent call first):
  /home/a/ake/Public/easybuild/software/Boost/1.71.0-gompi-2019b/lib/cmake/boost_system-1.71.0/boost_system-config.cmake:43 (include)
  /home/a/ake/Public/easybuild/software/Boost/1.71.0-gompi-2019b/lib/cmake/Boost-1.71.0/BoostConfig.cmake:117 (find_package)
  /home/a/ake/Public/easybuild/software/Boost/1.71.0-gompi-2019b/lib/cmake/Boost-1.71.0/BoostConfig.cmake:182 (boost_find_component)
  /cvmfs/ebsw.hpc2n.umu.se/amd64_ubuntu2004_bdw/software/CMake/3.15.3-GCCcore-8.3.0/share/cmake-3.15/Modules/FindBoost.cmake:443 (find_package)
  CMakeLists.txt:5 (FIND_PACKAGE)

Note that, for 1.71, it's just a warning so the cmake correctly finds all components.
So perhaps we should do just single build with system-layout for older boost versions...

Flamefire · 2021-06-03T14:14:49Z

Note that, for 1.71, it's just a warning so the cmake correctly finds all components.

Yes but the wrong ones: It uses the single threaded ones which would be a breaking change and likely wrong.

So perhaps we should do just single build with system-layout for older boost versions.

We need the tagged layout or it would be a breaking change in EB. However I think we should make the single-threaded build opt-in and clearly warn for known bad versions. Could you attach the 2 files (libboost_system-variant-*x64-shared.cmake) please?

Edit: Ok, found it: The multi-thread default was introduced in 1.73: boostorg/boost_install@c7a4270

akesandgren · 2021-06-03T15:11:54Z

By single build above I mean dropping threading=xxx and -layout=... which would produce the same libraries as we used to have, currently checking which version starts supplying cmake files and the -x86 suffix repsectively.

akesandgren · 2021-06-03T15:20:01Z

The threading=multi --layout=tagged option was added to boost.py in commit e2e4de1 back in 2017.

But 1.74.0 is the first easyconfig which actually turned on boost_multi_thread.
So up to, and including, Boost-1.72.0-gompi-2020a.eb we have only built the default build with system layout as far as I can see.

Flamefire · 2021-06-03T15:38:24Z

Ah you are right, we always build with system layout and without threading set anyway.

Given that the boost_multi_thread option is actually: Build also as tagged. And I remember someone in slack wanting the tagged layout. So I'd say:

Make the layout (maybe only "tagged" or "system"?) an EC option, but always create the symlinks as some dependencies might require that (-lboost_system)
deprecate boost_multi_thread
when boost_multi_thread is set, enable the tagged layout (backwards compat)
explicitly set threading=multi for clarity (or add a comment, or both)
add an EC option for optional threading=single with an error if system-layout is chosen.

--> New options tagged_layout and single_threaded

akesandgren · 2021-06-03T16:16:28Z

Note that 1.70.0 is the first version with cmake files included.
(Well I only checked version 1.69.0 and 1.70.0)

…o only add default links when appropriate.

akesandgren · 2021-06-04T06:42:47Z

@Flamefire @boegel How does this look?
We could say that single_threaded implies tagged_layout instead, and then just enforce it if they choose it...

akesandgren · 2021-06-04T06:46:42Z

Btw, unless I made a mistake in testing, this produces the exact same result as before for everything up to 1.68.0, from 1.69 it produces an identical result but with tagged layout by default.
And it works with CMake straight off from 1.70 (although until 1.73 cmake will complain slightly but still produce a working result)

Still have to finish my changes changes to cmakemake.py to make the last step to complete all of this.

easybuild/easyblocks/b/boost.py

verdurin · 2021-06-21T09:02:31Z

Any update on this one?

Flamefire · 2021-06-21T09:26:25Z

LGTM, started a couple tests

Flamefire · 2021-06-21T10:35:19Z

Test report by @Flamefire

Overview of tested easyconfigs (in order)

SUCCESS Boost-1.74.0-GCC-10.2.0.eb
SUCCESS Boost-1.72.0-gompic-2020a.eb
SUCCESS Boost-1.70.0-gompic-2019a.eb
SUCCESS Boost.Python-1.70.0-gompic-2019a.eb
SUCCESS Boost.Python-1.72.0-gompic-2020a.eb
SUCCESS Boost.Python-1.74.0-GCC-10.2.0.eb

Build succeeded for 6 out of 6 (6 easyconfigs in total)
taurusi6572.taurus.hrsk.tu-dresden.de - Linux RHEL 7.9, x86_64, Intel(R) Xeon(R) CPU E5-2680 v3 @ 2.50GHz (haswell), Python 2.7.5
See https://gist.github.com/05cab24bca0d3a3e41620da32ccfc3c1 for a full test report.

SebastianAchilles · 2021-06-21T12:09:50Z

Test report by @SebastianAchilles

Overview of tested easyconfigs (in order)

SUCCESS Boost-1.74.0-GCC-10.2.0.eb
SUCCESS Boost-1.72.0-gompic-2020a.eb
SUCCESS Boost.Python-1.72.0-gompic-2020a.eb
SUCCESS Boost.Python-1.74.0-GCC-10.2.0.eb

Build succeeded for 4 out of 4 (4 easyconfigs in total)
rocky8-eb - Linux rocky linux 8.4, x86_64, Intel(R) Core(TM) i7-6900K CPU @ 3.20GHz (broadwell), Python 3.6.8
See https://gist.github.com/c8cab758c850134e6944d1f7e07d86ea for a full test report.

SebastianAchilles · 2021-06-21T12:13:11Z

Test report by @SebastianAchilles

Overview of tested easyconfigs (in order)

SUCCESS Boost-1.74.0-GCC-10.2.0.eb
SUCCESS Boost-1.72.0-gompic-2020a.eb
SUCCESS Boost.Python-1.72.0-gompic-2020a.eb
SUCCESS Boost.Python-1.74.0-GCC-10.2.0.eb

Build succeeded for 4 out of 4 (4 easyconfigs in total)
centos8-eb - Linux centos linux 8.3.2011, x86_64, Intel(R) Core(TM) i7-8700 CPU @ 3.20GHz (skylake), Python 3.6.8
See https://gist.github.com/1c74d214e6fbace347d804b54de7b032 for a full test report.

verdurin · 2021-06-21T19:28:53Z

@boegel @ocaisa any chance of merging this one?

boegel · 2021-06-29T15:26:48Z

easybuild/easyblocks/b/boost.py

@@ -120,6 +123,13 @@ def prepare_step(self, *args, **kwargs):
    def configure_step(self):
        """Configure Boost build using custom tools"""

+        # boost_multi_thread is deprecated
+        if self.cfg['boost_multi_thread'] is not None:
+            self.log.deprecated("boost_multi_thread has been deprecated, it has been replaced by tagged_layout. "


@akesandgren The easyconfigs for Boost 1.74.0 should be updated accordingly, since they set boost_multi_thread = True.

Couldn't find a PR for that, did I overlook it?

No I didn't make one yet. Didn't know if we wanted to change that already or not.

easybuild/easyblocks/b/boost.py

…syblock

Co-authored-by: Alexander Grund <Flamefire@users.noreply.github.com>

boegel · 2021-06-29T22:11:35Z

Test report by @boegel

Overview of tested easyconfigs (in order)

SUCCESS Boost-1.74.0-GCC-10.2.0.eb
SUCCESS Boost-1.74.0-iccifort-2020.4.304.eb

Build succeeded for 2 out of 2 (2 easyconfigs in total)
node3553.doduo.os - Linux RHEL 8.2, x86_64, AMD EPYC 7552 48-Core Processor (zen2), Python 3.6.8
See https://gist.github.com/9ee8bc4ed6a7b68370e5b6762e22e342 for a full test report.

boegel · 2021-06-29T22:44:03Z

Test report by @boegel

Overview of tested easyconfigs (in order)

SUCCESS Boost-1.66.0-foss-2018a.eb
SUCCESS Boost-1.71.0-gompi-2019b.eb
SUCCESS Boost-1.71.0-iimpi-2019b.eb
SUCCESS Boost-1.72.0-GCCcore-9.3.0-no_mpi.eb
SUCCESS Boost-1.72.0-gompi-2020a.eb
SUCCESS Boost-1.72.0-iimpi-2020a.eb
SUCCESS Boost-1.74.0-GCC-10.2.0.eb
SUCCESS Boost-1.74.0-iccifort-2020.4.304.eb
SUCCESS Boost.Python-1.66.0-foss-2018a-Python-3.6.4.eb
SUCCESS Boost.Python-1.71.0-gompi-2019b.eb
SUCCESS Boost.Python-1.71.0-iimpi-2019b.eb
SUCCESS Boost.Python-1.72.0-gompi-2020a.eb
SUCCESS Boost.Python-1.72.0-iimpi-2020a.eb

Build succeeded for 13 out of 13 (13 easyconfigs in total)
node2603.swalot.os - Linux centos linux 7.9.2009, x86_64, Intel(R) Xeon(R) CPU E5-2660 v3 @ 2.60GHz (haswell), Python 3.6.8
See https://gist.github.com/e7aa10bcda07803253da552b2f0de3fa for a full test report.

easybuild/easyblocks/b/boost.py

…gren/easybuild-easyblocks into 20210602164441_new_pr_jgYrgPRJSF

boegel · 2021-06-30T06:59:54Z

Test report by @boegel

Overview of tested easyconfigs (in order)

SUCCESS Boost-1.66.0-foss-2018a.eb
SUCCESS Boost-1.72.0-iimpi-2020a.eb

Build succeeded for 2 out of 2 (2 easyconfigs in total)
node3102.skitty.os - Linux centos linux 7.9.2009, x86_64, Intel(R) Xeon(R) Gold 6140 CPU @ 2.30GHz, Python 3.6.8
See https://gist.github.com/94e3b1ad121dc67294c7614a92d09708 for a full test report.

boegel

lgtm

Followup to easybuilders#2456 This now saves some work by building Boost.MPI with the main build and installing directly into the final folder without an intermediate copy step Also a bit of cleanup, fixing comments, ...

Update boost.py to always build single and multi threaded versions.

663949d

akesandgren added bug fix enhancement labels Jun 2, 2021

akesandgren added this to the next release (4.4.1) milestone Jun 2, 2021

Fix long lines

5f7c823

akesandgren mentioned this pull request Jun 2, 2021

Boost build broken by looking for a suffix which does not exist #2376

Open

Flamefire reviewed Jun 2, 2021

View reviewed changes

Flamefire reviewed Jun 3, 2021

View reviewed changes

akesandgren added 2 commits June 3, 2021 12:28

Simplify code that makes links for default libs.

6d426ab

Drop unused init of variable.

13adbf1

Flamefire reviewed Jun 3, 2021

View reviewed changes

easybuild/easyblocks/b/boost.py Outdated Show resolved Hide resolved

akesandgren and others added 4 commits June 3, 2021 13:22

Add deprecation warning for boost_multi_thread

8d5b7f0

Update easybuild/easyblocks/b/boost.py

e319b38

Simplyfy lib globbing and replacement for default libs Co-authored-by: Alexander Grund <Flamefire@users.noreply.github.com>

Fix check for boost_multi_thread being used, drop import of print_war…

abf1bc2

…ning.

Add simplification of globbing/replacement for default libs suggested…

111036a

… by Alexander Grund <Flamefire@users.noreply.github.com>

akesandgren added 2 commits June 4, 2021 08:37

Add separate tagged_layout and single_threaded options. Adjust code t…

074f88b

…o only add default links when appropriate.

Fix long line.

14bf92b

Flamefire reviewed Jun 4, 2021

View reviewed changes

easybuild/easyblocks/b/boost.py Show resolved Hide resolved

Flamefire previously approved these changes Jun 4, 2021

View reviewed changes

akesandgren dismissed Flamefire’s stale review via c9d6181 June 14, 2021 14:31

Flamefire previously approved these changes Jun 21, 2021

View reviewed changes

akesandgren mentioned this pull request Jun 21, 2021

update CMakeMake to handle old and new Boost/Boost.Python builds using custom easyblock for Boost #2461

Merged

verdurin mentioned this pull request Jun 22, 2021

{vis}[GCC/10.3.0] GIMP v2.10.24 easybuilders/easybuild-easyconfigs#13216

Merged

boegel reviewed Jun 29, 2021

View reviewed changes

Flamefire reviewed Jun 29, 2021

View reviewed changes

easybuild/easyblocks/b/boost.py Outdated Show resolved Hide resolved

minor code style tweak for composed build/install command in Boost ea…

1d7e48c

…syblock

boegel dismissed Flamefire’s stale review via 1d7e48c June 29, 2021 19:22

improve deprecation message for boost_multi_thread

59eb54a

Co-authored-by: Alexander Grund <Flamefire@users.noreply.github.com>

akesandgren commented Jun 30, 2021

View reviewed changes

easybuild/easyblocks/b/boost.py Outdated Show resolved Hide resolved

boegel added 2 commits June 30, 2021 08:25

use os.path.join for bjam build/install command

c4908e0

Merge branch '20210602164441_new_pr_jgYrgPRJSF' of github.com:akesand…

414231b

…gren/easybuild-easyblocks into 20210602164441_new_pr_jgYrgPRJSF

boegel mentioned this pull request Jun 30, 2021

update Boost 1.74.0 easyconfigs to use tagged_layout rather than boost_multi_thread easybuilders/easybuild-easyconfigs#13300

Merged

boegel approved these changes Jun 30, 2021

View reviewed changes

boegel merged commit 2df97f0 into easybuilders:develop Jun 30, 2021

akesandgren deleted the 20210602164441_new_pr_jgYrgPRJSF branch June 30, 2021 11:04

boegel mentioned this pull request Jul 4, 2021

build step fails for manta-1.6.0-GCC-10.2.0-Python-2.7.18.eb easybuilders/easybuild-easyconfigs#13337

Closed

Flamefire mentioned this pull request Jul 14, 2021

Simplify Boost easyblock #2513

Merged

update custom easyblock for Boost to always build single and multi threaded versions #2456

update custom easyblock for Boost to always build single and multi threaded versions #2456

Conversation

akesandgren commented Jun 2, 2021 • edited Loading

akesandgren commented Jun 2, 2021

akesandgren commented Jun 2, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

akesandgren Jun 3, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

akesandgren commented Jun 3, 2021

akesandgren commented Jun 3, 2021 • edited Loading

Flamefire commented Jun 3, 2021 • edited Loading

akesandgren commented Jun 3, 2021

akesandgren commented Jun 3, 2021 • edited Loading

Flamefire commented Jun 3, 2021

akesandgren commented Jun 3, 2021 • edited Loading

akesandgren commented Jun 4, 2021

akesandgren commented Jun 4, 2021

verdurin commented Jun 21, 2021

Flamefire commented Jun 21, 2021

Flamefire commented Jun 21, 2021

Overview of tested easyconfigs (in order)

SebastianAchilles commented Jun 21, 2021

Overview of tested easyconfigs (in order)

SebastianAchilles commented Jun 21, 2021

Overview of tested easyconfigs (in order)

verdurin commented Jun 21, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

boegel commented Jun 29, 2021

Overview of tested easyconfigs (in order)

boegel commented Jun 29, 2021

Overview of tested easyconfigs (in order)

boegel commented Jun 30, 2021

Overview of tested easyconfigs (in order)

boegel left a comment

Choose a reason for hiding this comment

akesandgren commented Jun 2, 2021 •

edited

Loading

akesandgren Jun 3, 2021 •

edited

Loading

akesandgren commented Jun 3, 2021 •

edited

Loading

Flamefire commented Jun 3, 2021 •

edited

Loading

akesandgren commented Jun 3, 2021 •

edited

Loading

akesandgren commented Jun 3, 2021 •

edited

Loading