Skip to content

Conference call notes 20240925

Jump to bottom
Bart Oldeman edited this page Oct 23, 2024 · 2 revisions

(back to Conference calls)

Notes on the 255th EasyBuild conference call, Wednesday 25 Sept 2024 (15:00 UTC)

Attendees

List of attendees (21):

  • Simon Branford (Univ. of Birmingham, UK)
  • Bibek Chapagain (Barcelona Supercomputing Centre, Spain)
  • David Coster (Max Planck Institute for Plasma Physics, Germany)
  • Alex Domingo (Vrije Universiteit Brussel, Belgium)
  • Em Dragowsky (Case Western Reserve University, Ohio, US)
  • Davide Grassano (CECAM, Switzerland)
  • Jasper Grimm (University of York, UK)
  • Leonardo Honfi Camilo (Univ. of Wageningen, Netherlands)
  • Kenneth Hoste (HPC-UGent, Belgium)
  • Chia-Jung Hsu (Chalmers University of Technology, Sweden)
  • Adam Huffman (Big Data Institute, Oxford, UK)
  • Georgios Kafanas (University of Luxembourg)
  • Kurt Lust (UAntwerpen, Belgium + LUMI User Support Team)
  • Sam Moors (Vrije Universiteit Brussel, Belgium)
  • Sébastien Moretti (SIB, Switzerland)
  • Mikael Öhman (Chalmers University of Technology, Sweden)
  • Jan Reuter (JSC, Germany)
  • Jörg Saßmannshausen (Imperial College London, UK)
  • Åke Sandgren (Umeå University, Sweden)
  • Alain van Hoof (TU Eindhoven, Netherlands
  • Cintia Willemyns (Vrije Universiteit Brussel, Belgium)

Agenda

  • overview of recent developments
  • outlook to EasyBuild 5.0 release
  • 2024a/2024b update of common toolchains
  • Q&A

Recent developments

  • latest EasyBuild release: 4.9.4 (22 Sept 2024)
  • next EasyBuild release:
    • very likely EasyBuild v5.0.0 🔥
  • EasyBuild v5.0.0
    • detailed notes on latest developments in https://hackmd.io/tZtz6vMmRfGL6N9Rf__OEA
    • to test development version of EasyBuild 5.0: 
      # set up Python virtual environment, and jump into it
python3 -m venv eb5
source eb5/bin/activate

# install EasyBuild 5.0 development version into it
pip install https://github.com/easybuilders/easybuild-framework/archive/5.0.x.tar.gz
pip install https://github.com/easybuilders/easybuild-easyblocks/archive/5.0.x.tar.gz
pip install https://github.com/easybuilders/easybuild-easyconfigs/archive/5.0.x.tar.gz

# go!
eb --version
  • very soon (today?) the 5.0.x branches will be collapsed into develop branch
    • additional EasyBuild 4.9.x versions could still be done via 4.9.x branches, but none are planned
    • impact:
      • --from-pr will be broken with EasyBuild 4.x due to major version mismatch
        • --from-commit can be used as a workaround
      • open easyconfig PRs should be synced with develop branch before they get merged
        • due to changes in easyconfigs test suite implemented in 5.0.x
        • for example use_pip = True should no longer be used
        • same goes for framework & easyblocks

Merged PRs

  • docs (merged PRs)

    • ...

  • framework (merged PRs)

    • bug fixes
      • fix test_github_det_commit_status by using more recent commits (PR #4636)
      • implement workaround for permission error when copying read-only files that have extended attributes set and using Python 3.6 (PR #4642)
      • take into account alternate sysroot for /bin/bash used by run_cmd (PR #4646)
      • fix findPythonDeps.py script for cyclic dependencies (PR #4633)
    • enhancements
      • set $LMOD_TERSE_DECORATIONS to 'no' to avoid additional info in output produced by 'ml --terse avail' (PR #4648)
    • changes
      • use more granular exit codes when EasyBuildError is raised (PR #4534)
      • Remove trailing dots from backup message produced by --inject-checksums (PR #4632)
      • Drop support for pep8 package (was used for --check-contrib + --check-style) (PR #4634)
    • code cleanup
      • ...
    • EasyBuild 5.0 (separate 5.0.x branch)
      • sync with develop (20240923) (PR #4654)

  • easyblocks (merged PRs)

    • bug fixes
      • ignore Python from virtualenvs in GROMACS configure via -DPython3_FIND_VIRTUALENV=STANDARD (PR #3283)
        • this should (also) be done on CMakeMake, available since CMake 3.15?
      • don't wipe build environment before building opam in install step of OCaml easyblock (PR #3443)
      • copy EasyConfig instance in constructor of Bundle and Cargo easyblocks before making changes to it (PR #3448)
      • fix crash in GCC easyblock when cuda-compute-capabilities EasyBuild configuration option is not set (PR #3449)
    • enhancements
      • enhance OpenFOAM easyblock to also build the plugins for OpenFOAM >= v2406 (PR #3436)
      • allow Python bundles and packages to specify a maximum Python version for the system toolchain (PR #3431)
      • enhance binutils easyblock to explicitely pass msgpack configure option (PR #3438)
      • enhance custom easyblock for NCCL: add licence to NCCL installation (PR #3451)
      • Adding RISCV support to build Qt5 (PR #3462)
    • updates
      • update sanity check in OpenFOAM easyblock, since there's no modifyMesh in OpenFOAM 12 (PR #3434)
      • fix sanity check for Geant4 >= v11.2 (PR #3439)
    • changes
      • ...
    • new easyblocks
      • ...
    • code cleanup
      • ...
    • EasyBuild 5.0 (separate 5.0.x branch)
      • sync with develop (20240923) (PR #3459)

  • easyconfigs (merged PRs)

    • ~100 easyconfig PRs were merged since last conf call
    • bug fixes/reports
      • add alternative checksum for bold 1.3.0 extension in R-bundle-CRAN (PR #21370)
      • account for crates for easyconfigs using Cargo-based easyblock when determining checksums for patches (PR #21419)
      • fix build of librosa 0.10.1 in some environments by removing python -m build for soxr extension (PR #21434)
      • Fix repeated sanity check runs in manta (PR #21435)
      • change CREST version to 3.0pre-20240319 to indicate it is a pre-release (PR #21439)
      • avoid writing into ~/.stack directory during build for git-annex (PR #21452)
      • use -lncurses rather than -ltermcap for Term::ReadLine::GNU Perl extension (PR #21469)
    • enhancements
      • enable support for Apache ORC to Arrow v14.0.1 and v16.1.0 (PR #21056)
      • also build + install the plugins with OpenFOAM v2406 (PR #21332)
      • add seqPattern, genomation and ChIPseeker to R-bundle-Bioconductor v3.18 (PR #21474)
    • (noteworthy) new software
    • noteworthy software updates
      • ...
    • changes
      • use proper dependency for tensorboard in easyconfigs for TensorFlow v2.15.1 (PR #21337)
    • EasyBuild 5.0 (separate 5.0.x branch)
      • ...

Open (active) PRs

  • docs (open PRs + issues)

    • ...

  • framework (open PRs + issues)

    • bug fixes
      • Reverse order for parsing files in $XDG_CONFIG_DIRS (PR #4630)
    • enhancements
      • prepend to $PYTHONPATH or $EBPYTHONPREFIXES in generated module files by automatically scanning for python site package directories (PR #4539)
      • Copy build log and artifacts to a permanent location after failures (WIP PR #4601)
      • improve error message when missing commit message for --new-pr (PR #4640)
      • Add support for GCC optimized builds in RISCV SiFive u74mc architecture (PR #4641)
      • allow extensions to add text to the module file (PR #4652)
      • Ignore "ambiguous variable name" warning for EasyConfig style check (PR #4644)
      • Problem using $CPATH in modulefiles overwriting system paths (issue #3331)
        • add option search-path-cpp-headers to control how EasyBuild sets paths to headers at build time (PR #4645)
        • refactor generation of required environment variables in module files (PR #4653)
        • add option module-search-path-headers to control how modules set search paths to header files (PR #4655)
    • changes
      • Enable module-depends-on by default (PR #4500)
      • Rename source step (issue #4627)
      • Replace parallel easyconfig parameter by maxparallel (PR #4398)
      • Deprecate post_install_step, rename it to post_processing_step (issue #4656)
    • EasyBuild 5.0 (separate 5.0.x branch)

  • easyblocks (open PRs + issues)

    • bug fixes
      • In kokkos build in LAMMPS - use icx when detecting icelake (PR #3321)
      • run pip check only once for PythonBundle (PR #3432)
      • Make CMakeMake respect the toolchainopts when selecting build type (PR #3452)
        • to fix: CMakeMake could default CMAKE_BUILD_TYPE to RelWithDebInfo instead of Release (issue #3032)
      • Make MesonNinja respect the toolchainopts with buildtype as well as --debug and --optimization flags (PR #3454)
      • Avoid failure in Python sanity check when PIP_REQUIRE_VIRTUALENV is set (PR #3460)
      • Cargo easyblock produces confusing error message when same crate is listed twice (issue #3461)
    • enhancements
      • Use prefer_ebpythonprefixes flag to use EBPYTHONPREFIXES (PR #3343)
      • Set CMake hints for path to Python installation if it's direct or indirect dependency (when using CMake >= 3.12) (PR #3282)
      • enhance LLVM easyblock for compilation of clang/flang + other llvm-projects (PR #3373)
      • enhance CMakeMake easyblock to check whether correct Python installation was picked up by CMake (PR #3399)
      • Update NCL easyblock for intel (PR #3409)
      • ignore user python packages by using -s option when running Python code via "python -c" (PR #3440)
      • also install w90spn2spn.x + w90chk2chk.x for Wannier90 3.1.0 (PR #21383)
    • updates
      • update custom easyblock for CP2K >= v2024 (PR #3433)
      • update AOCC easyblock to work with LLVM 16 and above (PR #3437)
      • enhance AOCC EasyBlock to work with LLVM 16 and newer (PR #3458)
    • changes
      • ...
    • new
      • custom easyblock for DeepSpeed (PR #3450)
    • EasyBuild 5.0 (separate 5.0.x branch)
      • ...

  • easyconfigs (open PRs + issues)

    • bug fixes/reports
      • switch to CargoPythonBundle easyblock for orjson 3.9.15 (PR #21392)
    • enhancements
      • ignore user-installed packages when running python in sanitycheck (PR #21362)
    • (noteworthy) new software
    • software updates
    • changes
      • ...
    • EasyBuild 5.0 (separate 5.0.x branch)
      • ...

2024b common toolchains

  • GCC 14.2 as a base (see easyconfigs PR #21114)
  • we should define candidate toolchains using latest version of all components
    • are newer versions of Intel components compared to intel/2024a already available?
  • should we consider replacing intel with Clang-based toolchain as 2nd common toolchain next to foss?
    • Fortran frontend in Clang is becoming mature enough?

Q&A / others

  • Will EasyBuild 5.0 require a rebuild of all existing software installations?
    • No.
    • EasyBuild 5.0 will have RPATH linking enabled by default, but doing that on top of a software stack that isn't should work fine 99% of the time, and it's easy to disable RPATH linking again
  • installing ORCA in Rocky container
    • working fine, incl. test runs in ORCA v6.0
    • --sanity-check-only in Rocky container fails on those test runs
    • is actually a bug in the easyblock, it uses self.builddir in sanity_check_step, it shouldn't
Clone this wiki locally