Skip to content

Conference call notes 20230816

Jump to bottom
Kenneth Hoste edited this page Sep 27, 2023 · 14 revisions

(back to Conference calls)

Notes on the 227th EasyBuild conference call, Wednesday 16 Aug 2023 (08:00 UTC)

Attendees

List of attendees (9):

  • Sebastian Achilles (JSC, Germany)
  • Alexander Grund (TU Dresden, Germany)
  • Thomas Hayward-Schneider (Max Planck Institute for Plasma Physics, Germany)
  • Thomas Hoffman (EMBL, Heidelberg, Germany)
  • Kenneth Hoste (HPC-UGent, Belgium)
  • Adam Huffman (Big Data Institute, Oxford, UK)
  • Kurt Lust (UAntwerpen, Belgium + LUMI User Support Team)
  • Lara Peeters (HPC-UGent, Belgium)
  • Alain van Hoof (TU Eindhoven, Netherlands)

Agenda

  • overview of recent developments
  • Q&A

Recent developments

  • latest EasyBuild release: 4.8.0 (7 July 2023)
    • ETA for next EasyBuild release: end of Aug'23
  • recent changes
    • docs (merged PRs)
      • ...
    • framework (merged PRs)
      • bug fixes
        • fix test_add_and_remove_module_path by replacing string comparison of paths by checking whether they point to the same path (since symlinks may cause trouble) (PR #4312)
        • use sys.executable to obtain path to python command, rather than assuming that python command is available in $PATH (PR #4309)
        • ignore request for external module (meta)data when no modules tool is active (PR #4308)
        • enhance Toolchain.get_flag to handle lists (PR #4319)
      • enhancements
        • add build_and_install_loop hooks to run before and after the install loop for individual easyconfigs (PR #4304)
        • implement hooks for failure scenarios: crash_hook, cancel_hook, fail_hook (PR #4315)
          • remove crash hook (since it doesn't work) (PR #4325)
        • add postiter hook to the list of steps so the corresponding hook can be used (PR #4316)
        • add build_info_msg easyconfig parameter to print message during installation of an easyconfig (PR #4324)
        • add run_shell_cmd hook (PR #4323)
      • changes
        • ...
      • EasyBuild 5.0 (to 5.0.x branch)
        • initial implementation of run function to replace run_cmd + run_cmd_qa (PR #4284)
        • copy load_source function from easybuild.tools.py2vs3.py3 to easybuild.tools.filetools so it can be used in easybuild.tools.hooks (PR #4306)
        • disable trace output for commands run in get_source_tarball_from_git (PR #4310)
        • fix broken github tests in options test.framework.options by disabling trace output in download_repo and fetch_files_from_pr functions (PR #4311)
        • fix for test_get_source_tarball_from_git broken by trace output (PR #4310)
        • fix broken github tests in options test.framework.options by disabling trace output in download_repo and fetch_files_from_pr functions (PR #4311)
        • implement fail_on_error/in_dry_run/output_file options + enable caching for run function, and switch from run_cmd to run function in systemtools (PR #4314)
        • implement support in run function for splitting stdout and stderr output (split_stderr) + passing down environment to use, and switch to run function for running module commands (PR #4321)
        • also include command used + working directory in return value of run function (PR #4322)
    • easyblocks (merged PRs)
      • bug fixes
        • don't add MATLAB libraries to $LD_LIBRARY_PATH (PR #2981, fixes issue #2965)
        • conditionally add -Wno-unused-command-line-argument to $CFLAGS to fix error when installing imkl-FFTW with RPATH (PR #2975)
      • enhancements
        • add support to PerlModule easyblock to customize prefix option used in installation command via prefix_opt (PR #2979)
      • updates
        • update OpenMPI easyblock to fix sanity check for Clang-based compilers (PR #2774)
        • update Clang easyblock for versions >= 16 + run tests only for final stage of bootstrap build (PR #2929)
        • update LLVM easyblock for LLVM v16: symlink third-party to third-party-<version>.src (PR #2970)
        • update scipy easyblock for scipy >= 1.11.0 (PR #2971) + don't run scipy test suite in parallel for scipy >= 1.11 (PR #2980)
        • update sanity check for Mesa >= 22.3 (PR #2973)
        • update sanity check for OpenFOAM 11 (PR #2978)
      • changes
        • ...
      • new easyblocks
        • add custom easyblock for sympy to fix issue in test step when tmpdir is a symlink (PR #2949)
      • EasyBuild 5.0 (to 5.0.x branch)
        • ...
    • easyconfigs (merged PRs)
      • ~100 easyconfig PRs were merged since last conf call
      • bug fixes
        • add/fix patches for PyTorch 1.13.1 w/ foss/2022a (PR #18371)
        • add patch to fix CUDA race condition problem for Score-P v8.0 (PR #18411)
        • fix source URL for PyTables v3.5.2 (PR #18429)
        • add missing patches for PyTorch 1.12.0 w/ foss/2022a (PR #18430)
        • fix installation of Python bindings for Arrow by adding --no-build-isolation to pip install command (PR #18440 + PR #18441)
        • add patch for LIBSVM 3.30 to pass $CFLAGS + $LDFLAGS and run tests (PR #18454)
        • copy missing default_files directory for PICRUSt2 (PR #18459)
        • add patch for Automake 1.16.5 to fix help2man error (PR #18468)
        • add patch to fix quantization failure in PyTorch 1.11.0 on POWER (PR #18489)
        • add missing patches for PyTorch 1.12.0 w/ foss/2022a + CUDA 11.7.0 (PR #18491)
        • add patches to fix PyTorch 1.13.1 w/ foss/2022a on POWER + fix flaky test_jit_legacy test (PR #18500)
      • enhancements
        • ...
      • (noteworthy) new software
      • noteworthy software updates
      • changes
        • use custom easyblock for sympy to prevent test failures (PR #18428)
        • use GCC/12.2.0 toolchain for GSL 2.7, remove easyconfig for GSL 2.7 with GCCcore/12.2.0 (PR #18501)
        • use GCC/12.3.0 rather than GCCcore/12.3.0 for GSL v2.7 (PR #18502)
  • work in progress
    • docs (open PRs + issues)
    • framework (open PRs + issues)
      • reported bugs / bug fixes
        • improve error when checksum dict has no entry for a file (PR #4150)
        • fix error message when checksums is set to [None] (PR #4261)
        • remove use of Python3 FileExistsError in mkdir function (PR #4328)
          • this basically means that EasyBuild is hard broken with Python 2 in develop branch
          • we didn't catch this in CI because we're no longer testing with Python 2 (cfr. PR #4267)
          • looking into resuming running tests with Python 2 via CentOS 7.9 container...
        • add usedforsecurity=False to usage of md5 for FIPS mode (issue #4317)
        • avoid hardcoding /bin/bash (issue #4320)
      • enhancements
        • add end-to-end test for running EasyBuild in different Linux distros using containers (PR #3968)
        • add --silence-hook-trigger configuration option to supress printing of debug message every time a hook is triggered (PR #4329)
      • changes
        • ...
      • EasyBuild 5.0 (to 5.0.x branch)
        • implement support in 'run' function for running command in different working directory + switch to run function in filetools (PR #4327)
        • work-in-progress (no PRs yet)
          • improved error reporting by run function (now just raises subprocess.CalledProcessError)
          • sync with develop after introduction of run_shell_cmd hook in run_cmd
          • rename run to run_shell_cmd
          • should shell option for run function be renamed to use_bash?
    • easyblocks (open PRs + issues)
      • bug reports/fixes
        • enhance PythonPackage easyblock to deal with posix_local installation scheme used by Python in recent Debian/Ubuntu versions (PR #2977)
          • fix for: Python packages are being installed to local/lib/python*/dist-packages due to tweaks to sysconfig.py in OS (Ubuntu)(issue #2976)
      • enhancements
        • improve cpu-arch optimization settings for OpenCV (PR #2954)
        • add options to MesonNinja easyblock to customize build_cmd, install_cmd, builddir (PR #2963)
        • lld is missing for Clang < 14.0 even when build_lld is enabled (PR #2974)
      • updates
        • update CP2K easyblock for compatibility with CP2K v2023.1 release (GCC only) + fix GPU support (PR #2918)
        • update MXNet easyblock + don't try to install R extension by default (PR #2955)
        • update TensorFlow easyblock for v2.13 since LMDB is no longer a dependency (PR #2982)
        • enhance PyTorch easyblock to print individual failed tests (PR #2983)
      • new easyblocks
        • add generic CargoPythonBundle easyblock (PR #2964)
      • changes
        • ...
      • EasyBuild 5.0 (to 5.0.x branch)
        • ...
    • easyconfigs (open PRs + issues)
      • bug fixes/reports
        • fix spelling of BOOL values in CMake configopts (PR #18484)
        • add alternative checksum for Extrae v4.0.4 (PR #18564)
        • add patches to fix PyTorch-1.12.1 w/ foss/2021a + CUDA v11.3.1 on POWER (PR #18492)
        • add patches to fix PyTorch 1.12 (CPU-only) w/ foss/2022a on POWER (PR #18490)
        • add patches to fix PyTorch-1.12.1 w/ foss/2022a + CUDA v11.7.0 on POWER (PR #18494)
        • add patches to fix PyTorch-1.12.1 w/ foss/2021b + CUDA v11.5.2 on POWER (PR #18493)
      • enhancements
        • don't disable building of third and fourth derivates in libxc 6.1.0 (PR #18467)
        • run ESPResSo tests in parallel (PR #18517)
      • new software
        • ...
      • noteworthy software updates
      • changes
        • ...

Q&A

  • cctbx-base in AlphaPulldown PR #18506
    • see Thomas' suggestion w.r.t. failing tests + avoiding download during installation (draft PR with his easyconfig file coming up)
  • RFdiffusion PR #18359
    • how to deal with models?
    • should EasyBuild framework have concept of data files?
Clone this wiki locally