Sync meeting on EESSI test suite (2024 09 19)

Planning

• every 2 weeks on Thursday at 14:00 CE(S)T
• next meetings:
  • 17 October 14:00-15:00
  • 31 October 14:00-15:00

---

Meeting (2024-09-19)

Attending: Sam, Lara, Caspar

• Merged PRs

  • mpi4py tutorial (corresponding to the docs) #149,#172
  • Remove explicitely setting account (-A something) in the access field of the configs. Can (and probably should) be done through the environment variable SBATCH_ACCOUNT before running the test suite #171
  • Azure ReFrame config & CI config #173, #176
  • Add hook for measuring memory usage in jobs that set cgroups (v1 or v2, both supported) #174
  • Bump (default) ReFrame version used in daily runs #175
  • Show ReFrame warnings in yellow for better visibility #178
  • Limit runtime if ReFrame command in daily runs, so daily runs don't overlap #179
  • Specify how to request memory (i.e. set it as a resources field) in configs where this was still missing. Without it, you might still run into OOM issues #180
  • Make req_memory_per_node hook print a warning if the memory resources is not configured in the ReFrame config #182
  • Fix CI so that it works with ReFrame 4.6.2, which (intentionally) doesn't set PYTHONPATH for ReFrame's dependencies #183
  • Update Scorecards version to fix failing action in CI #184

• Open PRs

  • MetalWalls:

    • Crivella implemented changes (limiting to 256 cores), needs another review now
      • A quick look it now prints an attribute error, since his skip_if uses an unknown attribute in the message. Gave him a pointer
    • Blocked by https://github.com/reframe-hpc/reframe/pull/3233 still being open

  • Mixin class:

    • First attempt, using lammps as testing grounds. Moved hooks (partially) to mixin class (WIP).
    • Some great feedback from Sam, Caspar is trying to find time to have another look and process it
    • We can probably set some class variables in the child class, and use those in the init method of the mixin class as a way to still be able to use things set in the child in the initialization
      • E.g. child class sets module_name_regex, mixin class uses that to create a parameter(find_modules(module_name_regex)? Then, the child class is really 'config-only'

• Docs are merged now and available at https://www.eessi.io/docs/test-suite/writing-portable-tests/

  • Will probably change very soon again, once we have the Mixin class. Could still keep those docs as a reference for very non-standard tests, where inheriting from the Mixin class is simply impossible?

• Apply memory limits using memory hook for all tests

  • Actually: use the mixin class for all tests first. Then, one will be forced to specify a memory request (mixin class prints an error otherwise)
  • Caspar will go through the tests and update them where needed
  • Much easier to do now with the memory hook from #174

• OpenFOAM test

  • Satish has a test that works, but no ReFrame test => No progress
    • Switching to different test case, inspired by ExaFOAM project (lid-driven cavity case).
    • The Slurm batch jobs are working. Output needs to be checked but OpenFOAM is not accepting the file write frequency for me to check. It prints only the first time step which is correct.

• Xin has made a start with the autogenerated API docs

  • TODO: make a CI test that tries to do the autogeneration to make sure it still works

• TODO Caspar: current reporting of the test step is very limited, only the total number of tests. Can we print more? (which are run, which fail)

• Are there other tests we thing we need?

  • More tests for solvers from e.g. the PETSc library, Trilinos, etc. (Satish)
  • HPL test, Caspar asked for someone internally at SURF (Stefan) to see if he could port ours, using the new documentation we have in place.
  • Other low level tests: IOR, HPL, Stream, HPCG
  • WRF (high level application)