use reframe features + fixes for various issues

[smoors]

• fixes Improve GROMACS test by using new system features support in ReFrame #10
• fixes add support for filtering based on module name #7
• fixes add support for overriding core count to use for running tests #8
• fixes add support for manually filtering valid_systems #13
• fixes sub-optimal number of openmp threads #14
• fixes add support for setting custom environment variables in test jobs #15
• fixes properly handle gpus-per-node #19

changes:

• filter out CUDA modules in non-GPU nodes and support for filtering out non-CUDA modules in GPU nodes
  • requires adding features cpu and/or gpu to the partitions in the site config file
• support for specifying modules
  • via --setvar modules=<modulename>
• support for specifying systems:partitions
  • via --setvar valid_systems=<comma-separated-list>
• support for overriding tasks, cpus, gpus via:
  • --setvar num_tasks_per_node=<x>
  • --setvar num_cpus_per_task=<y>
  • --setvar num_gpus_per_node=<z>
• support for setting additional environment variables
  • via --setvar env_vars=<envar>:<value>

[casparvl]

I tested this (but only the default behaviour, no command line overriding yet). One thing I noticed is that the default behavior is different from the master branch. Given two GROMACS modules, GROMACS/2021.6-foss-2022a and GROMACS/2021.6-foss-2022a-CUDA-11.7.0, master branch generates (and runs) the following tests:

1. GROMACS/2021.6-foss-2022a-CUDA-11.7.0, with nb_impl=gpu on a GPU partition
2. GROMACS/2021.6-foss-2022a-CUDA-11.7.0, with nb_impl=cpu on a GPU partition
3. GROMACS/2021.6-foss-2022a, with nb_impl=cpu on a GPU partition
4. GROMACS/2021.6-foss-2022a, with nb_impl=cpu on a CPU partition

This PR only runs combinations 1 & 4 by default.

Of course, we might wonder if running 2 & 3 are particularly useful. I'd argue yes, for the following reasons:

• If one mounts EESSI on a GPU node, the GROMACS/2021.6-foss-2022a is available. Therefore, a user should just be able to assume it's tested and working.
• There may be good reasons to run certain use cases without the GPU, even if you have a GPU in your system (or at least, I wouldn't want to assume those cases don't exist) - and the end user might use either of the two modules to do so. Therefore, the CPU implementation should be tested with both modules, even if it is a GPU partition.

It's easy enough to adapt this PR to that same default behavior, but curious to know if you agree with my reasoning above :)

===

Edit: I see that it actually runs 1, 3, and 4 by default. I was confused by the listing, which only showed two checks:

- GROMACS_EESSI %benchmark_info=HECBioSim/hEGFRDimer %nb_impl=gpu %scale=('singlenode', 1) %module_name=GROMACS/2021.6-foss-2022a-CUDA-11.7.0 /54be990c
- GROMACS_EESSI %benchmark_info=HECBioSim/hEGFRDimer %nb_impl=cpu %scale=('singlenode', 1) %module_name=GROMACS/2021.6-foss-2022a /d12ed623

But of course the checks for the individual partitions are only generated later, so when I run, I get:

[ RUN      ] GROMACS_EESSI %benchmark_info=HECBioSim/hEGFRDimer %nb_impl=gpu %scale=('singlenode', 1) %module_name=GROMACS/2021.6-foss-2022a-CUDA-11.7.0 /54be990c @snellius:gpu_test+builtin
[ RUN      ] GROMACS_EESSI %benchmark_info=HECBioSim/hEGFRDimer %nb_impl=cpu %scale=('singlenode', 1) %module_name=GROMACS/2021.6-foss-2022a /d12ed623 @snellius:thin+builtin
[ RUN      ] GROMACS_EESSI %benchmark_info=HECBioSim/hEGFRDimer %nb_impl=cpu %scale=('singlenode', 1) %module_name=GROMACS/2021.6-foss-2022a /d12ed623 @snellius:gpu_test+builtin

[casparvl]

Since we also discussed shortly on how to reduce the number of GPUs used in the tests: I do indeed think that this should be done in the ReFrame config file. Simply specify a virtual ReFrame partition that only has 1 GPU per node. E.g. we have GPU nodes with 4 GPUs per node. I would then specify e.g.:

'name': 'gpu_exclusive',
...
'access':  ['-p gpu_test --exclusive'],
...
'processor': {
                        'num_cpus': 72,
                        'num_sockets': 2,
                        'num_cpus_per_socket': 36,
                        'arch': 'icelake',
                    },
'devices': [
                        {
                            'type': 'gpu',
                            'num_devices': 4,
                        }
                    ],

if I wanted the test to use the full GPU nodes, or

'name': 'gpu_test_quarter',
...
'access':  ['-p gpu_test --gpus-per-node 1'],
...
'processor': {
                        'num_cpus': 18,
                        'num_sockets': 1,
                        'num_cpus_per_socket': 18,
                        'arch': 'icelake',
                    },
'devices': [
                        {
                            'type': 'gpu',
                            'num_devices': 1,
                        }
                    ],

to 'fake' a partition with smaller nodes. In this example, one 'node' in this virtual partition would simply be a quarter of our real node. I tested this approach for single node tests, and there it works fine.

I think this approach will even work for multinode tests, but I cannot test it: our system only allows partial node allocation for single node jobs, multinode jobs have to be exclusive.

[casparvl]

Ok, so I also tested the --setvar options, and all seems to work as expected. Specifically, one of the tests I ran was

reframe --config-file settings_snellius.py --checkpath test-suite/eessi/reframe/eessi-checks/ -r -R -t CI -t singlenode --setvar modules=GROMACS/2021.6-foss-2022a --setvar num_tasks_per_node=16 --setvar num_cpus_per_task=2

That generated the job:

#!/bin/bash
#SBATCH --job-name="rfm_job"
#SBATCH --ntasks=16
#SBATCH --ntasks-per-node=16
#SBATCH --cpus-per-task=2
#SBATCH --output=rfm_job.out
#SBATCH --error=rfm_job.err
#SBATCH --time=0:30:0
#SBATCH -p gpu_test --exclusive
module load GROMACS/2021.6-foss-2022a
export OMP_NUM_THREADS=2
curl -LJO https://github.com/victorusu/GROMACS_Benchmark_Suite/raw/1.0.0/HECBioSim/hEGFRDimer/benchmark.tpr
mpirun -np 16 gmx_mpi mdrun -nb cpu -s benchmark.tpr -dlb yes -ntomp 2 -npme -1

Which looks fine.

[smoors]

thanks for the review

this is indeed a change I forgot to discuss with you
I agree that we should ideally also test non-gpu jobs on gpu nodes

on the other hand, users (hopefully) rarely do this (unless for benchmarking), and we actively discourage it at VUB
also, our GPU nodes are in high demand, so I don’t want to test this use case too often

I changed the code so you can now choose the behavior per partition in the config file, see 0f77a7a
with this change:

• if you want to test non-gpu jobs on gpu nodes, add both features 'cpu' and 'gpu' to the GPU partition
• if yo only want to test gpu jobs on gpu nodes, add only 'gpu'

what do you think of this solution?

[smoors]

Since we also discussed shortly on how to reduce the number of GPUs used in the tests: I do indeed think that this should be done in the ReFrame config file.

yes sure. it would still be nice to also have a cmd line option for this, but that can be added later.

[casparvl]

if you want to test non-gpu jobs on gpu nodes, add both features 'cpu' and 'gpu' to the GPU partition
if yo only want to test gpu jobs on gpu nodes, add only 'gpu'
what do you think of this solution?

Yeah, makes a lot of sense to me. I like it if this behavior can be controlled from the config side, as it means you can 'flip' that switch in one go for all tests.

I've noticed I'm thinking very much about 'what do we want in the EESSI CI?' (probably: test everything), so it's super useful that you also provide the point of view of 'what would an end-user/sysadmin from an HPC system want?' (i.e. more limited testing to save resources) :)

I'll give this another try to check the latest changes

[casparvl]

I refactored the logic a bit for the set of default test combinations. As I did so, I found out I actually forgot one combination in my listing above: GROMACS/2021.6-foss-2022a-CUDA-11.7.0 with nb_impl=cpu on CPU nodes. For some software, a CUDA-enabled build means it has to have a GPU in order to run. Not so for GROMACS if we specify nb_impl=cpu - that's totally valid.

Why would we want to test this? It could be that for certain versions of GROMACS in the EESSI stack, we only have a CUDA-aware build (simply because no one bothered to do the non-CUDA-aware build). If a researchers wants to use that particular version, but only has access to a CPU machine, that's perfectly fine and should work. Therefore, it should be tested.

So, all in all, on a system with 2 modules (one cuda aware, one non-cuda aware), and 2 partitions (one with GPUs, one without), you'd get five tests by default:

[ RUN      ] GROMACS_EESSI %benchmark_info=HECBioSim/hEGFRDimer %nb_impl=gpu %scale=('singlenode', 1) %module_name=GROMACS/2021.6-foss-2022a-CUDA-11.7.0 /54be990c @snellius:gpu_test+builtin
[ RUN      ] GROMACS_EESSI %benchmark_info=HECBioSim/hEGFRDimer %nb_impl=cpu %scale=('singlenode', 1) %module_name=GROMACS/2021.6-foss-2022a /d12ed623 @snellius:thin+builtin
[ RUN      ] GROMACS_EESSI %benchmark_info=HECBioSim/hEGFRDimer %nb_impl=cpu %scale=('singlenode', 1) %module_name=GROMACS/2021.6-foss-2022a /d12ed623 @snellius:gpu_test+builtin
[ RUN      ] GROMACS_EESSI %benchmark_info=HECBioSim/hEGFRDimer %nb_impl=cpu %scale=('singlenode', 1) %module_name=GROMACS/2021.6-foss-2022a-CUDA-11.7.0 /a6f0d10c @snellius:thin+builtin
[ RUN      ] GROMACS_EESSI %benchmark_info=HECBioSim/hEGFRDimer %nb_impl=cpu %scale=('singlenode', 1) %module_name=GROMACS/2021.6-foss-2022a-CUDA-11.7.0 /a6f0d10c @snellius:gpu_test+builtin

I'll do a PR with my changed logic to your branch. I also stripped out the 'skip' hooks, as those are no longer needed now that we use the features for this :)

[smoors]

Oof, tricky. I'm a bit cautious with implementing more features for this. I'm sure we'll have quite a collection of features soon enough anyway, we should probably restrain ourselves a bit to keep things simple to understand.

yeah, you're probably right.

One could argue that the default behaviour is desireable, also on your system: the test for the CUDA module on your CPU partition will fail on your system, showing you that indeed, that module does not provide a functional GROMACS. Suppose that some end-user runs our test suite on your system, it would actually be good if they see this result.

i'm ok with it being the default, but I also want an easy way to filter those cases out without having to manually specify every combination that I do want to test.

i have another idea that could work. i'll first merge your PR into mine, and then add my idea on top.

[smoors]

@casparvl i've kept the logic that you proposed and added 3 optional variables that users can set on the cmd line: module_regex_select, module_regex_skip, and run_mode.

this ensures that all valid combinations are tested by default, and at the same time gives a lot of flexibility to run a subset of those valid combinations. i think this is a good strategy to follow.

for example, if you want to run only with non-bonded forces on the GPU (which implies only GPU partitions and CUDA modules), you can add:

--setvar run_mode=gpu

if you want to test only the CUDA modules, you can add:

--setvar module_regex_select=CUDA

if you want to run only in non-GPU partitions and only with non-CUDA modules, you can add (this works when both 'cpu' and 'gpu' features are set on the GPU partitions):

--setvar module_regex_skip=CUDA --setvar run_mode=cpu --setvar valid_systems='-gpu'

of course, if you don't have the 'cpu' feature set on GPU partitions, the last example becomes simpler:

--setvar module_regex_skip=CUDA

there are many other possibilities, such as selecting (or skipping) one or a set of toolchain generations e.g. --setvar module_regex_select='(2021b|2022a)' or a range --setvar module_regex_select='202[0-2][ab]' , and so on.

if you agree with this approach, i hope we can merge this soon. i will then work on moving as much logic as possible out of the gromacs test, for maximal reuse in other tests.

[smoors]

note that i chose a generic variable name for run_mode on purpose, so we don't have to define a new name for every new test (even it supports other run modes than just 'cpu' and 'gpu'). each test can then define what the possible run modes are.

[casparvl]

Unless I'm missing something, any of those selections can already be made with the -n and -x options of ReFrame. From the official docs

-n, --name=NAME
Filter tests by name.
NAME is interpreted as a Python Regular Expression; any test whose display name matches NAME will be selected. The display name of a test encodes also any parameterization information. See Test Naming Scheme for more details on how the tests are automatically named by the framework.
...

I.e. all parametrisation information gets included in the name of the test, and can thus be filtered on with -n and -x. Some examples:

reframe --config-file settings_snellius.py --checkpath test-suite/eessi/reframe/eessi-checks/ -r -R -t CI -t singlenode --setvar run_mode=cpu

and

reframe --config-file settings_snellius.py --checkpath test-suite/eessi/reframe/eessi-checks/ -r -R -t CI -t singlenode -n '.*cpu.*'

Result in the same set of tests:

[ RUN      ] GROMACS_EESSI %benchmark_info=HECBioSim/hEGFRDimer %nb_impl=cpu %scale=('singlenode', 1) %module_name=GROMACS/2021.6-foss-2022a /d12ed623 @snellius:thin+default
[ RUN      ] GROMACS_EESSI %benchmark_info=HECBioSim/hEGFRDimer %nb_impl=cpu %scale=('singlenode', 1) %module_name=GROMACS/2021.6-foss-2022a /d12ed623 @snellius:gpu_test+default
[ RUN      ] GROMACS_EESSI %benchmark_info=HECBioSim/hEGFRDimer %nb_impl=cpu %scale=('singlenode', 1) %module_name=GROMACS/2021.6-foss-2022a-CUDA-11.7.0 /a6f0d10c @snellius:thin+default
[ RUN      ] GROMACS_EESSI %benchmark_info=HECBioSim/hEGFRDimer %nb_impl=cpu %scale=('singlenode', 1) %module_name=GROMACS/2021.6-foss-2022a-CUDA-11.7.0 /a6f0d10c @snellius:gpu_test+default

Similarly, the equivalent of

--setvar module_regex_select=CUDA

would be

-n '.*CUDA.*'

And the equivalent of

--setvar module_regex_skip=CUDA --setvar run_mode=cpu --setvar valid_systems='-gpu'

would be

-n '.*cpu' -x '.*CUDA' -S valid_systems='-gpu'

Also, I tested

-n '(2021b|2022a)'

and this also works as expected. In fact, the documentation states this explicit case can also be achieved with the more readable

-n 2021b -n 2022a

as ReFrame selects a test if it matches any of the -n arguments.

Admittedly, the syntax of doing things with -n and -x might be slightly more involved, especilly in "A and B"-type selections such as

-n '.*gpu.*CUDA'

, but it does correctly select the test with nb_impl=gpu and the CUDA based modules.

I'd personally not be in favour of introducing more variables just to achieve a slightly simpler selection syntax. Or am I overlooking something and can you make selections with these variables that you can't do with -n and -x?

[smoors]

all parametrisation information gets included in the name of the test, and can thus be filtered on with -n and -x

oh, i didn't know this, thanks for pointing it out!

so the full display name in your example is this, right?
GROMACS_EESSI %benchmark_info=HECBioSim/hEGFRDimer %nb_impl=cpu %scale=('singlenode', 1) %module_name=GROMACS/2021.6-foss-2022a

i'll check if those options indeed cover all use cases that we care about, but it probably will, as all info is in the name.

[smoors]

with the latest update, the test:

• by default sets the number of gpus per node based on the 'devices' attribute in the site config file
• supports overriding on de cmd line with --setvar num_gpus_per_node=x

note that you should no longer explicitly set 'access': ['--gpus-per-node=x'] in the config file, otherwise overriding is broken.

please test :)

[smoors]

the last commit changes the default behavior of num_cpus_per_task to be equal to the total number of cpus per node divided by the total number of gpus per node.

for example, for a node with 48 cores and 4 gpus:
if you set num_gpus_per_node=1, then by default you will get num_tasks_per_node=1 and num_cpus_per_task=12
if you set num_gpus_per_node=3, then by default you will get num_tasks_per_node=3 and num_cpus_per_task=12

i think this is a more sensible default: if you use only part of the GPUs you will probably also use only part of the CPUs.

[casparvl]

Ok, I tested a number of combinations. For context, our GPU nodes have 4 GPUs per node, and 72 CPU cores per node.

• With -n .*gpu --setvar num_gpus_per_node=1, we get:

#SBATCH --ntasks=1
#SBATCH --ntasks-per-node=1
#SBATCH --cpus-per-task=18
#SBATCH --gpus-per-node=1

• With -n .*gpu --setvar num_tasks_per_node=1, we get:

#SBATCH --ntasks=2
#SBATCH --ntasks-per-node=2
#SBATCH --cpus-per-task=18
#SBATCH --gpus-per-node=2

• With -n .*gpu --setvar num_cpus_per_task=2, we get:

#SBATCH --ntasks=4
#SBATCH --ntasks-per-node=4
#SBATCH --cpus-per-task=2
#SBATCH --gpus-per-node=4

• With -n .*gpu --setvar num_gpus_per_node=1 --setvar num_tasks_per_node=1, we get:

#SBATCH --ntasks=1
#SBATCH --ntasks-per-node=1
#SBATCH --cpus-per-task=18
#SBATCH --gpus-per-node=1

• With -n .*gpu --setvar num_gpus_per_node=1 --setvar num_tasks_per_node=2, we get:

#SBATCH --ntasks=2
#SBATCH --ntasks-per-node=2
#SBATCH --cpus-per-task=9
#SBATCH --gpus-per-node=1

• With -n .*gpu --setvar num_gpus_per_node=2 --setvar num_tasks_per_node=4, we get:

#SBATCH --ntasks=4
#SBATCH --ntasks-per-node=4
#SBATCH --cpus-per-task=9
#SBATCH --gpus-per-node=2

• With -n .*gpu --setvar num_gpus_per_node=1 --setvar num_tasks_per_node=1 --setvar num_cpus_per_task=2, we get:

#SBATCH --ntasks=1
#SBATCH --ntasks-per-node=1
#SBATCH --cpus-per-task=2
#SBATCH --gpus-per-node=1

All good so far. But now, what about CPU tests:

• With -n .*cpu -x .*CUDA --setvar num_cpus_per_task=2

#SBATCH --ntasks=72
#SBATCH --ntasks-per-node=72
#SBATCH --cpus-per-task=2

• With -n .*cpu -x .*CUDA --setvar num_tasks_per_node=36

#SBATCH --ntasks=36
#SBATCH --ntasks-per-node=36
#SBATCH --cpus-per-task=1

I haven't had the time yet to figure out where this is set exactly, and why it's not sensible.

Note that leaving everything default

• -n .*cpu -x .*CUDA

#SBATCH --ntasks=72
#SBATCH --ntasks-per-node=72
#SBATCH --cpus-per-task=1

does work fine.

I'm guessing this was not the result of your latest commits - it probably was like that ever since we added the options of overriding these options. But maybe we should just fix that right away as well...?

Anyway, nice work on the overrides for the GPU counts :)

[smoors]

I'm guessing this was not the result of your latest commits - it probably was like that ever since we added the options of overriding these options. But maybe we should just fix that right away as well...?

indeed, my original idea was that num_cpus_per_task and num_tasks_per_node should both be set at the same time, but we can do better and make sure that by default the total number of cores per node is always requested (unless both are set of course)

the last change should fix this.
i also took the opportunity to reorganize the code a bit.

[casparvl]

Looks good to me. Nice milestone @smoors , finally a merge of this PR! :)

[casparvl]

FYI: I tested all the above combinations again, and all produce the expected and desired results now :)

[satishskamath]

Joining a bit late to this party. Sorry for that. I have been testing the recent merge and am not exactly getting the same results as both of you are.

[EESSI pilot 2021.06] $ reframe -C /home/satishk/projects/eessi_reframe/settings_example.py -c $eessihome/eessi-checks/applications/ -t CI -t singlen
ode -l --performance-report --setvar valid_systems=snellius_eessi:cpu
[ReFrame Setup]
  version:           4.0.1
  command:           '/sw/arch/RHEL8/EB_production/2022/software/ReFrame/4.0.1/bin/reframe -C /home/satishk/projects/eessi_reframe/settings_example.p
y -c /gpfs/home5/satishk/projects/test-suite/eessi/reframe/eessi-checks/applications/ -t CI -t singlenode -l --performance-report --setvar valid_syst
ems=snellius_eessi:cpu'
  launched by:       satishk@tcn2.local.snellius.surf.nl
  working directory: '/gpfs/home5/satishk/projects/eessi_reframe'
  settings files:    '<builtin>', '/home/satishk/projects/eessi_reframe/settings_example.py'
  check search path: '/gpfs/home5/satishk/projects/test-suite/eessi/reframe/eessi-checks/applications'
  stage directory:   '/scratch-shared/satishk/reframe_output/staging'
  output directory:  '/gpfs/home5/satishk/projects/eessi_reframe/output'
  log files:         '/gpfs/home5/satishk/projects/eessi_reframe/reframe.log'

[List of matched checks]
- GROMACS_EESSI %benchmark_info=HECBioSim/hEGFRDimer %nb_impl=gpu %scale=('singlenode', 1) %module_name=GROMACS/2021.6-foss-2022a /ff91f18a
- GROMACS_EESSI %benchmark_info=HECBioSim/hEGFRDimer %nb_impl=gpu %scale=('singlenode', 1) %module_name=GROMACS/2021.6-foss-2022a-CUDA-11.7.0 /54be99
0c
- GROMACS_EESSI %benchmark_info=HECBioSim/hEGFRDimer %nb_impl=gpu %scale=('singlenode', 1) %module_name=GROMACS/2020.4-foss-2020a-Python-3.8.2 /0972b
423
- GROMACS_EESSI %benchmark_info=HECBioSim/hEGFRDimer %nb_impl=gpu %scale=('singlenode', 1) %module_name=GROMACS/2020.1-foss-2020a-Python-3.8.2 /b2862
df7
- GROMACS_EESSI %benchmark_info=HECBioSim/hEGFRDimer %nb_impl=cpu %scale=('singlenode', 1) %module_name=GROMACS/2021.6-foss-2022a /d12ed623
- GROMACS_EESSI %benchmark_info=HECBioSim/hEGFRDimer %nb_impl=cpu %scale=('singlenode', 1) %module_name=GROMACS/2021.6-foss-2022a-CUDA-11.7.0 /a6f0d1
0c
- GROMACS_EESSI %benchmark_info=HECBioSim/hEGFRDimer %nb_impl=cpu %scale=('singlenode', 1) %module_name=GROMACS/2020.4-foss-2020a-Python-3.8.2 /80c7f
af7
- GROMACS_EESSI %benchmark_info=HECBioSim/hEGFRDimer %nb_impl=cpu %scale=('singlenode', 1) %module_name=GROMACS/2020.1-foss-2020a-Python-3.8.2 /a4450
ec7
Found 8 check(s)

Log file(s) saved in '/gpfs/home5/satishk/projects/eessi_reframe/reframe.log'
[EESSI pilot 2021.06] $ 

with -n I get the right filtering. My question is, why does it report gpu tests when I choose the cpu partition?

[smoors]

@satishskamath currently, if you specify valid_systems on the command line, no automatic filtering is done, so you have to do the filtering yourself.

the reason is that unfortunately, combining systemname:partitionname with features is not supported in the current version of ReFrame (4.0.5).

now that i think about it, maybe it's possible check the features of the partitions that are specified on the cmd line, and filter the tests based on that? something to try and/or discuss..

[smoors]

i created an issue for this: #21