Ctest jjobs framework

[TerrenceMcGuinness-NOAA]

Description

We create a CMakeList.txt to demonstrate how it can be added to the main CMake in $HOMEgfs/sorc/CMakeLists.txt to add functional tests for selected JJOBs for experments in $HOMEgfs/ci/pr/cases.

Each of these tests has 4 components:

1. Create Experiment Directory
2. Stage Input Data
3. Run Experiment
4. Validate output

You can run the above test for each JJOB using CMake Labels (or all of the JJOBS for case)

ctest -L C48_ATM_fcst_gfs
ctest -L C48_ATM

Here is an end-to-end demo as an example:

tmcguinness ctests (ctest_jjobs_framework) $ ls
cases  CMakeLists.txt  scripts


tmcguinness ctests (ctest_jjobs_framework) $ scripts/generate_ctest_args.py --yaml cases/C48_ATM.yaml cases/C48_S2SW.yaml 
-DCASE_LIST='C48_ATM C48_S2SW' -DJOB_NAMES_C48_ATM='C48_ATM_fcst_gfs C48_ATM_gfs_atmos_pro' -DJOB_NAMES_C48_S2SW='C48_S2SW_fcst_gfs C48_S2SW_gfs_atmos_pro'
 

tmcguinness ctest (ctest_jjobs_framework) $ mkdir build; cd build


tmcguinness build (ctest_jjobs_framework) $ cmake .. -DCASE_LIST='C48_ATM C48_S2SW' -DJOB_NAMES_C48_ATM='C48_ATM_fcst_gfs C48_ATM_gfs_atmos_pro' -DJOB_NAMES_C48_S2SW='C48_S2SW_fcst_gfs C48_S2SW_gfs_atmos_pro'
-- The C compiler identification is GNU 11.5.0
-- The CXX compiler identification is GNU 11.5.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Executing: git rev-parse --short HEAD
-- Configuring done (0.7s)
-- Generating done (0.0s)
-- Build files have been written to: /home/tmcguinness/NOAA/global-workflow_forked2/ci/ctests/build


tmcguinness build (ctest_jjobs_framework) $ ctest -L C48_ATM_fcst_gfs
Test project /home/tmcguinness/NOAA/global-workflow_forked2/ci/ctests/build
    Start 1: test_C48_ATM_fcst_gfs
1/4 Test #1: test_C48_ATM_fcst_gfs ............   Passed    0.61 sec
    Start 2: stage_C48_ATM_fcst_gfs
2/4 Test #2: stage_C48_ATM_fcst_gfs ...........   Passed    0.01 sec
    Start 3: run_C48_ATM_fcst_gfs
3/4 Test #3: run_C48_ATM_fcst_gfs .............   Passed    0.00 sec
    Start 4: validate_C48_ATM_fcst_gfs
4/4 Test #4: validate_C48_ATM_fcst_gfs ........   Passed    0.01 sec

100% tests passed, 0 tests failed out of 4

Label Time Summary:
C48_ATM             =   0.63 sec*proc (4 tests)
C48_ATM_fcst_gfs    =   0.63 sec*proc (4 tests)

Total Test time (real) =   0.64 sec


tmcguinness build (ctest_jjobs_framework) $ ls RUNTESTS/EXPDIR/
C48_ATM_C48_ATM_fcst_gfs_36d497ee


tmcguinness build (ctest_jjobs_framework) $ ctest -L C48_S2SW_gfs_atmos_pro
Test project /home/tmcguinness/NOAA/global-workflow_forked2/ci/ctests/build
    Start 13: test_C48_S2SW_gfs_atmos_pro
1/4 Test #13: test_C48_S2SW_gfs_atmos_pro .......   Passed    0.70 sec
    Start 14: stage_C48_S2SW_gfs_atmos_pro
2/4 Test #14: stage_C48_S2SW_gfs_atmos_pro ......   Passed    0.01 sec
    Start 15: run_C48_S2SW_gfs_atmos_pro
3/4 Test #15: run_C48_S2SW_gfs_atmos_pro ........   Passed    0.00 sec
    Start 16: validate_C48_S2SW_gfs_atmos_pro
4/4 Test #16: validate_C48_S2SW_gfs_atmos_pro ...   Passed    0.01 sec

100% tests passed, 0 tests failed out of 4

Label Time Summary:
C48_S2SW                  =   0.71 sec*proc (4 tests)
C48_S2SW_gfs_atmos_pro    =   0.71 sec*proc (4 tests)

Total Test time (real) =   0.72 sec


tmcguinness build (ctest_jjobs_framework) $ ls RUNTESTS/EXPDIR/
C48_ATM_C48_ATM_fcst_gfs_36d497ee  C48_S2SW_C48_S2SW_gfs_atmos_pro_36d497ee

Type of change

• Bug fix (fixes something broken)
• New feature (adds functionality)
• Maintenance (code refactor, clean-up, new CI test, etc.)