Skip to content

GSI Ctests (regression tests)

Jump to bottom
Ming Hu edited this page Oct 1, 2024 · 41 revisions

To use ctest to run the regression tests:

1. Build and install control for the regression tests.

This is usually [NOAA-EMC/GSI develop], the head of the branch into which you want to merge:

2. Build and install the head of the branch containing your changes.

This is the update for the regression tests. Please remember to include --recursive when you clone your fork containing the branch with your changes.

Your branch with your changes should be installed at the same directory level as your control. For example, if you install the branch with your changes in $DIR_ROOT, the control gsi.x and enkf.x need to be in $DIR_ROOT../develop/install/bin. Script ush/build.sh sets CONTROLPATH="$DIR_ROOT/../develop/install/bin". If your control gsi.x and enkf.x are in a different location, set CONTROLPATH to the appropriate path in ush/build.sh.

3. Run regression tests

cd into build in the working copy of the branch containing your changes and use the following ctest command to run the cases.

  • ctest -N lists all available ctests. Currently 9 tests are available.
  • To launch only the global tests, execute ctest -R global.
  • To concurrently launch multiple tests , execute ctest -j X where X is the number of tests to concurrently run.

If not run under EMC account, need to set following three parameters in the head of regression/regression_var.sh:

  • ptmp="Space to run regression cases"
  • group="Your Group"
  • accnt="Your Account"

4. check results:

Each ctest consists of four runs for the indicated configuration

  1. high mpi task & thread count for update
  2. low mpi task & thread count for update
  3. high mpi task & thread count for control
  4. low mpi task & thread count for control

Failed or Passed will be echoed to the screen as ctests complete. For example:

ctest -R rrfs_3denvar_glbens

Test project /scratch1/BMC/wrfruc/mhu/code/gsi/test3/GSI/build

`Start 7: rrfs_3denvar_glbens`

1/1 Test #7: rrfs_3denvar_glbens .............. Passed 546.88 sec

100% tests passed, 0 tests failed out of 1

Total Test time (real) = 546.89 sec

Once all ctests complete a summary report for each test will be written to the path specified by variable noscrub in regression/regression_var.sh

NOTES:

  • At present NOAA-EMC/GSI regression tests are only available on Hera, Orion, and WCOSS2.

5. Run RRFS EnKF regression test case:

The default EnKF case is for global (global_enkf)only. If the PR only includes code changes for GSI, the above regression tests are good. If the PR has code changes for EnKF, the regression test should be rerun to test the RRFS EnKF case. Here are steps to run RRFS EnKF case:

  • change ENKF_MODE in ush/build.sh from ENKF_MODE=${ENKF_MODE:-"GFS"} to ENKF_MODE=${ENKF_MODE:-"FV3REG"}
  • edit regression/multi_regression.sh, replace global_enkf with rrfs_enkf_conv .
  • recompile both control branch and your branch.
  • under your branch build directory, run: ctest -R rrfs_enkf_conv
  • check and report the results following the above step.

How to add a case to ctest

Add new case name

  1. Add new case name to the case list section in CMakeLists.txt:

For GSI regression test:

list(APPEND GSI_REG_TEST_NAMES global_3dvar global_4dvar global_4denvar hwrf_nmm_d2 hwrf_nmm_d3 rtma rrfs_3denvar_glbens netcdf_fv3_regional )

For EnKF regression test

list(APPEND ENKF_REG_TEST_NAMES global_enkf )

  1. Add new case name to list regtests_all and regtests_debug in multi_regression.sh

Add namelist

Add namelist of the new case to regression_namelists.sh and regression_namelists_db.sh

Setup compute resources

Add a section to setup compute resources with case in regression_param.sh, for example rrfs_3denvar_glbens

Add case data

Add case data information in regression_var.sh, for example rrfs_3denvar_glbens

Add run script

Add run script for this new case, for example rrfs_3denvar_glbens.sh. The run scripts need to use information from pervious setups.

Verification

May also need to add code to decide how to compare the results in regression_test.sh, for example rrfs

Clone this wiki locally