Common build and test scripts, and configuration files, for CIs and manual runs

[danielabdi-noaa]

Description

Currently continuous integration platforms for SRW app, Jenkins and Github actions, use separate scripts for building and testing. Jenkins has own srw_build and srw_test scripts, while github actions uses test/build.sh and setup_WE2E_tests.sh. Also the test configuration approach is different for both. This is a maintenance nightmare and I believe there should be common build/test scripts and configuration files for building the SRW app, configuring WE2E tests, and running the actual tests for both. The scripts should also allow for direct manual execution of the tests. Some observations:

Build

• Jenkins uses cmake directly instead of devbuild.sh. With the recent move to "exec" binary directory, this breaks.
  Direct cmake build is outdated in my opinion.
• Github actions uses own test/build.sh that calls devbuild.sh. This one does a rigorous check for existence of all binaries (including gsi and rrfs_utls binaries that are not currently used). This is much better than the Jenkins's one that tests only presence of the ufs_model binary.

Test

• Jenkins is again broken because Cheynne moved to using conda recently but the conda activation logic in srw_build is not updated. Also WE2E/get_expts_status.sh can be used to avoid code duplication for checking status of experiments
• Github action's setup_WE2E conda activation logic is broken for Cheyenne although it is supposed to work only on hera and jet

Configuration

• Jenkins embeds "fundamental" and "comprehensive" test cases in the test script itself. I think it is better to have that in a file that can be customized for each platform.
• Github actions have separate files "hera.txt", "jet.txt" etc but is only limited to fundamental tests. Also the list is different from Jenkin's but it really shouldn't be.

Solution

There should be one script for build and test, and one configuration methodology shared among different CIs and manual tests as well. That will take away the load of from CI programmers shoulder's who may not be super-familiar with SRW app configuration to troubleshoot problems. It looks to me like the Github actions build and test script are more mature, although they need generalization to work on all platforms. So modifying those to work for both CIs and manual testing looks to me like a good solution.

Separate configuration text files for fundamental, comprehensive and custom tests. If a machine specific form of the tests is available e.g fundamental.hera, that will be used instead of the default fundamental.

Requirements**

None

Acceptance Criteria (Definition of Done)

Need to run both CIs with shared build and test script, shared config files. Manual run of the test cases should be possible using the same scripts. If possible move environment setup such as module loads and conda activate away from the scripts, so that when they change later the CIs continue to work fine.

[MichaelLueken]

Completed by #378