Enable AWS Parallel Works platform and Add Comprehensive End-To-End Tests

[jessemcfarland]

DESCRIPTION OF CHANGES:

First, the AWS Parallel Works platform has been activated. A couple of additional minor changes were necessary in order to get a successful build and test run. These changes include mapping the Parallel Works cluster names to noaacloud and ensuring the the PROJ_LIB environment variable is set.

Second, support for the comprehensive workflow/end-to-end tests was added to the Jenkins pipeline and unified test script. A boolean parameter, SRW_WE2E_COMPREHENSIVE_TESTS, was added to the Jenkins pipeline. The parameter can be used to execute the comprehensive test suite manually on the desired branches. In addition, logic was added to the test stage to scan Pull Request labels for a specific label, run_we2e_comprehensive_tests. If set, the value of the SRW_WE2E_COMPREHENSIVE_TESTS parameter is overridden. The list of comprehensive workflow/end-to-end tests was added to the unified test script.

TESTS CONDUCTED:

This PR is the test. Two new labels should be created: run_we2e_fundamental_tests and run_we2e_comprehensive_tests. First, the run_we2e_fundamental_tests label should be applied to the PR. This should result in the pipeline executing the default set of tests. Next, the run_we2e_fundamental_tests label should be removed and the run_we2e_comprehensive_tests label should be added. In addition, a comment with the text "REBUILD" should be added to the PR. This should result in the pipeline executing the comprehensive set of tests. NOTE: The tests results can be found under the S3 Artifacts section of a Jenkins Build.

[JeffBeck-NOAA]

@jessemcfarland, thanks for getting the full suite of comprehensive tests integrated into Jenkins! Can we use a run_we2e_fundamental_tests label instead of run_we2e_default_tests? This will match the 'fundamental' vs 'comprehensive' paradigm. Thanks!

[jessemcfarland]

@jessemcfarland, thanks for getting the full suite of comprehensive tests integrated into Jenkins! Can we use a run_we2e_fundamental_tests label instead of run_we2e_default_tests? This will match the 'fundamental' vs 'comprehensive' paradigm. Thanks!

Yes! That's actually a copy/paste mistake, it works as you described though.

[gsketefian]

I think there are more fundamental tests than these. Same for comprehensive tests. Will comment more soon.

[gsketefian]

@jessemcfarland Here's the latest set of fundamental tests:

grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16
grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_RRFS_v1beta
grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_RAP_suite_HRRR
grid_RRFS_CONUS_25km_ics_NAM_lbcs_NAM_suite_HRRR
grid_RRFS_CONUS_25km_ics_NAM_lbcs_NAM_suite_RRFS_v1beta
grid_RRFS_CONUScompact_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16
grid_RRFS_CONUScompact_25km_ics_HRRR_lbcs_HRRR_suite_HRRR
grid_RRFS_CONUScompact_25km_ics_HRRR_lbcs_RAP_suite_HRRR
grid_RRFS_CONUScompact_25km_ics_HRRR_lbcs_RAP_suite_RRFS_v1beta
grid_SUBCONUS_Ind_3km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16
grid_SUBCONUS_Ind_3km_ics_HRRR_lbcs_RAP_suite_HRRR
grid_SUBCONUS_Ind_3km_ics_HRRR_lbcs_RAP_suite_RRFS_v1beta
nco_grid_RRFS_CONUScompact_25km_ics_HRRR_lbcs_RAP_suite_HRRR
community_ensemble_2mems
custom_ESGgrid
deactivate_tasks
inline_post
MET_ensemble_verification
MET_verification
nco_ensemble
pregen_grid_orog_sfc_climo
specify_DOT_OR_USCORE
specify_DT_ATMOS_LAYOUT_XY_BLOCKSIZE
specify_RESTART_INTERVAL
specify_template_filenames

Also, you can remove subhourly_post and subhourly_post_ensemble_2mems from the comprehensive tests because I think the subhourly post feature (really should be called "subhourly forecast output" or something similar) is broken.

Please go ahead and update the list of tests here. I know I said it during the meeting, but I'll mention again here for the record that @mkavulich is going to add a variable to the WE2E test configuration files that indicates whether a given test is part of the fundamental list, and he will add a flag to run_WE2E_tests.sh that runs just the fundamental set of tests. Once that feature is in, you won't have to maintain a list in this script.

Finally, I just checked the run_WE2E_tests.sh script, and I don't see any if-statements, etc, that remove tests that shouldn't be run on certain platforms, e.g. removing get_from_HPSS_... on platforms that don't have access to NOAA HPSS. I remember @mkavulich mentioned he wanted to add that to the script (so you won't have to do it in this script and the capability will be available to users who run tests from the command line), but I don't know where that stands.

[jessemcfarland]

@gsketefian I updated the fundamental tests lists (and subsequently the comprehensive tests lists). NOTE: I have logic in the script that adds the MET_ensemble_verification, MET_verification, and pregen_grid_orog_sfc_climo tests only when the platform is NOT Gaea or one of the Parallel Works clusters. Is this correct?

Can you verify that the comprehensive tests list is correct as well? Is there a definitive list of all tests somewhere in the repository?

[gsketefian]

Hi @jessemcfarland, I just checked on Gaea, and it looks like METplus is not installed there (and the MET version is pretty old; it's v8.1, but latest is v10.1.1), so you're right, the MET_ensemble_verification and MET_verification tests cannot be run there. For pregen_grid_orog_sfc_climo, I cannot tell if the files needed to run that test are in the staging directory because the staging directory does not have read permission for "other" (so can't see contents). I assume you tried running pregen_grid_orog_sfc_climo and it failed, is that right?

For now, let's do what you're doing and remove these 3 tests on gaea and noaacloud, but we should discuss this at the next code management meeting. I've put some points to discuss in the meeting notes (feel free to raise other concerns there). One outstanding question is what to do with tests that work on some platforms but not others. Do we still run the test so we're reminded that there's a failure that needs to be fixed? Removing them here in this script is a bit sneaky in that the user may not realize it's happening and will assume all tests passed.

For the comprehensive tests, the idea is that we don't need a list and should run all available tests. So eventually (after @mkavulich is done with his changes), we shouldn't need a list here. For now, below is a list that you can use for the develop branch (I think the one you have was for the last release branch). I've removed subhourly_post and subhourly_post_ensemble_2mems from the list as well as a couple of others that are actually alternate names (these are nco_inline_post and template_vars, which are just symlinks to nco_grid_RRFS_CONUScompact_25km_ics_HRRR_lbcs_RAP_suite_HRRR and deactivate_tasks, respectively).

MET_ensemble_verification
MET_verification
community_ensemble_008mems
community_ensemble_2mems
custom_ESGgrid
custom_GFDLgrid
custom_GFDLgrid__GFDLgrid_USE_NUM_CELLS_IN_FILENAMES_eq_FALSE
custom_GFDLgrid__GFDLgrid_USE_NUM_CELLS_IN_FILENAMES_eq_TRUE
deactivate_tasks
get_from_HPSS_ics_FV3GFS_lbcs_FV3GFS_fmt_grib2_2019061200
get_from_HPSS_ics_FV3GFS_lbcs_FV3GFS_fmt_grib2_2019101818
get_from_HPSS_ics_FV3GFS_lbcs_FV3GFS_fmt_grib2_2020022518
get_from_HPSS_ics_FV3GFS_lbcs_FV3GFS_fmt_grib2_2020022600
get_from_HPSS_ics_FV3GFS_lbcs_FV3GFS_fmt_grib2_2021010100
get_from_HPSS_ics_FV3GFS_lbcs_FV3GFS_fmt_nemsio
get_from_HPSS_ics_FV3GFS_lbcs_FV3GFS_fmt_nemsio_2019061200
get_from_HPSS_ics_FV3GFS_lbcs_FV3GFS_fmt_nemsio_2019101818
get_from_HPSS_ics_FV3GFS_lbcs_FV3GFS_fmt_nemsio_2020022518
get_from_HPSS_ics_FV3GFS_lbcs_FV3GFS_fmt_nemsio_2020022600
get_from_HPSS_ics_FV3GFS_lbcs_FV3GFS_fmt_nemsio_2021010100
get_from_HPSS_ics_FV3GFS_lbcs_FV3GFS_fmt_netcdf_2021062000
get_from_HPSS_ics_FV3GFS_lbcs_FV3GFS_fmt_netcdf_2022060112_48h
get_from_HPSS_ics_GSMGFS_lbcs_GSMGFS
get_from_HPSS_ics_HRRR_lbcs_RAP
get_from_HPSS_ics_RAP_lbcs_RAP
get_from_NOMADS_ics_FV3GFS_lbcs_FV3GFS_fmt_nemsio
inline_post
nco_ensemble
pregen_grid_orog_sfc_climo
specify_DOT_OR_USCORE
specify_DT_ATMOS_LAYOUT_XY_BLOCKSIZE
specify_EXTRN_MDL_SYSBASEDIR_ICS_LBCS
specify_RESTART_INTERVAL
specify_template_filenames
grid_CONUS_25km_GFDLgrid_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16
grid_CONUS_3km_GFDLgrid_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16
grid_RRFS_AK_13km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16
grid_RRFS_AK_3km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16
grid_RRFS_CONUS_13km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v15p2
grid_RRFS_CONUS_13km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16
grid_RRFS_CONUS_13km_ics_FV3GFS_lbcs_FV3GFS_suite_HRRR
grid_RRFS_CONUS_13km_ics_FV3GFS_lbcs_FV3GFS_suite_RRFS_v1beta
grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_2017_gfdlmp
grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_2017_gfdlmp_regional
grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v15p2
grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16
grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_HRRR
grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_RRFS_v1beta
grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_RAP_suite_HRRR
grid_RRFS_CONUS_25km_ics_GSMGFS_lbcs_GSMGFS_suite_GFS_2017_gfdlmp
grid_RRFS_CONUS_25km_ics_GSMGFS_lbcs_GSMGFS_suite_GFS_v15p2
grid_RRFS_CONUS_25km_ics_GSMGFS_lbcs_GSMGFS_suite_GFS_v16
grid_RRFS_CONUS_25km_ics_NAM_lbcs_NAM_suite_HRRR
grid_RRFS_CONUS_25km_ics_NAM_lbcs_NAM_suite_RRFS_v1beta
grid_RRFS_CONUS_3km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v15_thompson_mynn_lam3km
grid_RRFS_CONUS_3km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v15p2
grid_RRFS_CONUS_3km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16
grid_RRFS_CONUS_3km_ics_FV3GFS_lbcs_FV3GFS_suite_HRRR
grid_RRFS_CONUS_3km_ics_FV3GFS_lbcs_FV3GFS_suite_RRFS_v1beta
grid_RRFS_CONUScompact_13km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16
grid_RRFS_CONUScompact_13km_ics_HRRR_lbcs_RAP_suite_HRRR
grid_RRFS_CONUScompact_13km_ics_HRRR_lbcs_RAP_suite_RRFS_v1beta
grid_RRFS_CONUScompact_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16
grid_RRFS_CONUScompact_25km_ics_HRRR_lbcs_HRRR_suite_HRRR
grid_RRFS_CONUScompact_25km_ics_HRRR_lbcs_HRRR_suite_RRFS_v1beta
grid_RRFS_CONUScompact_25km_ics_HRRR_lbcs_RAP_suite_HRRR
grid_RRFS_CONUScompact_25km_ics_HRRR_lbcs_RAP_suite_RRFS_v1beta
grid_RRFS_CONUScompact_3km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16
grid_RRFS_CONUScompact_3km_ics_HRRR_lbcs_RAP_suite_GFS_v15p2
grid_RRFS_CONUScompact_3km_ics_HRRR_lbcs_RAP_suite_HRRR
grid_RRFS_CONUScompact_3km_ics_HRRR_lbcs_RAP_suite_RRFS_v1beta
grid_RRFS_NA_13km_ics_FV3GFS_lbcs_FV3GFS_suite_RRFS_v1beta
grid_RRFS_NA_3km_ics_FV3GFS_lbcs_FV3GFS_suite_RRFS_v1beta
grid_RRFS_SUBCONUS_3km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16
grid_RRFS_SUBCONUS_3km_ics_HRRR_lbcs_RAP_suite_GFS_v15p2
grid_SUBCONUS_Ind_3km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16
grid_SUBCONUS_Ind_3km_ics_HRRR_lbcs_RAP_suite_HRRR
grid_SUBCONUS_Ind_3km_ics_HRRR_lbcs_RAP_suite_RRFS_v1beta
nco_grid_RRFS_CONUS_13km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16
nco_grid_RRFS_CONUS_3km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v15_thompson_mynn_lam3km
nco_grid_RRFS_CONUScompact_25km_ics_HRRR_lbcs_RAP_suite_HRRR

[gsketefian]

@jessemcfarland Please let me know if/when you'd like me to take another look at this PR. Thanks.

[jessemcfarland]

REBUILD

[jessemcfarland]

REBUILD

[JeffBeck-NOAA]

@jessemcfarland, do you have any updates on this PR? Thanks!

[jessemcfarland]

@jessemcfarland, do you have any updates on this PR? Thanks!

@JeffBeck-NOAA I'm working on rebasing the branch and making adjustments to paths related to the incorporation of regional_workflow repo now. I'll schedule a run after that to see where we are, but I think there have been some changes to the platforms that will require other code updates and vice versa.

[jessemcfarland]

REBUILD

[jessemcfarland]

@JeffBeck-NOAA Here is the current summary of the failures, the full log for each platform/stage can be viewed in Jenkins. I'll need some assistance in determining the appropriate fix for each failure. NOTE: All platforms were working (executing the tests) when this pipeline was tested on the latest release branch.

• Cheyenne: Fails during test stage; fails to load conda module because python module is already loaded.
• Gaea: Fails during test stage; see error below

The directory (COMIN) that needs to be specified when running the workflow in NCO mode (RUN_ENVIR set to "nco") AND using the FV3GFS or the GSMGFS as the external model for ICs and/or LBCs has not been specified for this machine (MACHINE): MACHINE= "GAEA"

• Hera: Comprehensive tests are executed successfully, however, there are 19 failures. I need someone to investigate these failures and determine if any tests should be omitted for Hera.
• Jet: Comprehensive tests are executed successfully, however, there are 24 failures. I need someone to investigate these failures and determine if any tests should be omitted for Jet.
• Orion: Fails during test stage; OCL_ICD_FILENAMES_RESET: unbound variable error during conda initialization.
• Parallel Works AWS: Fails during build stage; fails to load libpng module.

[danielabdi-noaa]

@jessemcfarland I would ignore 17 of those failures (get_from_HPSS*) on hera/jet due to issue #349
Manual re-run solves all of those, most likely because it was a timeout issue

get_from_HPSS_ics_FV3GFS_lbcs_FV3GFS_fmt_grib2_2019061200 FAILURE
get_from_HPSS_ics_FV3GFS_lbcs_FV3GFS_fmt_grib2_2019101818 FAILURE
get_from_HPSS_ics_FV3GFS_lbcs_FV3GFS_fmt_grib2_2020022518 FAILURE
get_from_HPSS_ics_FV3GFS_lbcs_FV3GFS_fmt_grib2_2020022600 FAILURE
get_from_HPSS_ics_FV3GFS_lbcs_FV3GFS_fmt_grib2_2021010100 FAILURE
get_from_HPSS_ics_FV3GFS_lbcs_FV3GFS_fmt_nemsio FAILURE
get_from_HPSS_ics_FV3GFS_lbcs_FV3GFS_fmt_nemsio_2019061200 FAILURE
get_from_HPSS_ics_FV3GFS_lbcs_FV3GFS_fmt_nemsio_2019101818 FAILURE
get_from_HPSS_ics_FV3GFS_lbcs_FV3GFS_fmt_nemsio_2020022518 FAILURE
get_from_HPSS_ics_FV3GFS_lbcs_FV3GFS_fmt_nemsio_2020022600 FAILURE
get_from_HPSS_ics_FV3GFS_lbcs_FV3GFS_fmt_nemsio_2021010100 FAILURE
get_from_HPSS_ics_FV3GFS_lbcs_FV3GFS_fmt_netcdf_2021062000 FAILURE
get_from_HPSS_ics_GSMGFS_lbcs_GSMGFS FAILURE
get_from_HPSS_ics_HRRR_lbcs_RAP FAILURE
get_from_HPSS_ics_RAP_lbcs_RAP FAILURE
get_from_NOMADS_ics_FV3GFS_lbcs_FV3GFS_fmt_nemsio FAILURE

That leaves out two failures for Hera

nco_grid_RRFS_CONUS_13km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16 FAILURE
nco_grid_RRFS_CONUS_3km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v15_thompson_mynn_lam3km FAILURE

These should not have failed comparing to the last comprhensive test done during merge here #343

For Jet, besides the above 19 failures

nco_ensemble FAILURE
pregen_grid_orog_sfc_climo FAILURE
grid_RRFS_CONUScompact_25km_ics_HRRR_lbcs_RAP_suite_RRFS_v1beta FAILURE
grid_RRFS_CONUS_25km_ics_GSMGFS_lbcs_GSMGFS_suite_GFS_v16 FAILURE
grid_RRFS_CONUScompact_3km_ics_HRRR_lbcs_RAP_suite_GFS_v15p2 FAILURE
grid_RRFS_NA_3km_ics_FV3GFS_lbcs_FV3GFS_suite_RRFS_v1beta FAILURE

The first two are known failures on Jet comparing to #343. The rest are most likely due to forecast runnign out of time given that they are mostly 3km runs. You would have to look at the log files.

For Orion, I believe I have had that same issue for months when I last tried to incorporate your changes from release to develop in PR #778 The Jenkins pipeline page seem to be no more accessible but here is the link
https://jenkins-epic.woc.noaa.gov/blue/organizations/jenkins/ufs-srweather-app%2Fpipeline/detail/PR-309/6/pipeline/145/
Gaea and Cheynne were working fine at the time though but there is an upgrade to conda for cheynne since then.

In my opinion, it is going to be hard to a get a green on comprehensive tests given some test cases fail first time they are run
but a re-run works just fine.

[MichaelLueken]

As discussed in this afternoon's meeting, since there are two approvals, I will merge this work into the official repository.