Skip to content

Commit

Permalink
Adding new E2E non-compact grids (#786)
Browse files Browse the repository at this point in the history 
* update IC/LBC file paths to match new IC/LBC file structure (#766)

* update IC/LBC file paths to match new IC/LBC file structure

* change forecast length

* Update config.community.sh

These changes reflects the SRW team's preference for a 12-hours forecast. The gfs.t18z.pgrb2.0p25.f012 file will need to be added to all Level 1 systems.

Co-authored-by: Gillian Petro <96886803+gspetro-NOAA@users.noreply.github.com>

* add a machine file for MacOS (#777)

Co-authored-by: Natalie Perlin <perlin.natalie@gmail.com>

* Add columns for relative test and time step to CSV file containing WE2E test info (#776)

## DESCRIPTION OF CHANGES: 
This PR modifies the script `get_WE2Etest_names_subdirs_descs.sh` that creates the CSV file containing test information the columns for relative cost and time step (`DT_ATMOS`).  This allows users to have a better idea of what the cost of each test is (the relative cost is defined such that 1 unit represents the cost of running a single 6-hour forecast on the `RRFS_CONUS_25km` grid).  

Other related changes:
* Add the flag `generate_csv_file` to the `run_WE2E_tests.sh` script to allow users to skip the CSV file generation step (since it can take a while).
* Use arguments for the function `set_predef_grid_params()` so that it can be called not just from the experiment generation scripts but also others (in this case from `get_WE2Etest_names_subdirs_descs.sh`).  Necessary for this PR.
* Move some constants (`NH0`, `NH3`, `NH4`) from `config_defaults.sh` to `constants.sh` to make it easier to access them.
* Place the contents of `constants.sh` at the start of each experiment's `var_defns.sh` so that these constants are available to the j-jobs and ex-scripts when `var_defns.sh` is sourced.
* Use variable `valid_vals_BOOLEAN` in many more places (which allows many other `valid_vals_...` variables in `valid_param_vals.sh` and `setup.sh` to be removed).
* Several other bug fixes.

## TESTS CONDUCTED: 
The following tests (with results) were conducted on Hera.  There were only three (previously existing) failures.
```
* grid_CONUS_25km_GFDLgrid_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16:               SUCCESS
* grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_2017_gfdlmp:           SUCCESS
* grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_2017_gfdlmp_regional:  SUCCESS
* grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v15p2:                 SUCCESS
* grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16:                   SUCCESS
* grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_RAP_suite_HRRR:                         SUCCESS
* grid_RRFS_CONUS_25km_ics_GSMGFS_lbcs_GSMGFS_suite_GFS_2017_gfdlmp:           SUCCESS
* grid_RRFS_CONUS_25km_ics_GSMGFS_lbcs_GSMGFS_suite_GFS_v15p2:                 SUCCESS
* grid_RRFS_CONUS_25km_ics_GSMGFS_lbcs_GSMGFS_suite_GFS_v16:                   FAILURE
* grid_RRFS_CONUS_25km_ics_NAM_lbcs_NAM_suite_HRRR:                            SUCCESS
* grid_RRFS_CONUS_25km_ics_NAM_lbcs_NAM_suite_RRFS_v1beta:                     SUCCESS
* grid_RRFS_CONUScompact_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16:            SUCCESS
* grid_RRFS_CONUScompact_25km_ics_HRRR_lbcs_HRRR_suite_HRRR:                   SUCCESS
* grid_RRFS_CONUScompact_25km_ics_HRRR_lbcs_HRRR_suite_RRFS_v1beta:            SUCCESS
* grid_RRFS_CONUScompact_25km_ics_HRRR_lbcs_RAP_suite_HRRR:                    SUCCESS
* grid_RRFS_CONUScompact_25km_ics_HRRR_lbcs_RAP_suite_RRFS_v1alpha:            SUCCESS
* grid_RRFS_CONUScompact_25km_ics_HRRR_lbcs_RAP_suite_RRFS_v1beta:             SUCCESS
* grid_SUBCONUS_Ind_3km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16:                  SUCCESS
* grid_SUBCONUS_Ind_3km_ics_HRRR_lbcs_RAP_suite_HRRR:                          SUCCESS
* grid_SUBCONUS_Ind_3km_ics_HRRR_lbcs_RAP_suite_RRFS_v1beta:                   SUCCESS
* GST_release_public_v1:                                                       SUCCESS
* MET_ensemble_verification:                                                   SUCCESS
* MET_verification:                                                            SUCCESS
* community_ensemble_008mems:                                                  SUCCESS
* community_ensemble_2mems:                                                    SUCCESS
* custom_ESGgrid:                                                              SUCCESS
* custom_GFDLgrid:                                                             SUCCESS
* custom_GFDLgrid__GFDLgrid_USE_GFDLgrid_RES_IN_FILENAMES_eq_FALSE:            SUCCESS
* custom_GFDLgrid__GFDLgrid_USE_GFDLgrid_RES_IN_FILENAMES_eq_TRUE:             SUCCESS
* deactivate_tasks:                                                            SUCCESS
* get_from_HPSS_ics_FV3GFS_lbcs_FV3GFS_fmt_grib2_2019061200:                   SUCCESS
* get_from_HPSS_ics_FV3GFS_lbcs_FV3GFS_fmt_grib2_2019101818:                   SUCCESS
* get_from_HPSS_ics_FV3GFS_lbcs_FV3GFS_fmt_grib2_2020022518:                   SUCCESS
* get_from_HPSS_ics_FV3GFS_lbcs_FV3GFS_fmt_grib2_2020022600:                   SUCCESS
* get_from_HPSS_ics_FV3GFS_lbcs_FV3GFS_fmt_grib2_2021010100:                   SUCCESS
* get_from_HPSS_ics_FV3GFS_lbcs_FV3GFS_fmt_nemsio:                             SUCCESS
* get_from_HPSS_ics_FV3GFS_lbcs_FV3GFS_fmt_nemsio_2019061200:                  SUCCESS
* get_from_HPSS_ics_FV3GFS_lbcs_FV3GFS_fmt_nemsio_2019101818:                  SUCCESS
* get_from_HPSS_ics_FV3GFS_lbcs_FV3GFS_fmt_nemsio_2020022518:                  SUCCESS
* get_from_HPSS_ics_FV3GFS_lbcs_FV3GFS_fmt_nemsio_2020022600:                  SUCCESS
* get_from_HPSS_ics_FV3GFS_lbcs_FV3GFS_fmt_nemsio_2021010100:                  SUCCESS
* get_from_HPSS_ics_FV3GFS_lbcs_FV3GFS_fmt_netcdf_2021062000:                  SUCCESS
* get_from_HPSS_ics_GSMGFS_lbcs_GSMGFS:                                        SUCCESS
* get_from_HPSS_ics_HRRR_lbcs_RAP:                                             SUCCESS
* get_from_HPSS_ics_RAP_lbcs_RAP:                                              SUCCESS
* get_from_NOMADS_ics_FV3GFS_lbcs_FV3GFS_fmt_nemsio:                           SUCCESS
* inline_post:                                                                 SUCCESS
* nco_ensemble:                                                                SUCCESS
* nco_inline_post:                                                             SUCCESS
* pregen_grid_orog_sfc_climo:                                                  SUCCESS
* specify_DOT_OR_USCORE:                                                       SUCCESS
* specify_DT_ATMOS_LAYOUT_XY_BLOCKSIZE:                                        SUCCESS
* specify_EXTRN_MDL_SYSBASEDIR_ICS_LBCS:                                       SUCCESS
* specify_RESTART_INTERVAL:                                                    SUCCESS
* specify_template_filenames:                                                  SUCCESS
* subhourly_post:                                                              FAILURE
* subhourly_post_ensemble_2mems:                                               FAILURE
```

## DOCUMENTATION:
Documentation is updated via PR #[278](ufs-community/ufs-srweather-app#278) in the ``ufs-srweather-app`` repo.

* Bug fix for calculation of number of time steps for a given WE2E test. (#782)

## DESCRIPTION OF CHANGES: 
This PR fixes a bug in the way the number of time steps (`num_time_steps` in `get_WE2Etest_names_subdirs_descs.sh`) is calculated for each WE2E test when creating the CSV file containing information about the available WE2E tests.

## TESTS CONDUCTED: 
Reran the `run_WE2E_tests.sh` script (which in turn calls `get_WE2Etest_names_subdirs_descs.sh`), and a CSV file containing the adjusted numbers was successfully created.

## DOCUMENTATION:
None needed.

* Bugfix:  Pass domain name to python plotting scripts (#783)

## DESCRIPTION OF CHANGES: 
PR #[763](#763) introduced the experiment variable `POST_OUTPUT_DOMAIN_NAME` and used it to modify the names of the UPP output files.  This PR adds `POST_OUTPUT_DOMAIN_NAME` as an argument to the python plotting scripts to enable those scripts to form the proper UPP output file names to read in.  It also modifies the batch scripts (`qsub_job.sh`, `qsub_job_diff.sh`, `sq_job.sh`, and `sq_job_diff.sh`) to add the new argument to the calls to the python plotting scripts that are called.

## TESTS CONDUCTED: 
Both python plotting scripts `plot_allvars.py` and `plot_allvars_diff.py` were run on Hera on preexisting UPP output files and completed successfully.  The batch scripts `sq_job.sh` and `sq_job_diff.sh` were also run successfully on Hera (`qsub_job.sh` and `qsub_job_diff.sh` were not run on e.g. Cheyenne).

## DEPENDENCIES:
PR # in `ufs-srweather-app`.

## DOCUMENTATION:
Modifications to the documentation are in PR #.

## ISSUE: 
Partially resolves issue #781 (still need to introduce tests in `regional_workflow` that run the plotting scripts).

## CONTRIBUTORS: 
@mark-a-potts pointed out the problem and first made the necessary fixes in the `NOAA-EPIC` fork of `regional_workflow`; @gsketefian introduced those changes to this PR and made edits to the comments and documentation.

* created new non-compact grids

* fixed typo

* Update scripts to work with the latest hashes of UFS_UTILS and UPP (#775)

* update input namelist of chgres_cube

* update diag_table templates

* update scripts

* back to original

* specify miniconda version on Jet

Co-authored-by: Gillian Petro <96886803+gspetro-NOAA@users.noreply.github.com>
Co-authored-by: Natalie Perlin <68030316+natalie-perlin@users.noreply.github.com>
Co-authored-by: Natalie Perlin <perlin.natalie@gmail.com>
Co-authored-by: gsketefian <31046882+gsketefian@users.noreply.github.com>
Co-authored-by: Chan-Hoo.Jeon-NOAA <60152248+chan-hoo@users.noreply.github.com>
  • Loading branch information
@EdwardSnyder-NOAA @gspetro-NOAA
@natalie-perlin @gsketefian @chan-hoo
6 people authored Jun 10, 2022
1 parent 980d592 commit cf8a20c
Show file tree
Hide file tree
Showing 35 changed files with 2,322 additions and 1,262 deletions.
4 changes: 2 additions & 2 deletions modulefiles/tasks/jet/miniconda_regional_workflow
View file
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
#%Module
module use -a /contrib/miniconda3/modulefiles
module load miniconda3
module use /contrib/miniconda3/modulefiles
module load miniconda3/4.5.12

setenv SRW_ENV regional_workflow
1 change: 0 additions & 1 deletion scripts/exregional_make_ics.sh
View file
Original file line number Diff line number Diff line change
Expand Up @@ -519,7 +519,6 @@ fi
#
settings="
'config': {
'fix_dir_input_grid': ${FIXgsm},
'fix_dir_target_grid': ${FIXLAM},
'mosaic_file_target_grid': ${FIXLAM}/${CRES}${DOT_OR_USCORE}mosaic.halo$((10#${NH4})).nc,
'orog_dir_target_grid': ${FIXLAM},
Expand Down
1 change: 0 additions & 1 deletion scripts/exregional_make_lbcs.sh
View file
Original file line number Diff line number Diff line change
Expand Up @@ -443,7 +443,6 @@ list file has not specified for this external LBC model (EXTRN_MDL_NAME_LBCS):
#
settings="
'config': {
'fix_dir_input_grid': ${FIXgsm},
'fix_dir_target_grid': ${FIXLAM},
'mosaic_file_target_grid': ${FIXLAM}/${CRES}${DOT_OR_USCORE}mosaic.halo$((10#${NH4})).nc,
'orog_dir_target_grid': ${FIXLAM},
Expand Down
Loading

1 comment on commit cf8a20c

@gsketefian
Copy link
Collaborator

@gsketefian gsketefian commented on cf8a20c Jun 10, 2022 via email

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please sign in to comment.