[develop]: Use yaml format for specifying experiment parameters (ufs-…

…community#778)

* Construct var_defns components from dictionary.

* Bring back config_defaults.yaml

* Add support for sourcing yaml file into shell script.

* Remove newline for printing config, json config fix.

* Make QUILTING a sub-dictionary in predef_grids

* Reorganize config_defaults.yaml by task and feature.

* Bug fix with QUILTING=true.

* Structure a dictionary based on a template dictionary.

* Convert all WE2E config files to yaml.

* Take care of problematic chars when converting to shell string.

* Process only selected keys of config.

* Add symlinked yaml config files.

* Actually use yaml config files for WE2E tests.

* Delete all shell WE2E configs.

* Don't check for single quotes in test description.

* Make WE2E work with yaml configs.

* Make yaml default config format.

* Bug fix in run_WE2E script.

* Add utility to check validity of yaml config file.

* Add config utility interface in ush directory.

* Remove unused check_expt_config_vars script.

* Add description to default config.

* Reorganize source_config.

* Add XML as one of the config formats.

* Update custom_ESGgrid config.

* Bug fix due to update.

* Change ensemble seed.

* Change POST_OUTPUT group due to merge.

* Make xml and ini configs work.

* Maintain config structure down to var_defns.

* Add function to load structured shell config, put description under metadata

* Flatten dicts before importing env now that shell config is structured.

* Support python regex for selecting dict keys.

* Add capability of sourcing task specific portion of config file.

* Access var_defns via env variable.

* Make names of tasks consistent with ex- and j- job script names.

* Append pid to temp file.

* Prettify user config, don't use " in xml texts.

* Compare timestamp of csv vs all files instead of directory.

* Fixes for some pylint suggestions.

* Convert new configs to yaml.

* Format python files with black (no functional change).

* More readable yaml/json formats by using more data types.
Only datetime type is now in quotes.

* More readable yaml config files for WE2E and default configs.

* Make config_defaults itself more readable.

* Correct pyyaml list indentation issue.

* Fix indentation in all config files.

* Use unquoted WTIME in config_defaults

* Cosmotic changes.

* Fix due to merge.

* Make __init__.py clearer.

* Fixes due to merge.

* Minor edits of comments.

* Remove wcoss_dell_p3 from workflow (ufs-community#810)

* remove wcoss_dell_p3

* remove block for tide and gyre

* Replace deprecated NCAR python environment with conda on Cheyenne (ufs-community#812)

* Fix issue on get_extrn_lbcs when FCST_LEN_HRS>=40 with netcdf (ufs-community#814)

* activate b file on hpss for >40h

* add a new we2e test for fcst_len_hrs>40

* reduce fcst time for we2e

* Convert new test case to yaml.

* Fix formatting due to merge.

* Convert new test case to yaml.

* Fix unittest.

* Merge develop

* Remove exception logic from __init__.py

* Minor change to cmd concat.

* Make grid gen methods return dictionary, simplifis code a lot.

* Add a comment why we are suppressing yaml import exception.

* Minor change to beautify unittest output.

* Add status badge for functional tests.

* Reorder tasks in config_default and we2e test cases to match order in FV3LAM.xml

* Keep single quotes and newlines in we2e test description.

* Revert back to not rounding to 10 digits

Co-authored-by: Chan-Hoo.Jeon-NOAA <60152248+chan-hoo@users.noreply.github.com>
Co-authored-by: Michael Kavulich <kavulich@ucar.edu>

.github/workflows/python_func_tests.yaml

@@ -1,4 +1,4 @@
-name: Python unittests
+name: Python functional tests
 on:
   push:
     paths:

.github/workflows/python_unittests.yaml

@@ -1,5 +1,5 @@
 name: Python unittests
-on: [push, pull_request]
+on: [push, pull_request, workflow_dispatch]
 jobs:
 
   python_unittests:

README.md

@@ -1,4 +1,5 @@
 [![Python unittests](https://github.com/ufs-community/regional_workflow/actions/workflows/python_unittests.yaml/badge.svg)](https://github.com/ufs-community/regional_workflow/actions/workflows/python_unittests.yaml)
+[![Python functional tests](https://github.com/ufs-community/regional_workflow/actions/workflows/python_func_tests.yaml/badge.svg)](https://github.com/ufs-community/regional_workflow/actions/workflows/python_func_tests.yaml)
 
 # Regional Workflow
 

scripts/exregional_make_grid.sh

@@ -587,7 +587,7 @@ halo failed."
 #
 #-----------------------------------------------------------------------
 #
-python3  $USHDIR/link_fix.py \
+python3 $USHDIR/link_fix.py \
   --path-to-defns ${GLOBAL_VAR_DEFNS_FP} \
   --file-group "grid" || \
 print_err_msg_exit "\

scripts/exregional_make_orog.sh

@@ -572,7 +572,7 @@ cd_vrfy -
 #
 #-----------------------------------------------------------------------
 #
-python3  $USHDIR/link_fix.py \
+python3 $USHDIR/link_fix.py \
   --path-to-defns ${GLOBAL_VAR_DEFNS_FP} \
   --file-group "orog" || \
 print_err_msg_exit "\

scripts/exregional_make_sfc_climo.sh

@@ -233,7 +233,7 @@ esac
 #
 #-----------------------------------------------------------------------
 #
-python3  $USHDIR/link_fix.py \
+python3 $USHDIR/link_fix.py \
   --path-to-defns ${GLOBAL_VAR_DEFNS_FP} \
   --file-group "sfc_climo" || \
 print_err_msg_exit "\

tests/WE2E/get_WE2Etest_names_subdirs_descs.sh

@@ -405,30 +405,10 @@ function get_WE2Etest_names_subdirs_descs() {
         var_name_at \
         vars_to_extract
 
-  local grid_gen_method \
-        \
-        gfdlgrid_lon_t6_ctr \
-        gfdlgrid_lat_t6_ctr \
-        gfdlgrid_num_cells \
-        gfdlgrid_stretch_fac \
-        gfdlgrid_refine_ratio \
-        gfdlgrid_istart_of_rgnl_dom_on_t6g \
-        gfdlgrid_iend_of_rgnl_dom_on_t6g \
-        gfdlgrid_jstart_of_rgnl_dom_on_t6g \
-        gfdlgrid_jend_of_rgnl_dom_on_t6g \
-        \
-        esggrid_lon_ctr \
-        esggrid_lat_ctr \
-        esggrid_nx \
-        esggrid_ny \
-        esggrid_pazi \
-        esggrid_wide_halo_width \
-        esggrid_delx \
-        esggrid_dely \
-        \
-        nx \
-        ny \
-        dta
+  local dta \
+        nxny \
+        dta_r \
+        nxny_r
 #
 #-----------------------------------------------------------------------
 #
@@ -591,7 +571,11 @@ information on all WE2E tests:
 # statement will be false.
 #
     if [ -z "${generate_csv_file}" ]; then
-      mod_time_subdir=$( stat --format=%Y "${subdir_fp}" )
+      if [ -f "${subdir_fp}/*.yaml" ]; then
+        mod_time_subdir=$( stat --format=%Y "${subdir_fp}"/*.yaml | sort -n | tail -1 )
+      else
+        mod_time_subdir="0"
+      fi
       if [ "${mod_time_subdir}" -gt "${mod_time_csv}" ]; then
         generate_csv_file="TRUE"
         print_info_msg "
@@ -633,7 +617,7 @@ to generate a new CSV file:
 # "config.${test_name}.sh", in which case it will be equal to ${test_name}.
 # Otherwise, it will be a null string.
 #
-      regex_search="^config\.(.*)\.sh$"
+      regex_search="^config\.(.*)\.yaml$"
       test_name_or_null=$( printf "%s\n" "${crnt_item}" | \
                            sed -n -r -e "s/${regex_search}/\1/p" )
 #
@@ -887,7 +871,7 @@ name (test_name) is not 1:
   test_name = \"${test_name}\"
   num_occurrences = ${num_occurrences}
 These configuration files all have the name
-  \"config.${test_name}.sh\"
+  \"config.${test_name}.yaml\"
 and are located in the following category subdirectories under 
 test_configs_basedir:
   subdirs = ( $( printf "\"%s\" " "${subdirs[@]}" ))
@@ -995,80 +979,11 @@ configuration files of the primary WE2E tests...
 # leading spaces, the hash symbol, and possibly another space and append
 # what remains to the local variable test_desc.
 #
-      config_fn="config.${test_name}.sh"
-      test_desc=""
-      while read -r line; do
-
-        regex_search="^[ ]*(#)([ ]{0,1})(.*)"
-        hash_or_null=$( printf "%s" "${line}" | \
-                        sed -n -r -e "s/${regex_search}/\1/p" )
-#
-# If the current line is part of the file header containing the test
-# description, then...
-#
-        if [ "${hash_or_null}" = "#" ]; then
-#
-# Strip from the current line any leading whitespace followed by the
-# hash symbol possibly followed by a single space.  If what remains is
-# empty, it means there are no comments on that line and it is just a
-# separator line.  In that case, simply add a newline to test_desc.
-# Otherwise, append what remains after stripping to what test_desc
-# already contains, followed by a single space in preparation for
-# appending the next (stripped) line.
-#
-          stripped_line=$( printf "%s" "${line}" | \
-                           sed -n -r -e "s/${regex_search}/\3/p" )
-          if [ -z "${stripped_line}" ]; then
-            test_desc="\
-${test_desc}
-
-"
-          else
-            test_desc="\
-${test_desc}${stripped_line} "
-          fi
-#
-# If the current line is not part of the file header containing the test
-# description, break out of the while-loop (and thus stop reading the
-# file).
-#
-        else
-          break
-        fi
-
-      done < "${config_fn}"
-#
-# At this point, test_desc contains a description of the current test.
-# Note that:
-#
-# 1) It will be empty if the configuration file for the current test
-#    does not contain a header describing the test.
-# 2) It will contain newlines if the description header contained lines
-#    that start with the hash symbol and contain no other characters.
-#    These are used to delimit paragraphs within the description.
-# 3) It may contain leading and trailing whitespace.
-#
-# Next, for clarity, we remove any leading and trailing whitespace using
-# bash's pattern matching syntax.
-#
-# Note that the right-hand sides of the following two lines are NOT
-# regular expressions.  They are expressions that use bash's pattern
-# matching syntax (gnu.org/software/bash/manual/html_node/Pattern-Matching.html,
-# wiki.bash-hackers.org/syntax/pattern) used in substring removal
-# (tldp.org/LDP/abs/html/string-manipulation.html).  For example,
-#
-#   ${var%%[![:space:]]*}
-#
-# says "remove from var its longest substring that starts with a non-
-# space character".
-#
-# First remove leading whitespace.
-#
-      test_desc="${test_desc#"${test_desc%%[![:space:]]*}"}"
-#
-# Now remove trailing whitespace.
-#
-      test_desc="${test_desc%"${test_desc##*[![:space:]]}"}"
+      config_fn="config.${test_name}.yaml"
+      config_fp="${test_configs_basedir}/$subdir/$config_fn"
+      test_desc="$(config_to_yaml_str $config_fp -k "metadata")"
+      test_desc="${test_desc:27}"
+      test_desc="${test_desc::${#test_desc}-1}"
 #
 # Finally, save the description of the current test as the next element
 # of the array prim_test_descs.
@@ -1079,10 +994,11 @@ ${test_desc}${stripped_line} "
 # variables specified in "vars_to_extract".  Then save the value in the
 # arrays specified by "prim_array_names_vars_to_extract".
 #
+      config_content=$(config_to_shell_str $config_fp)
       for (( k=0; k<=$((num_vars_to_extract-1)); k++ )); do
 
         var_name="${vars_to_extract[$k]}"
-        cmd=$( grep "^[ ]*${var_name}=" "${config_fn}" )
+        cmd=$( grep "^[ ]*${var_name}=" <<< "${config_content}" )
         eval $cmd
 
         if [ -z "${!var_name+x}" ]; then
@@ -1689,49 +1605,8 @@ containing information on all WE2E tests:
   fi
 
   if [ ! -z "${outvarname_test_descs}" ]; then
-#
-# We want to treat all characters in the test descriptions literally
-# when evaluating the array specified by outvarname_test_descs below 
-# using the eval function because otherwise, characters such as "$", 
-# "(", ")", etc will be interpreted as indicating the value of a variable, 
-# the start of an array, the end of an array, etc, and lead to errors.  
-# Thus, below, when forming the array that will be passed to eval, we 
-# will surround each element of the local array test_descs in single 
-# quotes.  However, the test descriptions themselves may include single 
-# quotes (e.g. when a description contains a phrase such as "Please see 
-# the User's Guide for...").  In order to treat these single quotes 
-# literally (as opposed to as delimiters indicating the start or end of 
-# array elements), we have to pass them as separate strings by replacing 
-# each single quote with the following series of characters:
-#
-#   '"'"'
-#
-# In this, the first single quote indicates the end of the previous
-# single-quoted string, the "'" indicates a string containing a literal
-# single quote, and the last single quote inidicates the start of the
-# next single-quoted string.
-#
-# For example, let's assume there are only two WE2E tests to consider.
-# Assume the description of the first is
-#
-#   Please see the User's Guide.
-#
-# and that of the second is:
-#
-#   See description of ${DOT_OR_USCORE} in the configuration file.
-#
-# Then, if outvarname_test_descs is set to "some_array", the exact string 
-# we want to pass to eval is:
-#
-#   some_array=('Please see the User'"'"'s Guide.' 'See description of ${DOT_OR_USCORE} in the configuration file.')
-#
-    test_descs_esc_sq=()
-    for (( i=0; i<=$((num_tests-1)); i++ )); do
-      test_descs_esc_sq[$i]=$( printf "%s" "${test_descs[$i]}" | \
-                               sed -r -e "s/'/'\"'\"'/g" )
-    done
-    test_descs_str="( "$( printf "'%s' " "${test_descs_esc_sq[@]}" )")"
-    eval ${outvarname_test_descs}="${test_descs_str}"
+    test_descs_str="( "$( printf "'%s' " "${test_descs[@]}" )")"
+    eval ${output_varname_test_descs}="${test_descs_str}"
   fi
 #
 #-----------------------------------------------------------------------

tests/WE2E/run_WE2E_tests.sh

@@ -712,7 +712,7 @@ for (( i=0; i<=$((num_tests_to_run-1)); i++ )); do
 # Generate the full path to the current WE2E test's configuration file.
 # Then ensure that this file exists.
 #
-  test_config_fp="${avail_WE2E_test_configs_basedir}/${test_subdir}/config.${test_name}.sh"
+  test_config_fp="${avail_WE2E_test_configs_basedir}/${test_subdir}/config.${test_name}.yaml"
 
   if [ ! -f "${test_config_fp}" ]; then
     print_err_msg_exit "\
@@ -732,8 +732,8 @@ Please correct and rerun."
 #
 #-----------------------------------------------------------------------
 #
-  . ${ushdir}/config_defaults.sh
-  . ${test_config_fp}
+  source_config ${ushdir}/config_defaults.yaml
+  source_config ${test_config_fp}
 #
 #-----------------------------------------------------------------------
 #
@@ -742,7 +742,7 @@ Please correct and rerun."
 # have.  Once this variable is constructed, we will write its contents
 # to the generic configuration file that the experiment generation script
 # reads in (specified by the variable EXPT_CONFIG_FN in the default 
-# configuration file config_defaults.sh sourced above) and then run that
+# configuration file config_defaults.yaml sourced above) and then run that
 # script to generate an experiment for the current WE2E test.
 #
 # We name the multiline variable that will contain the contents of the
@@ -760,7 +760,7 @@ Please correct and rerun."
 # that depend on the input arguments to this script (as opposed to 
 # variable settings in the test configuration file specified by 
 # test_config_fp).  Note that any values of these parameters specified 
-# in the default experiment configuration file (config_defaults.sh) 
+# in the default experiment configuration file (config_defaults.yaml) 
 # or in the test configuraiton file (test_config_fp) that were sourced 
 # above will be overwritten by the settings below.
 #
@@ -846,7 +846,7 @@ VERBOSE=\"${VERBOSE}\""
 # The following section is a copy of this WE2E test's configuration file.
 #
 "
-  expt_config_str=${expt_config_str}$( cat "${test_config_fp}" )
+  expt_config_str=${expt_config_str}$( config_to_shell_str "${test_config_fp}" )
   expt_config_str=${expt_config_str}"
 #
 # End of section from this test's configuration file.
@@ -1172,14 +1172,11 @@ MAXTRIES_RUN_POST=\"${MAXTRIES_RUN_POST}\""
   fi
 #
 #-----------------------------------------------------------------------
-#
-# Set the full path to the configuration file that the experiment 
-# generation script reads in.  Then write the contents of expt_config_str 
-# to that file.
-#
+# Write content to a temporary config file
 #-----------------------------------------------------------------------
 #
-  expt_config_fp="$ushdir/${EXPT_CONFIG_FN}"
+  temp_file="$PWD/_config_temp_.sh"
+  expt_config_fp="${temp_file}"
   printf "%s" "${expt_config_str}" > "${expt_config_fp}"
 #
 #-----------------------------------------------------------------------
@@ -1276,6 +1273,19 @@ exist or is not a directory:
 #
 #-----------------------------------------------------------------------
 #
+# Set the full path to the configuration file that the experiment 
+# generation script reads in.  Then write the contents of expt_config_str 
+# to that file.
+#
+#-----------------------------------------------------------------------
+#
+  expt_config_fp="$ushdir/${EXPT_CONFIG_FN}"
+  ext="${EXPT_CONFIG_FN##*.}"
+  config_to_str "${ext}" "${temp_file}" -t "$ushdir/config_defaults.yaml" >"${expt_config_fp}"
+  rm -rf "${temp_file}"
+#
+#-----------------------------------------------------------------------
+#
 # Call the experiment generation script to generate an experiment 
 # directory and a rocoto workflow XML for the current WE2E test to run. 
 #