Sub-hourly post-processing enabled by some new config.sh entries. (#439)

## DESCRIPTION OF CHANGES: 
Modified the Jinja-formatted FV3LAM_wflow.xml template workflow to accommodate sub-hourly post-processing tasks that rely on sub-hourly FV3 output as a dependency. All changes are _additions_ to existing code and include the addition of a few keyword variables in the config.sh script. These new flags include...

- SUB_HOURLY_POST: a logical flag indicating whether nor not sub-hourly post-processing is to be used
- DT_SUBHOURLY_POST_MNTS: the increment in minutes to sub-divide the hour

Additional post-processing tasks were added to FV3LAM_wflow.xml to account for the different FV3 output file names depending on whether sub-hourly FV3 output is used (the first FV3 output file has a different naming structure than the remaining output files).

setup.sh was updated to check whether valid entries were used for these two variables and also check that DT_ATMOS divides evenly into DT_SUBHOURLY_POST_MNTS so that the FV3 output is consistent with the requested frequency of UPP output. config_defaults.sh and valid_param_vals.sh were also updated accordingly.

## TESTS CONDUCTED: 
Have run generate_FV3LAM_wflow.sh on a large variety of settings of SUB_HOURLY_POST and DT_SUBHOURLY_POST_MNTS. Note that setup.sh is configured such that DT_SUBHOURLY_POST_MNTS = 0 will cause SUB_HOURLY_POST to be ignored. I have successfully tested cases in which DT_ATMOS _does not_ divide evenly into DT_SUBHOURLY_POST_MNTS and when DT_SUBHOURLY_POST_MNTS is specified as anything other than a two-digit value (strings vs. open integers both work).

The resulting workflows run successfully with rocotorun and output no error messages.

## ISSUE: 
Resolves issue #434

## CONTRIBUTORS:
@gsketefian.  Contributions:

1) Fixed bug in setup.sh in the test that checks whether DT_SUBHOURLY_POST_MNTS is set to 0:  should use the -eq operator instead of ==.
2) Fixed bug in the jinja XML template for rocoto (FV3LAM_wflow.xml) as follows:  rearranged the post-processing tasks so that the post task is run for only the first minute of the last hour (e.g. if the forecast is 3 hours long, post is run for 3:00 but not for 3:15, 3:30, etc).
3) Ran the following 3 WE2E tests [note that tests (b) and (c) are not yet in the regional_workflow repo and will be included in a future PR]:
  a) **grid_RRFS_CONUS_25km_ics_HRRR_lbcs_RAP_suite_RRFS_v1beta**.  This is without subhourly post-processing, i.e. SUB_HOURLY_POST is set to "FALSE".
  b) **subhourly_post**.  This is with subhourly post-processing, i.e. SUB_HOURLY_POST set is set to "TRUE" (with DT_SUBHOURLY_POST_MNTS set to "12" minutes).
  c) **subhourly_post_ensemble_2mems**.  This is with subhourly post-processing and with ensemble forecasts enabled, i.e. SUB_HOURLY_POST and DO_ENSEMBLES are both set to "TRUE" (with DT_SUBHOURLY_POST_MNTS is set to "12" minutes and NUM_ENS_MEMBERS set to "2" members).  This test is run because the changes in the jinja XML template FV3LAM_wflow.xml needed to add subhourly post invovle code that executes ensemble forecasts.
**All three tests were successful.**

Note that this PR changes the names of the grib2 files that UPP generates such that they now always include the minutes -- regardless of whether SUB_HOURLY_POST is set to "TRUE" or "FALSE" (if set to "FALSE", the minutes are always "00").  For example, previously, the grib2 file for forecast hour 1 was named `rrfs.t00z.bgdawpf001.tm00.grib2`; henceforth, it will be named `rrfs.t00z.bgdawpf00100.tm00.grib2`.

jobs/JREGIONAL_RUN_FCST

@@ -73,6 +73,7 @@ mkdir_vrfy -p ${run_dir}/RESTART
 #
 #-----------------------------------------------------------------------
 #
+
 $SCRIPTSDIR/exregional_run_fcst.sh \
   cdate="${CDATE}" \
   cycle_dir="${CYCLE_DIR}" \

jobs/JREGIONAL_RUN_POST

@@ -88,7 +88,7 @@ else
 fi
 mkdir_vrfy -p "${postprd_dir}"
 
-fhr_dir="${postprd_dir}/$fhr"
+fhr_dir="${postprd_dir}/$fhr$fmn"
 check_for_preexist_dir_file "${fhr_dir}" "delete"
 mkdir_vrfy -p "${fhr_dir}"
 
@@ -110,6 +110,19 @@ fi
 #
 #-----------------------------------------------------------------------
 #
+# check if all necessary environmental variables are included
+#
+#-----------------------------------------------------------------------
+#
+if [ -n "${fmn}" ] && [ -z "${DT_ATMOS}" ]; then
+  print_err_msg_exit "\
+ERROR: Environmental variable \$fmn is present, so \$dt_atmos is also required
+to be set (use same value as in FV3 namelist). This is needed so that
+UPP will process the correct file for the first output timestep model fields."
+fi
+#
+#-----------------------------------------------------------------------
+#
 # Call the ex-script for this J-job and pass to it the necessary varia-
 # bles.
 #
@@ -120,7 +133,9 @@ $SCRIPTSDIR/exregional_run_post.sh \
   run_dir="${run_dir}" \
   postprd_dir="${postprd_dir}" \
   fhr_dir="${fhr_dir}" \
-  fhr="${fhr}" || \
+  fhr="${fhr}" \
+  fmn="${fmn}" \
+  dt_atmos="${DT_ATMOS}" || \
 print_err_msg_exit "\
 Call to ex-script corresponding to J-job \"${scrfunc_fn}\" failed."
 #

scripts/exregional_run_fcst.sh

@@ -451,7 +451,10 @@ fi
 create_model_configure_file \
   cdate="$cdate" \
   nthreads=${OMP_NUM_THREADS:-1} \
-  run_dir="${run_dir}" || print_err_msg_exit "\
+  run_dir="${run_dir}" \
+  sub_hourly_post="${SUB_HOURLY_POST}" \
+  dt_subhourly_post_mnts="${DT_SUBHOURLY_POST_MNTS}" \
+  dt_atmos="${DT_ATMOS}" || print_err_msg_exit "\
 Call to function to create a model configuration file for the current
 cycle's (cdate) run directory (run_dir) failed:
   cdate = \"${cdate}\"

scripts/exregional_run_post.sh

@@ -61,6 +61,8 @@ valid_args=( \
 "postprd_dir" \
 "fhr_dir" \
 "fhr" \
+"fmn" \
+"dt_atmos" \
 )
 process_args valid_args "$@"
 #
@@ -204,20 +206,32 @@ tmmark="tm00"
 #
 #-----------------------------------------------------------------------
 #
-dyn_file="${run_dir}/dynf${fhr}.nc"
-phy_file="${run_dir}/phyf${fhr}.nc"
+mnts_secs_str=""
+if [ "${SUB_HOURLY_POST}" = "TRUE" ]; then
+  if [ ${fhr}${fmn} = "00000" ]; then
+    mnts_secs_str=":"`date --utc --date "${yyyymmdd} ${hh} UTC + ${dt_atmos} seconds" +%M:%S`
+  else
+    mnts_secs_str=":${fmn}:00"
+  fi
+else
+  fmn="00"
+fi
+
+dyn_file="${run_dir}/dynf${fhr}${mnts_secs_str}.nc"
+phy_file="${run_dir}/phyf${fhr}${mnts_secs_str}.nc"
+post_time=$( date --utc --date "${yyyymmdd} ${hh} UTC + ${fhr} hours +${fmn} minutes" "+%Y%m%d%H%M" )
 
-post_time=$( date --utc --date "${yyyymmdd} ${hh} UTC + ${fhr} hours" "+%Y%m%d%H" )
 post_yyyy=${post_time:0:4}
 post_mm=${post_time:4:2}
 post_dd=${post_time:6:2}
 post_hh=${post_time:8:2}
+post_mn=${post_time:10:2}
 
 cat > itag <<EOF
 ${dyn_file}
 netcdf
 grib2
-${post_yyyy}-${post_mm}-${post_dd}_${post_hh}:00:00
+${post_yyyy}-${post_mm}-${post_dd}_${post_hh}:${post_mn}:00
 FV3R
 ${phy_file}
 
@@ -268,8 +282,13 @@ The \${fhr} variable contains too few or too many characters:
   fhr = \"$fhr\""
 fi
 
-mv_vrfy BGDAWP.GrbF${post_fhr} ${postprd_dir}/${NET}.t${cyc}z.bgdawpf${fhr}.${tmmark}.grib2
-mv_vrfy BGRD3D.GrbF${post_fhr} ${postprd_dir}/${NET}.t${cyc}z.bgrd3df${fhr}.${tmmark}.grib2
+dot_post_mn_or_null=""
+if [ "${post_mn}" != "00" ]; then
+  dot_post_mn_or_null=".${post_mn}"
+fi
+post_fn_suffix="GrbF${post_fhr}${dot_post_mn_or_null}"
+mv_vrfy BGDAWP.${post_fn_suffix} ${postprd_dir}/${NET}.t${cyc}z.bgdawpf${fhr}${post_mn}.${tmmark}.grib2
+mv_vrfy BGRD3D.${post_fn_suffix} ${postprd_dir}/${NET}.t${cyc}z.bgrd3df${fhr}${post_mn}.${tmmark}.grib2
 
 #Link output for transfer to Jet
 # Should the following be done only if on jet??
@@ -280,10 +299,10 @@ mv_vrfy BGRD3D.GrbF${post_fhr} ${postprd_dir}/${NET}.t${cyc}z.bgrd3df${fhr}.${tm
 # instead of calling sed.
 start_date=$( echo "${cdate}" | sed 's/\([[:digit:]]\{2\}\)$/ \1/' )
 basetime=$( date +%y%j%H%M -d "${start_date}" )
-ln_vrfy -fs ${postprd_dir}/${NET}.t${cyc}z.bgdawpf${fhr}.${tmmark}.grib2 \
-            ${postprd_dir}/BGDAWP_${basetime}f${fhr}00
-ln_vrfy -fs ${postprd_dir}/${NET}.t${cyc}z.bgrd3df${fhr}.${tmmark}.grib2 \
-            ${postprd_dir}/BGRD3D_${basetime}f${fhr}00
+ln_vrfy -fs ${postprd_dir}/${NET}.t${cyc}z.bgdawpf${fhr}${post_mn}.${tmmark}.grib2 \
+            ${postprd_dir}/BGDAWP_${basetime}f${fhr}${post_mn}
+ln_vrfy -fs ${postprd_dir}/${NET}.t${cyc}z.bgrd3df${fhr}${post_mn}.${tmmark}.grib2 \
+            ${postprd_dir}/BGRD3D_${basetime}f${fhr}${post_mn}
 
 rm_vrfy -rf ${fhr_dir}
 #

ush/config_defaults.sh

@@ -375,13 +375,23 @@ WFLOW_LAUNCH_LOG_FN="log.launch_FV3LAM_wflow"
 # FCST_LEN_HRS:
 # The length of each forecast, in integer hours.
 #
+# SUB_HOURLY_POST:
+# Logical flag to indicate whether sub-hourly FV3 output should be
+# post-processed. If TRUE, then DT_SUBHOURLY_POST_MNTS should also be specified.
+#
+# DT_SUB_HOURLY_POST_MNTS:
+# Temporal spacing in minutes for post-processed output files. This needs
+# to be a two-digit integer between 00 and 59. Setting to 00 will override
+# the value of SUB_HOURLY_POST to FALSE.
+#
 #-----------------------------------------------------------------------
 #
 DATE_FIRST_CYCL="YYYYMMDD"
 DATE_LAST_CYCL="YYYYMMDD"
 CYCL_HRS=( "HH1" "HH2" )
 FCST_LEN_HRS="24"
-#
+SUB_HOURLY_POST="FALSE"
+DT_SUBHOURLY_POST_MNTS="00"
 #-----------------------------------------------------------------------
 #
 # Set initial and lateral boundary condition generation parameters.  

ush/create_model_configure_file.sh

@@ -49,6 +49,9 @@ function create_model_configure_file() {
 cdate \
 run_dir \
 nthreads \
+sub_hourly_post \
+dt_subhourly_post_mnts \
+dt_atmos \
   )
   process_args valid_args "$@"
 #
@@ -75,7 +78,7 @@ nthreads \
         dot_quilting_dot \
         dot_print_esmf_dot \
         settings \
-        model_config_fp
+        model_config_fp 
 #
 #-----------------------------------------------------------------------
 #
@@ -170,6 +173,21 @@ run directory (run_dir):
 
   fi
 
+  # IF using sub-hourly FV3/UPP output, then the FV3 output interval needs to be specified in
+  # units of multiples of model time steps (nsout). nsout is guaranteed to be an integer because 
+  # consistency between ${DT_SUBHOURLY_POST_MNTS} and ${DT_ATMOS} was enforced at the time of script generation.
+  # ELSE only nfhout is used and set to 1-hour output.
+  if [ "${sub_hourly_post}" = "TRUE" ]; then
+   nsout=$(( dt_subhourly_post_mnts*60 / dt_atmos ))
+   nfhout=0
+  else
+   nfhout=1
+   nsout=-1
+  fi
+  settings="${settings}
+  'nfhout': ${nfhout}
+  'nsout': ${nsout}"
+
   print_info_msg $VERBOSE "
 The variable \"settings\" specifying values to be used in the \"${MODEL_CONFIG_FN}\"
 file has been set as follows:

ush/generate_FV3LAM_wflow.sh

@@ -282,6 +282,12 @@ settings="\
   'ensmem_indx_name': ${ensmem_indx_name}
   'uscore_ensmem_name': ${uscore_ensmem_name}
   'slash_ensmem_subdir': ${slash_ensmem_subdir}
+#
+# Parameters associated with subhourly post-processed output
+#
+  'sub_hourly_post': ${SUB_HOURLY_POST}
+  'delta_min': ${DT_SUBHOURLY_POST_MNTS}
+  'first_fv3_file_tstr': "000:"`date -d "${DATE_FIRST_CYCL} +${DT_ATMOS} seconds" +%M:%S`
 " # End of "settings" variable.
 
 print_info_msg $VERBOSE "

ush/setup.sh

@@ -365,6 +365,33 @@ fi
 #
 #-----------------------------------------------------------------------
 #
+# Make sure that SUB_HOURLY_POST is set to a valid value.
+#
+#-----------------------------------------------------------------------
+#
+check_var_valid_value "SUB_HOURLY_POST" "valid_vals_SUB_HOURLY_POST"
+#
+# Set SUB_HOURLY_POST to either "TRUE" or "FALSE" so we don't have to consider
+# other valid values later on.
+#
+SUB_HOURLY_POST=${SUB_HOURLY_POST^^}
+if [ "$SUB_HOURLY_POST" = "TRUE" ] || \
+   [ "$SUB_HOURLY_POST" = "YES" ]; then
+  SUB_HOURLY_POST="TRUE"
+elif [ "$SUB_HOURLY_POST" = "FALSE" ] || \
+     [ "$SUB_HOURLY_POST" = "NO" ]; then
+  SUB_HOURLY_POST="FALSE"
+fi
+
+if [ "${DT_SUBHOURLY_POST_MNTS}" -eq "0" ]; then
+  SUB_HOURLY_POST="FALSE"
+  print_info_msg "NOTE: since you have set DT_SUBHOURLY_POST_MNTS to '00', then 
+  SUB_HOURLY_POST is being overwritten to FALSE. If you do not want this, you 
+  must set DT_SUBHOURLY_POST_MNTS to something other than '00.'"
+fi
+#
+#-----------------------------------------------------------------------
+#
 # Make sure that DOT_OR_USCORE is set to a valid value.
 #
 #-----------------------------------------------------------------------
@@ -581,6 +608,20 @@ fi
 #
 #-----------------------------------------------------------------------
 #
+# Check that DT_SUBHOURLY_POST_MNTS is a string consisting of exactly 2 digits between "00" and "59"
+#
+#-----------------------------------------------------------------------
+#
+min_or_null=$( printf "%s" "${DT_SUBHOURLY_POST_MNTS}" | \
+                sed -n -r -e "s/^([0-5][0-9])$/\1/p" )
+if [ -z "${min_or_null}" ]; then
+  print_err_msg_exit "\
+DT_SUBHOURLY_POST_MNTS must be a 2-digit integer between 00 and 59 inclusive, as "MM".
+  DT_SUBHOURLY_POST_MNTS = \"${DT_SUBHOURLY_POST_MNTS}\""
+fi
+#
+#-----------------------------------------------------------------------
+#
 # Check that all elements of CYCL_HRS are strings consisting of exactly
 # 2 digits that are between "00" and "23", inclusive.
 #
@@ -975,6 +1016,22 @@ The forecast model main time step (DT_ATMOS) is set to a null string:
 Please set this to a valid numerical value in the user-specified experiment
 configuration file (EXPT_CONFIG_FP) and rerun:
   EXPT_CONFIG_FP = \"${EXPT_CONFIG_FP}\""
+else
+  if [ "${SUB_HOURLY_POST}" = "TRUE" ]; then
+    ((rem = (DT_SUBHOURLY_POST_MNTS*60) % DT_ATMOS))
+    if [ ${rem} -ne 0 ]; then
+      print_err_msg_exit "\
+When performing sub-hourly post (i.e. SUB_HOURLY_POST set to \"TRUE\"), the time
+interval specified by DT_SUBHOURLY_POST_MNTS (after converting to seconds) must be evenly divisible 
+by the time step DT_ATMOS used in the forecast model, i.e. the remainder (rem) must 
+be zero.  In this case, it is not:
+  SUB_HOURLY_POST = \"${SUB_HOURLY_POST}\"
+  DT_SUBHOURLY_POST_MNTS = \"${DT_SUBHOURLY_POST_MNTS}\"
+  DT_ATMOS = \"${DT_ATMOS}\"
+  rem = \$(( (DT_SUBHOURLY_POST_MNTS*60) %% DT_ATMOS )) = $rem
+Please reset DT_SUBHOURLY_POST_MNTS and/or DT_ATMOS so that the remainder is zero."
+    fi
+  fi
 fi
 
 if [ -z "${LAYOUT_X}" ]; then