Create separate GSI monitor jobs and remove from vrfy job (#1983)

This PR will move the GSI monitor tasks from the `vrfy` job into their own separate jobs in
the rocoto mesh. This takes care of part of issue #235.

Changes in this PR:

1. Create new individual GSI monitor jobs in the rocoto mesh to match how it is done in
operations. The new jobs are: `gdasverfozn`, `gdasverfrad`, `gdasvminmon`, and `gfsvminmon`.
The names correspond to the names in operations and the ecf script names. The jobs are only
added to cycled mode, none of the jobs are run in forecast-only mode.
2. Updates to GSI monitor scripts:
  * Update `ush/jjob_header.sh` arguments in JJOB scripts to call new individual job names and
configs.
  * Introduce `COM_*` variables and use of the `generate_com` utility. This results in the outputs
landing in `COM` subfolders, like occurs in operations, instead of landing in the online archive.
  * Some variable cleanup in JJOB and ush scripts, includes cleaning up usage of `DATA` variable
and its removal at the end of the jobs.
  * Add missing exit statements to `JGFS_ATMOS_VMINMON` and `JGDAS_ATMOS_VERFRAD`.
3. Create new rocoto job scripts for the new individual GSI monitor jobs.
4. Create new configs for the new individual GSI monitor jobs.
5. Add new individual GSI monitor jobs to resource configuration, using the same resources as
in operations.
6. Move and rename GSI monitor job switches from `config.vrfy` to `config.base`. Also add
switches to setup system. New switches to control each job being added to the rocoto mesh
are: `DO_VERFOZN`, `DO_VERFRAD`, `DO_VMINMON`
7. Add GSI monitor job output to archival job and into resulting HPSS tarballs or local tarballs.
This now matches what is done in operations.
8. Remove the GSI monitor tasks from the rocoto `vrfy` job and associated `config.vrfy`.

Example output locations and files (not showing contents of oznmon/horiz or oznmon/time for
brevity, see archival script for full list of output files):
```
kate.friedman@dlogin09:/lfs/h2/emc/ptmp/kate.friedman/comrot/testmonitor2> ll gdas.20211221/00/products/atmos/*mon
gdas.20211221/00/products/atmos/minmon:
total 112
-rw-r--r-- 1 kate.friedman emc 14030 Oct 25 15:00 2021122100.costs.txt
-rw-r--r-- 1 kate.friedman emc 84932 Oct 25 15:00 2021122100.cost_terms.txt
-rw-r--r-- 1 kate.friedman emc   808 Oct 25 15:00 2021122100.gnorms.ieee_d
-rw-r--r-- 1 kate.friedman emc   808 Oct 25 15:00 2021122100.reduction.ieee_d
-rw-r--r-- 1 kate.friedman emc    80 Oct 25 15:00 gnorm_data.txt

gdas.20211221/00/products/atmos/oznmon:
total 8
drwxr-sr-x 2 kate.friedman emc 4096 Oct 25 15:05 horiz
drwxr-sr-x 2 kate.friedman emc 4096 Oct 25 15:05 time

gdas.20211221/00/products/atmos/radmon:
total 21036
-rw-r--r-- 1 kate.friedman emc      231 Oct 25 15:12 bad_diag.2021122100
-rw-r--r-- 1 kate.friedman emc     9035 Oct 25 15:12 bad_pen.2021122100
-rw-r--r-- 1 kate.friedman emc     1449 Oct 25 15:12 low_count.2021122100
-rw-r--r-- 1 kate.friedman emc 20523403 Oct 25 15:07 radmon_angle.tar.gz
-rw-r--r-- 1 kate.friedman emc   217272 Oct 25 15:08 radmon_bcoef.tar.gz
-rw-r--r-- 1 kate.friedman emc   502151 Oct 25 15:10 radmon_bcor.tar.gz
-rw-r--r-- 1 kate.friedman emc   264480 Oct 25 15:12 radmon_time.tar.gz
-rw-r--r-- 1 kate.friedman emc      684 Oct 25 15:12 warning.2021122100
kate.friedman@dlogin09:/lfs/h2/emc/ptmp/kate.friedman/comrot/testmonitor2> ll gfs.20211221/00/products/atmos/minmon/
total 88
-rw-r--r-- 1 kate.friedman emc 10530 Oct 25 14:55 2021122100.costs.txt
-rw-r--r-- 1 kate.friedman emc 63882 Oct 25 14:55 2021122100.cost_terms.txt
-rw-r--r-- 1 kate.friedman emc   808 Oct 25 14:55 2021122100.gnorms.ieee_d
-rw-r--r-- 1 kate.friedman emc   608 Oct 25 14:55 2021122100.reduction.ieee_d
-rw-r--r-- 1 kate.friedman emc    80 Oct 25 14:55 gnorm_data.txt
```

Lingering work and considerations:

1. Issue #1925
2. Consider not hardcoding the list of types in archival and perhaps tie it to something reliable or
configurable (maybe into `config.verfozn`?):
`subtyplist="gome_metop-b omi_aura ompslp_npp ompsnp_n20 ompsnp_npp ompstc8_n20 ompstc8_npp sbuv2_n19"`

Resolves #1908

jobs/JGDAS_ATMOS_VERFOZN

@@ -4,7 +4,7 @@
 # Set up environment for GDAS Ozone Monitor job
 #############################################################
 source "${HOMEgfs}/ush/preamble.sh"
-source "${HOMEgfs}/ush/jjob_header.sh" -e "vrfy" -c "base vrfy"
+source "${HOMEgfs}/ush/jjob_header.sh" -e "verfozn" -c "base verfozn"
 
 export OZNMON_SUFFIX=${OZNMON_SUFFIX:-${NET}}
 
@@ -42,15 +42,12 @@ export p_cyc=${pdate:8:2}
 #---------------------------------------------
 # OZN_TANKDIR - WHERE OUTPUT DATA WILL RESIDE
 #
-export OZN_TANKDIR=${OZN_TANKDIR:-$(compath.py ${envir}/${NET}/${gfs_ver})}
-export TANKverf_ozn=${TANKverf_ozn:-${OZN_TANKDIR}/${RUN}.${PDY}/${cyc}/atmos/oznmon}
-export TANKverf_oznM1=${TANKverf_oznM1:-${OZN_TANKDIR}/${RUN}.${P_PDY}/${p_cyc}/atmos/oznmon}
-
 YMD=${PDY} HH=${cyc} generate_com -rx COM_ATMOS_ANALYSIS
+YMD=${PDY} HH=${cyc} generate_com -rx COM_ATMOS_OZNMON
 
-if [[ ! -d ${TANKverf_ozn} ]]; then
-   mkdir -p -m 775 ${TANKverf_ozn}
-fi
+export TANKverf_ozn=${TANKverf_ozn:-${COM_ATMOS_OZNMON}}
+
+if [[ ! -d ${TANKverf_ozn} ]]; then mkdir -p -m 775 ${TANKverf_ozn} ; fi
 
 #---------------------------------------
 #  set up validation file
@@ -77,10 +74,6 @@ err=$?
 ################################
 # Remove the Working Directory
 ################################
-KEEPDATA=${KEEPDATA:-NO}
-cd ${DATAROOT}
-if [ ${KEEPDATA} = NO ] ; then
-  rm -rf ${DATA}
-fi
+[[ "${KEEPDATA}" = "NO" ]] && rm -rf "${DATA}"
 
 exit 0

jobs/JGDAS_ATMOS_VERFRAD

@@ -4,18 +4,13 @@
 # Set up environment for GDAS Radiance Monitor job
 #############################################################
 source "${HOMEgfs}/ush/preamble.sh"
-source "${HOMEgfs}/ush/jjob_header.sh" -e "vrfy" -c "base vrfy"
+source "${HOMEgfs}/ush/jjob_header.sh" -e "verfrad" -c "base verfrad"
 
 export COMPONENT="atmos"
 
-export RAD_DATA_IN=${DATA}
-
 export RADMON_SUFFIX=${RADMON_SUFFIX:-${RUN}}
 export CYCLE_INTERVAL=${CYCLE_INTERVAL:-6}
 
-mkdir -p ${RAD_DATA_IN}
-cd ${RAD_DATA_IN}
-
 ##############################################
 # Specify Execution Areas
 ##############################################
@@ -52,13 +47,15 @@ export p_cyc=${pdate:8:2}
 # COMOUT - WHERE GSI OUTPUT RESIDES
 # TANKverf - WHERE OUTPUT DATA WILL RESIDE
 #############################################
-export TANKverf=${TANKverf:-$(compath.py ${envir}/${NET}/${gfs_ver})}
-export TANKverf_rad=${TANKverf_rad:-${TANKverf}/${RUN}.${PDY}/${cyc}/atmos/radmon}
-export TANKverf_radM1=${TANKverf_radM1:-${TANKverf}/${RUN}.${P_PDY}/${p_cyc}/atmos/radmon}
-
 YMD=${PDY} HH=${cyc} generate_com -rx COM_ATMOS_ANALYSIS
+YMD=${PDY} HH=${cyc} generate_com -rx COM_ATMOS_RADMON
+YMD=${P_PDY} HH=${p_cyc} generate_com -rx COM_ATMOS_RADMON_PREV:COM_ATMOS_RADMON_TMPL
 
-mkdir -p -m 775 ${TANKverf_rad}
+export TANKverf_rad=${TANKverf_rad:-${COM_ATMOS_RADMON}}
+export TANKverf_radM1=${TANKverf_radM1:-${COM_ATMOS_RADMON_PREV}}
+
+if [[ ! -d ${TANKverf_rad} ]]; then mkdir -p -m 775 ${TANKverf_rad} ; fi
+if [[ ! -d ${TANKverf_radM1} ]]; then mkdir -p -m 775 ${TANKverf_radM1} ; fi
 
 ########################################
 # Set necessary environment variables
@@ -89,9 +86,6 @@ fi
 ################################
 # Remove the Working Directory
 ################################
-KEEPDATA=${KEEPDATA:-YES}
-cd ${DATAROOT}
-if [ ${KEEPDATA} = NO ] ; then
-  rm -rf ${RAD_DATA_IN}
-fi
+[[ "${KEEPDATA}" = "NO" ]] && rm -rf "${DATA}"
 
+exit 0

jobs/JGDAS_ATMOS_VMINMON

@@ -4,7 +4,7 @@
 # GDAS Minimization Monitor (MinMon) job
 ###########################################################
 source "${HOMEgfs}/ush/preamble.sh"
-source "${HOMEgfs}/ush/jjob_header.sh" -e "vrfy" -c "base vrfy"
+source "${HOMEgfs}/ush/jjob_header.sh" -e "vminmon" -c "base vminmon"
 
 ###########################################################
 # obtain unique process id (pid) and make temp directories
@@ -40,12 +40,15 @@ export p_cyc=${pdate:8:2}
 #############################################
 # TANKverf - WHERE OUTPUT DATA WILL RESIDE
 #############################################
-export M_TANKverf=${M_TANKverf:-${COM_IN}/${RUN}.${PDY}/${cyc}/atmos/minmon}
-export M_TANKverfM1=${M_TANKverfM1:-${COM_IN}/${RUN}.${P_PDY}/${p_cyc}/atmos/minmon}
-
 YMD=${PDY} HH=${cyc} generate_com -rx COM_ATMOS_ANALYSIS
+YMD=${PDY} HH=${cyc} generate_com -rx COM_ATMOS_MINMON
+YMD=${P_PDY} HH=${p_cyc} generate_com -rx COM_ATMOS_MINMON_PREV:COM_ATMOS_MINMON_TMPL
+
+export M_TANKverf=${M_TANKverf:-${COM_ATMOS_MINMON}}
+export M_TANKverfM1=${M_TANKverfM1:-${COM_ATMOS_MINMON_PREV}}
 
-mkdir -p -m 775 ${M_TANKverf}
+if [[ ! -d ${M_TANKverf} ]]; then mkdir -p -m 775 ${M_TANKverf} ; fi
+if [[ ! -d ${M_TANKverfM1} ]]; then mkdir -p -m 775 ${M_TANKverfM1} ; fi
 
 
 ########################################
@@ -65,10 +68,6 @@ err=$?
 ################################
 # Remove the Working Directory
 ################################
-KEEPDATA=${KEEPDATA:-NO}
-cd ${DATAROOT}
-if [ ${KEEPDATA} = NO ] ; then
-  rm -rf ${DATA}
-fi
+[[ "${KEEPDATA}" = "NO" ]] && rm -rf "${DATA}"
 
 exit 0

jobs/JGFS_ATMOS_VMINMON

@@ -4,7 +4,7 @@
 # GFS Minimization Monitor (MinMon) job
 ###########################################################
 source "${HOMEgfs}/ush/preamble.sh"
-source "${HOMEgfs}/ush/jjob_header.sh" -e "vrfy" -c "base vrfy"
+source "${HOMEgfs}/ush/jjob_header.sh" -e "vminmon" -c "base vminmon"
 
 ###########################################################
 # obtain unique process id (pid) and make temp directories
@@ -39,12 +39,15 @@ export p_cyc=${pdate:8:2}
 # TANKverf - WHERE OUTPUT DATA WILL RESIDE
 #############################################
 YMD=${PDY} HH=${cyc} generate_com -rx COM_ATMOS_ANALYSIS
-YMD=${P_PDY} HH=${p_cyc} generate_com -rx COM_ATMOS_ANALYSIS_PREV:COM_ATMOS_ANALYSIS_TMPL
+YMD=${PDY} HH=${cyc} generate_com -rx COM_ATMOS_MINMON
+YMD=${P_PDY} HH=${p_cyc} generate_com -rx COM_ATMOS_MINMON_PREV:COM_ATMOS_MINMON_TMPL
 
-M_TANKverf=${M_TANKverf:-${COM_ATMOS_ANALYSIS}/minmon}
-export M_TANKverfM1=${M_TANKverfM1:-${COM_ATMOS_ANALYSIS_PREV}/minmon}
+export M_TANKverf=${M_TANKverf:-${COM_ATMOS_MINMON}}
+export M_TANKverfM1=${M_TANKverfM1:-${COM_ATMOS_MINMON_PREV}}
+
+if [[ ! -d ${M_TANKverf} ]]; then mkdir -p -m 775 ${M_TANKverf} ; fi
+if [[ ! -d ${M_TANKverfM1} ]]; then mkdir -p -m 775 ${M_TANKverfM1} ; fi
 
-mkdir -p -m 775 ${M_TANKverf}
 
 ########################################
 # Set necessary environment variables
@@ -63,11 +66,6 @@ err=$?
 ################################
 # Remove the Working Directory
 ################################
-KEEPDATA=${KEEPDATA:-NO}
-cd ${DATAROOT}
-
-if [ ${KEEPDATA} = NO ] ; then
-  rm -rf ${DATA}
-fi
-
+[[ "${KEEPDATA}" = "NO" ]] && rm -rf "${DATA}"
 
+exit 0

jobs/JGLOBAL_ARCHIVE

@@ -18,7 +18,8 @@ YMD=${PDY} HH=${cyc} generate_com -rx COM_ATMOS_ANALYSIS COM_ATMOS_BUFR COM_ATMO
   COM_OBS COM_TOP \
   COM_OCEAN_HISTORY COM_OCEAN_INPUT COM_OCEAN_RESTART COM_OCEAN_XSECT COM_OCEAN_2D COM_OCEAN_3D \
   COM_OCEAN_ANALYSIS \
-  COM_WAVE_GRID COM_WAVE_HISTORY COM_WAVE_STATION
+  COM_WAVE_GRID COM_WAVE_HISTORY COM_WAVE_STATION \
+  COM_ATMOS_OZNMON COM_ATMOS_RADMON COM_ATMOS_MINMON
 
 for grid in "0p25" "0p50" "1p00"; do
   YMD=${PDY} HH=${cyc} GRID=${grid} generate_com -rx "COM_ATMOS_GRIB_${grid}:COM_ATMOS_GRIB_GRID_TMPL"

jobs/rocoto/verfozn.sh

@@ -0,0 +1,22 @@
+#! /usr/bin/env bash
+
+source "${HOMEgfs}/ush/preamble.sh"
+
+###############################################################
+# Source FV3GFS workflow modules
+source "${HOMEgfs}/ush/load_fv3gfs_modules.sh"
+status=$?
+(( status != 0 )) && exit "${status}"
+
+export job="verfozn"
+export jobid="${job}.$$"
+
+###############################################################
+# Execute the JJOB
+echo
+echo "=============== START TO RUN OZMON DATA EXTRACTION ==============="
+
+"${HOMEgfs}/jobs/JGDAS_ATMOS_VERFOZN"
+status=$?
+
+exit "${status}"

jobs/rocoto/verfrad.sh

@@ -0,0 +1,22 @@
+#! /usr/bin/env bash
+
+source "${HOMEgfs}/ush/preamble.sh"
+
+###############################################################
+# Source FV3GFS workflow modules
+source "${HOMEgfs}/ush/load_fv3gfs_modules.sh"
+status=$?
+(( status != 0 )) && exit "${status}"
+
+export job="verfrad"
+export jobid="${job}.$$"
+
+###############################################################
+# Execute the JJOB
+echo
+echo "=============== START TO RUN RADMON DATA EXTRACTION ==============="
+
+"${HOMEgfs}/jobs/JGDAS_ATMOS_VERFRAD"
+status=$?
+
+exit "${status}"

jobs/rocoto/vminmon.sh

@@ -0,0 +1,22 @@
+#! /usr/bin/env bash
+
+source "${HOMEgfs}/ush/preamble.sh"
+
+###############################################################
+# Source FV3GFS workflow modules
+source "${HOMEgfs}/ush/load_fv3gfs_modules.sh"
+status=$?
+(( status != 0 )) && exit "${status}"
+
+export job="vminmon"
+export jobid="${job}.$$"
+
+###############################################################
+# Execute the JJOB
+echo
+echo "=============== START TO RUN MINMON ==============="
+
+"${HOMEgfs}/jobs/J${RUN^^}_ATMOS_VMINMON"
+status=$?
+
+exit "${status}"

jobs/rocoto/vrfy.sh

@@ -22,11 +22,6 @@ CDATEm1=$(${NDATE} -24 "${PDY}${cyc}")
 export CDATEm1
 export PDYm1=${CDATEm1:0:8}
 
-CDATEm1c=$(${NDATE} -06 "${PDY}${cyc}")
-PDYm1c=${CDATEm1c:0:8}
-pcyc=${CDATEm1c:8:2}
-
-
 ###############################################################
 # TODO: We can likely drop support for these dev-only grib1 precip files
 echo
@@ -57,51 +52,6 @@ if [[ "${RUNMOS}" == "YES" && "${CDUMP}" == "gfs" ]]; then
 fi
 
 
-###############################################################
-echo
-echo "=============== START TO RUN RADMON DATA EXTRACTION ==============="
-
-if [[ "${VRFYRAD}" == "YES" && "${CDUMP}" == "${CDFNL}" && "${PDY}${cyc}" != "${SDATE}" ]]; then
-
-    export EXP=${PSLOT}
-    export TANKverf_rad="${TANKverf}/stats/${PSLOT}/${RUN}.${PDY}/${cyc}"
-    export TANKverf_radM1="${TANKverf}/stats/${PSLOT}/${RUN}.${PDYm1c}/${pcyc}"
-    export MY_MACHINE=${machine}
-
-    ${VRFYRADSH}
-
-fi
-
-
-###############################################################
-echo
-echo "=============== START TO RUN OZMON DATA EXTRACTION ==============="
-if [[ "${VRFYOZN}" == "YES" && "${CDUMP}" == "${CDFNL}" && "${PDY}${cyc}" != "${SDATE}" ]]; then
-
-    export EXP=${PSLOT}
-    export TANKverf_ozn="${TANKverf_ozn}/stats/${PSLOT}/${RUN}.${PDY}/${cyc}"
-    export TANKverf_oznM1="${TANKverf_ozn}/stats/${PSLOT}/${RUN}.${PDYm1c}/${pcyc}"
-    export MY_MACHINE=${machine}
-
-    ${VRFYOZNSH}
-
-fi
-
-
-###############################################################
-echo
-echo "=============== START TO RUN MINMON ==============="
-if [[ "${VRFYMINMON}" == "YES" && "${PDY}${cyc}" != "${SDATE}" && "${MODE}" = "cycled" ]]; then
-
-    export M_TANKverfM0="${M_TANKverf}/stats/${PSLOT}/${RUN}.${PDY}/${cyc}"
-    export M_TANKverfM1="${M_TANKverf}/stats/${PSLOT}/${RUN}.${PDYm1c}/${pcyc}"
-    export MY_MACHINE=${machine}
-
-    ${VRFYMINSH}
-
-fi
-
-
 ################################################################################
 echo
 echo "=============== START TO RUN CYCLONE TRACK VERIFICATION ==============="

parm/config/gfs/config.base.emc.dyn

@@ -57,6 +57,9 @@ export DO_BUFRSND="NO"     # BUFR sounding products
 export DO_GEMPAK="NO"      # GEMPAK products
 export DO_AWIPS="NO"       # AWIPS products
 export DO_VRFY="YES"       # VRFY step
+export DO_VERFOZN="YES"    # Ozone data assimilation monitoring
+export DO_VERFRAD="YES"    # Radiance data assimilation monitoring
+export DO_VMINMON="YES"    # GSI minimization monitoring
 
 # NO for retrospective parallel; YES for real-time parallel
 #  arch.sh uses REALTIME for MOS.  Need to set REALTIME=YES

parm/config/gfs/config.com

@@ -63,6 +63,8 @@ declare -rx COM_ATMOS_GENESIS_TMPL=${COM_BASE}'/products/atmos/cyclone/genesis_v
 declare -rx COM_ATMOS_TRACK_TMPL=${COM_BASE}'/products/atmos/cyclone/tracks'
 declare -rx COM_ATMOS_GOES_TMPL=${COM_BASE}'/products/atmos/goes_sim'
 declare -rx COM_ATMOS_IMAGERY_TMPL=${COM_BASE}'/products/atmos/imagery'
+declare -rx COM_ATMOS_OZNMON_TMPL=${COM_BASE}'/products/atmos/oznmon'
+declare -rx COM_ATMOS_RADMON_TMPL=${COM_BASE}'/products/atmos/radmon'
 declare -rx COM_ATMOS_MINMON_TMPL=${COM_BASE}'/products/atmos/minmon'
 declare -rx COM_ATMOS_WMO_TMPL=${COM_BASE}'/products/atmos/wmo'
 

parm/config/gfs/config.resources

@@ -14,7 +14,8 @@ if [[ $# -ne 1 ]]; then
     echo "atmensanlinit atmensanlrun atmensanlfinal"
     echo "landanl"
     echo "aeroanlinit aeroanlrun aeroanlfinal"
-    echo "anal sfcanl analcalc analdiag fcst post vrfy fit2obs metp arch cleanup echgres"
+    echo "anal sfcanl analcalc analdiag fcst post echgres"
+    echo "verfozn verfrad vminmon vrfy fit2obs metp arch cleanup"
     echo "eobs ediag eomg eupd ecen esfc efcs epos earc"
     echo "init_chem mom6ic ocnpost"
     echo "waveinit waveprep wavepostsbs wavepostbndpnt wavepostbndpntbll wavepostpnt"
@@ -714,6 +715,34 @@ elif [[ ${step} = "post" ]]; then
     if [[ "${npe_node_post_gfs}" -gt "${npe_node_max}" ]]; then export npe_node_post_gfs=${npe_node_max} ; fi
     export is_exclusive=True
 
+elif [[ ${step} = "verfozn" ]]; then
+
+    export wtime_verfozn="00:05:00"
+    export npe_verfozn=1
+    export nth_verfozn=1
+    export npe_node_verfozn=1
+    export memory_verfozn="1G"
+
+elif [[ ${step} = "verfrad" ]]; then
+
+    export wtime_verfrad="00:20:00"
+    export npe_verfrad=1
+    export nth_verfrad=1
+    export npe_node_verfrad=1
+    export memory_verfrad="5G"
+
+elif [[ ${step} = "vminmon" ]]; then
+
+    export wtime_vminmon="00:05:00"
+    export npe_vminmon=1
+    export nth_vminmon=1
+    export npe_node_vminmon=1
+    export wtime_vminmon_gfs="00:05:00"
+    export npe_vminmon_gfs=1
+    export nth_vminmon_gfs=1
+    export npe_node_vminmon_gfs=1
+    export memory_vminmon="1G"
+
 elif [[ ${step} = "vrfy" ]]; then
 
     export wtime_vrfy="03:00:00"

parm/config/gfs/config.verfozn

@@ -0,0 +1,9 @@
+#! /usr/bin/env bash
+
+########## config.verfozn ##########
+echo "BEGIN: config.verfozn"
+
+# Get task specific resources
+. "${EXPDIR}/config.resources" verfozn
+
+echo "END: config.verfozn"