From 4bc0c9cb70a8f7379a4c62ecaf10cd7c324302c0 Mon Sep 17 00:00:00 2001 From: Christina Holt Date: Tue, 18 Apr 2023 17:46:25 -0600 Subject: [PATCH 01/10] Removing references to TN_* variables --- docs/UsersGuide/source/ConfigWorkflow.rst | 196 ++-------------------- docs/UsersGuide/source/RunSRW.rst | 73 ++++---- 2 files changed, 51 insertions(+), 218 deletions(-) diff --git a/docs/UsersGuide/source/ConfigWorkflow.rst b/docs/UsersGuide/source/ConfigWorkflow.rst index 2f71065b92..5ee5884096 100644 --- a/docs/UsersGuide/source/ConfigWorkflow.rst +++ b/docs/UsersGuide/source/ConfigWorkflow.rst @@ -151,7 +151,7 @@ METplus Parameters * ``SS`` refers to the two-digit valid seconds of the hour ``CCPA_OBS_DIR``: (Default: "") - User-specified location of top-level directory where CCPA hourly precipitation files used by METplus are located. This parameter needs to be set for both user-provided observations and for observations that are retrieved from the NOAA :term:`HPSS` (if the user has access) via the ``TN_GET_OBS_CCPA`` task. (This task is activated in the workflow by setting ``RUN_TASK_GET_OBS_CCPA: true``). + User-specified location of top-level directory where CCPA hourly precipitation files used by METplus are located. This parameter needs to be set for both user-provided observations and for observations that are retrieved from the NOAA :term:`HPSS` (if the user has access) via the ``GET_OBS_CCPA`` task. (This task is activated in the workflow by setting ``RUN_TASK_GET_OBS_CCPA: true``). METplus configuration files require the use of a predetermined directory structure and file names. If the CCPA files are user-provided, they need to follow the anticipated naming structure: ``{YYYYMMDD}/ccpa.t{HH}z.01h.hrap.conus.gb2``, where YYYYMMDD and HH are as described in the note :ref:`above `. When pulling observations from NOAA HPSS, the data retrieved will be placed in the ``CCPA_OBS_DIR`` directory. This path must be defind as ``//ccpa/proc``. METplus is configured to verify 01-, 03-, 06-, and 24-h accumulated precipitation using hourly CCPA files. @@ -159,7 +159,7 @@ METplus Parameters There is a problem with the valid time in the metadata for files valid from 19 - 00 UTC (i.e., files under the "00" directory). The script to pull the CCPA data from the NOAA HPSS (``scripts/exregional_get_obs_ccpa.sh``) has an example of how to account for this and organize the data into a more intuitive format. When a fix is provided, it will be accounted for in the ``exregional_get_obs_ccpa.sh`` script. ``MRMS_OBS_DIR``: (Default: "") - User-specified location of top-level directory where MRMS composite reflectivity files used by METplus are located. This parameter needs to be set for both user-provided observations and for observations that are retrieved from the NOAA :term:`HPSS` (if the user has access) via the ``TN_GET_OBS_MRMS`` task (activated in the workflow by setting ``RUN_TASK_GET_OBS_MRMS: true``). When pulling observations directly from NOAA HPSS, the data retrieved will be placed in this directory. Please note, this path must be defind as ``//mrms/proc``. + User-specified location of top-level directory where MRMS composite reflectivity files used by METplus are located. This parameter needs to be set for both user-provided observations and for observations that are retrieved from the NOAA :term:`HPSS` (if the user has access) via the ``GET_OBS_MRMS`` task (activated in the workflow by setting ``RUN_TASK_GET_OBS_MRMS: true``). When pulling observations directly from NOAA HPSS, the data retrieved will be placed in this directory. Please note, this path must be defind as ``//mrms/proc``. METplus configuration files require the use of a predetermined directory structure and file names. Therefore, if the MRMS files are user-provided, they need to follow the anticipated naming structure: ``{YYYYMMDD}/MergedReflectivityQCComposite_00.50_{YYYYMMDD}-{HH}{mm}{SS}.grib2``, where YYYYMMDD and {HH}{mm}{SS} are as described in the note :ref:`above `. @@ -167,7 +167,7 @@ METplus Parameters METplus is configured to look for a MRMS composite reflectivity file for the valid time of the forecast being verified; since MRMS composite reflectivity files do not always exactly match the valid time, a script (within the main script that retrieves MRMS data from the NOAA HPSS) is used to identify and rename the MRMS composite reflectivity file to match the valid time of the forecast. The script to pull the MRMS data from the NOAA HPSS has an example of the expected file-naming structure: ``scripts/exregional_get_obs_mrms.sh``. This script calls the script used to identify the MRMS file closest to the valid time: ``ush/mrms_pull_topofhour.py``. ``NDAS_OBS_DIR``: (Default: "") - User-specified location of the top-level directory where NDAS prepbufr files used by METplus are located. This parameter needs to be set for both user-provided observations and for observations that are retrieved from the NOAA :term:`HPSS` (if the user has access) via the ``TN_GET_OBS_NDAS`` task (activated in the workflow by setting ``RUN_TASK_GET_OBS_NDAS: true``). When pulling observations directly from NOAA HPSS, the data retrieved will be placed in this directory. Please note, this path must be defined as ``//ndas/proc``. METplus is configured to verify near-surface variables hourly and upper-air variables at 00 and 12 UTC with NDAS prepbufr files. + User-specified location of the top-level directory where NDAS prepbufr files used by METplus are located. This parameter needs to be set for both user-provided observations and for observations that are retrieved from the NOAA :term:`HPSS` (if the user has access) via the ``GET_OBS_NDAS`` task (activated in the workflow by setting ``RUN_TASK_GET_OBS_NDAS: true``). When pulling observations directly from NOAA HPSS, the data retrieved will be placed in this directory. Please note, this path must be defined as ``//ndas/proc``. METplus is configured to verify near-surface variables hourly and upper-air variables at 00 and 12 UTC with NDAS prepbufr files. METplus configuration files require the use of predetermined file names. Therefore, if the NDAS files are user-provided, they need to follow the anticipated naming structure: ``prepbufr.ndas.{YYYYMMDDHH}``, where YYYYMMDDHH is as described in the note :ref:`above `. The script to pull the NDAS data from the NOAA HPSS (``scripts/exregional_get_obs_ndas.sh``) has an example of how to rename the NDAS data into a more intuitive format with the valid time listed in the file name. @@ -285,7 +285,7 @@ Set File Name Parameters Name of the file (a shell script) containing definitions of the primary and secondary experiment variables (parameters). This file is sourced by many scripts (e.g., the J-job scripts corresponding to each workflow task) in order to make all the experiment variables available in those scripts. The primary variables are defined in the default configuration script (``config_defaults.yaml``) and in ``config.yaml``. The secondary experiment variables are generated by the experiment generation script. ``EXTRN_MDL_VAR_DEFNS_FN``: (Default: "extrn_mdl_var_defns") - Name of the file (a shell script) containing the definitions of variables associated with the external model from which :term:`ICs` or :term:`LBCs` are generated. This file is created by the ``TN_GET_EXTRN_*`` task because the values of the variables it contains are not known before this task runs. The file is then sourced by the ``TN_MAKE_ICS`` and ``TN_MAKE_LBCS`` tasks. + Name of the file (a shell script) containing the definitions of variables associated with the external model from which :term:`ICs` or :term:`LBCs` are generated. This file is created by the ``GET_EXTRN_*`` task because the values of the variables it contains are not known before this task runs. The file is then sourced by the ``MAKE_ICS`` and ``MAKE_LBCS`` tasks. ``WFLOW_LAUNCH_SCRIPT_FN``: (Default: "launch_FV3LAM_wflow.sh") Name of the script that can be used to (re)launch the experiment's Rocoto workflow. @@ -386,18 +386,6 @@ Verification Parameters ``GET_OBS``: (Default: "get_obs") Set the name of the Rocoto workflow task used to load proper module files for ``GET_OBS_*`` tasks. Users typically do not need to change this value. -``TN_VX``: (Default: "run_vx") - Set the name of the Rocoto workflow task used to load proper module files for ``VX_*`` tasks. Users typically do not need to change this value. - -``TN_VX_ENSGRID``: (Default: "run_ensgridvx") - Set the name of the Rocoto workflow task that runs METplus grid-to-grid ensemble verification for 1-h accumulated precipitation. Users typically do not need to change this value. - -``TN_VX_ENSGRID_PROB_REFC``: (Default: "run_ensgridvx_prob_refc") - Set the name of the Rocoto workflow task that runs METplus grid-to-grid verification for ensemble probabilities for composite reflectivity. Users typically do not need to change this value. - -``MAXTRIES_VX_ENSGRID_PROB_REFC``: (Default: 1) - Maximum number of times to attempt ``TN_VX_ENSGRID_PROB_REFC``. - .. _NCOModeParms: @@ -427,59 +415,17 @@ A standard set of environment variables has been established for *nco* mode to s ``OPSROOT``: (Default: "") The operations root directory in *nco* mode. -.. _workflow-switches: - -WORKFLOW SWITCHES Configuration Parameters -============================================= - -These parameters set flags that determine whether various workflow tasks should be run. When non-default parameters are selected for the variables in this section, they should be added to the ``workflow_switches:`` section of the ``config.yaml`` file. Note that the ``TN_MAKE_GRID``, ``TN_MAKE_OROG``, and ``TN_MAKE_SFC_CLIMO`` are all :term:`cycle-independent` tasks, i.e., if they are run, they only run once at the beginning of the workflow before any cycles are run. - -Baseline Workflow Tasks --------------------------- - -``RUN_TASK_MAKE_GRID``: (Default: true) - Flag that determines whether to run the grid file generation task (``TN_MAKE_GRID``). If this is set to true, the grid generation task is run and new grid files are generated. If it is set to false, then the scripts look for pre-generated grid files in the directory specified by ``GRID_DIR`` (see :numref:`Section %s ` below). Valid values: ``True`` | ``False`` - -``RUN_TASK_MAKE_OROG``: (Default: true) - Same as ``RUN_TASK_MAKE_GRID`` but for the orography generation task (``TN_MAKE_OROG``). Flag that determines whether to run the orography file generation task (``TN_MAKE_OROG``). If this is set to true, the orography generation task is run and new orography files are generated. If it is set to false, then the scripts look for pre-generated orography files in the directory specified by ``OROG_DIR`` (see :numref:`Section %s ` below). Valid values: ``True`` | ``False`` - -``RUN_TASK_MAKE_SFC_CLIMO``: (Default: true) - Same as ``RUN_TASK_MAKE_GRID`` but for the surface climatology generation task (``TN_MAKE_SFC_CLIMO``). Flag that determines whether to run the surface climatology file generation task (``TN_MAKE_SFC_CLIMO``). If this is set to true, the surface climatology generation task is run and new surface climatology files are generated. If it is set to false, then the scripts look for pre-generated surface climatology files in the directory specified by ``SFC_CLIMO_DIR`` (see :numref:`Section %s ` below). Valid values: ``True`` | ``False`` - -``RUN_TASK_GET_EXTRN_ICS``: (Default: true) - Flag that determines whether to run the ``TN_GET_EXTRN_ICS`` task. - -``RUN_TASK_GET_EXTRN_LBCS``: (Default: true) - Flag that determines whether to run the ``TN_GET_EXTRN_LBCS`` task. - -``RUN_TASK_MAKE_ICS``: (Default: true) - Flag that determines whether to run the ``TN_MAKE_ICS`` task. - -``RUN_TASK_MAKE_LBCS``: (Default: true) - Flag that determines whether to run the ``TN_MAKE_LBCS`` task. - -``RUN_TASK_RUN_FCST``: (Default: true) - Flag that determines whether to run the ``TN_RUN_FCST`` task. - -``RUN_TASK_RUN_POST``: (Default: true) - Flag that determines whether to run the ``TN_RUN_POST`` task. Valid values: ``True`` | ``False`` - -``RUN_TASK_RUN_PRDGEN``: (Default: false) - Flag that determines whether to run the ``TN_RUN_PRDGEN`` task. Valid values: ``True`` | ``False`` - -.. _VXTasks: - Verification Tasks -------------------- ``RUN_TASK_GET_OBS_CCPA``: (Default: false) - Flag that determines whether to run the ``TN_GET_OBS_CCPA`` task, which retrieves the :term:`CCPA` hourly precipitation files used by METplus from NOAA :term:`HPSS`. See :numref:`Section %s ` for additional parameters related to this task. + Flag that determines whether to run the ``GET_OBS_CCPA`` task, which retrieves the :term:`CCPA` hourly precipitation files used by METplus from NOAA :term:`HPSS`. See :numref:`Section %s ` for additional parameters related to this task. ``RUN_TASK_GET_OBS_MRMS``: (Default: false) - Flag that determines whether to run the ``TN_GET_OBS_MRMS`` task, which retrieves the :term:`MRMS` composite reflectivity files used by METplus from NOAA HPSS. See :numref:`Section %s ` for additional parameters related to this task. + Flag that determines whether to run the ``GET_OBS_MRMS`` task, which retrieves the :term:`MRMS` composite reflectivity files used by METplus from NOAA HPSS. See :numref:`Section %s ` for additional parameters related to this task. ``RUN_TASK_GET_OBS_NDAS``: (Default: false) - Flag that determines whether to run the ``TN_GET_OBS_NDAS`` task, which retrieves the :term:`NDAS` PrepBufr files used by METplus from NOAA HPSS. See :numref:`Section %s ` for additional parameters related to this task. + Flag that determines whether to run the ``GET_OBS_NDAS`` task, which retrieves the :term:`NDAS` PrepBufr files used by METplus from NOAA HPSS. See :numref:`Section %s ` for additional parameters related to this task. ``RUN_TASK_VX_GRIDSTAT``: (Default: false) Flag that determines whether to run the grid-stat verification task. The :ref:`MET Grid-Stat tool ` provides verification statistics for a matched forecast and observation grid. See :numref:`Section %s ` for additional parameters related to this task. Valid values: ``True`` | ``False`` @@ -513,9 +459,6 @@ Basic Task Parameters For each workflow task, certain parameter values must be passed to the job scheduler (e.g., Slurm), which submits a job for the task. Typically, users do not need to adjust the default values. - ``TN_MAKE_GRID``: (Default: "make_grid") - Set the name of this :term:`cycle-independent` Rocoto workflow task. Users typically do not need to change this value. - ``NNODES_MAKE_GRID``: (Default: 1) Number of nodes to use for the job. @@ -633,9 +576,6 @@ MAKE_OROG Configuration Parameters Non-default parameters for the ``make_orog`` task are set in the ``task_make_orog:`` section of the ``config.yaml`` file. -``TN_MAKE_OROG``: (Default: "make_orog") - Set the name of this :term:`cycle-independent` Rocoto workflow task. Users typically do not need to change this value. - ``NNODES_MAKE_OROG``: (Default: 1) Number of nodes to use for the job. @@ -654,12 +594,14 @@ Non-default parameters for the ``make_orog`` task are set in the ``task_make_oro ``OMP_NUM_THREADS_MAKE_OROG``: (Default: 6) The number of OpenMP threads to use for parallel regions. + + ``OMP_STACKSIZE_MAKE_OROG``: (Default: "2048m") Controls the size of the stack for threads created by the OpenMP implementation. ``OROG_DIR``: (Default: "") - The directory containing pre-generated orography files to use when ``TN_MAKE_OROG`` is set to false. - + The directory containing pre-generated orography files to use when the ``MAKE_OROG`` task is not meant to run. +f .. _make-sfc-climo: MAKE_SFC_CLIMO Configuration Parameters @@ -667,9 +609,6 @@ MAKE_SFC_CLIMO Configuration Parameters Non-default parameters for the ``make_sfc_climo`` task are set in the ``task_make_sfc_climo:`` section of the ``config.yaml`` file. -``TN_MAKE_SFC_CLIMO``: "make_sfc_climo" - Set the name of this :term:`cycle-independent` Rocoto workflow task. Users typically do not need to change this value. - ``NNODES_MAKE_SFC_CLIMO``: (Default: 2) Number of nodes to use for the job. @@ -692,7 +631,7 @@ Non-default parameters for the ``make_sfc_climo`` task are set in the ``task_mak Controls the size of the stack for threads created by the OpenMP implementation. ``SFC_CLIMO_DIR``: (Default: "") - The directory containing pre-generated surface climatology files to use when ``TN_MAKE_SFC_CLIMO`` is set to false. + The directory containing pre-generated surface climatology files to use when the ``MAKE_SFC_CLIMO`` is not meant to run. .. _task_get_extrn_ics: @@ -708,9 +647,6 @@ Basic Task Parameters For each workflow task, certain parameter values must be passed to the job scheduler (e.g., Slurm), which submits a job for the task. -``TN_GET_EXTRN_ICS``: (Default: "get_extrn_ics") - Set the name of this Rocoto workflow task. Users typically do not need to change this value. - ``NNODES_GET_EXTRN_ICS``: (Default: 1) Number of nodes to use for the job. @@ -791,9 +727,6 @@ Basic Task Parameters For each workflow task, certain parameter values must be passed to the job scheduler (e.g., Slurm), which submits a job for the task. -``TN_GET_EXTRN_LBCS``: (Default: "get_extrn_lbcs") - Set the name of this Rocoto workflow task. Users typically do not need to change this value. - ``NNODES_GET_EXTRN_LBCS``: (Default: 1) Number of nodes to use for the job. @@ -862,9 +795,6 @@ Basic Task Parameters For each workflow task, certain parameter values must be passed to the job scheduler (e.g., Slurm), which submits a job for the task. -``TN_MAKE_ICS``: (Default: "make_ics") - Set the name of this Rocoto workflow task. Users typically do not need to change this value. - ``NNODES_MAKE_ICS``: (Default: 4) Number of nodes to use for the job. @@ -889,7 +819,7 @@ For each workflow task, certain parameter values must be passed to the job sched FVCOM Parameter ------------------- ``USE_FVCOM``: (Default: false) - Flag that specifies whether to update surface conditions in FV3-:term:`LAM` with fields generated from the Finite Volume Community Ocean Model (:term:`FVCOM`). If set to true, lake/sea surface temperatures, ice surface temperatures, and ice placement will be overwritten using data provided by FVCOM. Setting ``USE_FVCOM`` to true causes the executable ``process_FVCOM.exe`` in the ``TN_MAKE_ICS`` task to run. This, in turn, modifies the file ``sfc_data.nc`` generated by ``chgres_cube`` during the ``make_ics`` task. Note that the FVCOM data must already be interpolated to the desired FV3-LAM grid. Valid values: ``True`` | ``False`` + Flag that specifies whether to update surface conditions in FV3-:term:`LAM` with fields generated from the Finite Volume Community Ocean Model (:term:`FVCOM`). If set to true, lake/sea surface temperatures, ice surface temperatures, and ice placement will be overwritten using data provided by FVCOM. Setting ``USE_FVCOM`` to true causes the executable ``process_FVCOM.exe`` in the ``MAKE_ICS`` task to run. This, in turn, modifies the file ``sfc_data.nc`` generated by ``chgres_cube`` during the ``make_ics`` task. Note that the FVCOM data must already be interpolated to the desired FV3-LAM grid. Valid values: ``True`` | ``False`` ``FVCOM_WCSTART``: (Default: "cold") Define if this is a "warm" start or a "cold" start. Setting this to "warm" will read in ``sfc_data.nc`` generated in a RESTART directory. Setting this to "cold" will read in the ``sfc_data.nc`` generated from ``chgres_cube`` in the ``make_ics`` portion of the workflow. Valid values: ``"cold"`` | ``"COLD"`` | ``"warm"`` | ``"WARM"`` @@ -906,9 +836,6 @@ MAKE_LBCS Configuration Parameters Non-default parameters for the ``make_lbcs`` task are set in the ``task_make_lbcs:`` section of the ``config.yaml`` file. -``TN_MAKE_LBCS``: (Default: "make_lbcs") - Set the name of this Rocoto workflow task. Users typically do not need to change this value. - ``NNODES_MAKE_LBCS``: (Default: 4) Number of nodes to use for the job. @@ -942,9 +869,6 @@ Basic Task Parameters For each workflow task, certain parameter values must be passed to the job scheduler (e.g., Slurm), which submits a job for the task. -``TN_RUN_FCST``: (Default: "run_fcst") - Set the name of this Rocoto workflow task. Users typically do not need to change this value. - ``NNODES_RUN_FCST``: (Default: "") Number of nodes to use for the job. This is calculated in the workflow generation scripts, so there is no need to set it in the configuration file. @@ -1016,14 +940,14 @@ Write-Component (Quilting) Parameters ----------------------------------------- .. note:: - The :term:`UPP` (called by the ``TN_RUN_POST`` task) cannot process output on the native grid types ("GFDLgrid" and "ESGgrid"), so output fields are interpolated to a **write component grid** before writing them to an output file. The output files written by the UFS Weather Model use an Earth System Modeling Framework (:term:`ESMF`) component, referred to as the **write component**. This model component is configured with settings in the ``model_configure`` file, as described in `Section 4.2.3 `__ of the UFS Weather Model documentation. + The :term:`UPP` (called by the ``RUN_POST`` task) cannot process output on the native grid types ("GFDLgrid" and "ESGgrid"), so output fields are interpolated to a **write component grid** before writing them to an output file. The output files written by the UFS Weather Model use an Earth System Modeling Framework (:term:`ESMF`) component, referred to as the **write component**. This model component is configured with settings in the ``model_configure`` file, as described in `Section 4.2.3 `__ of the UFS Weather Model documentation. ``QUILTING``: (Default: true) .. attention:: The regional grid requires the use of the write component, so users generally should not need to change the default value for ``QUILTING``. - Flag that determines whether to use the write component for writing forecast output files to disk. If set to true, the forecast model will output files named ``dynf$HHH.nc`` and ``phyf$HHH.nc`` (where ``HHH`` is the 3-digit forecast hour) containing dynamics and physics fields, respectively, on the write-component grid. For example, the output files for the 3rd hour of the forecast would be ``dynf$003.nc`` and ``phyf$003.nc``. (The regridding from the native FV3-LAM grid to the write-component grid is done by the forecast model.) If ``QUILTING`` is set to false, then the output file names are ``fv3_history.nc`` and ``fv3_history2d.nc``, and they contain fields on the native grid. Although the UFS Weather Model can run without quilting, the regional grid requires the use of the write component. Therefore, QUILTING should be set to true when running the SRW App. If ``QUILTING`` is set to false, the ``TN_RUN_POST`` (meta)task cannot run because the :term:`UPP` code called by this task cannot process fields on the native grid. In that case, the ``TN_RUN_POST`` (meta)task will be automatically removed from the Rocoto workflow XML. The :ref:`INLINE POST ` option also requires ``QUILTING`` to be set to true in the SRW App. Valid values: ``True`` | ``False`` + Flag that determines whether to use the write component for writing forecast output files to disk. If set to true, the forecast model will output files named ``dynf$HHH.nc`` and ``phyf$HHH.nc`` (where ``HHH`` is the 3-digit forecast hour) containing dynamics and physics fields, respectively, on the write-component grid. For example, the output files for the 3rd hour of the forecast would be ``dynf$003.nc`` and ``phyf$003.nc``. (The regridding from the native FV3-LAM grid to the write-component grid is done by the forecast model.) If ``QUILTING`` is set to false, then the output file names are ``fv3_history.nc`` and ``fv3_history2d.nc``, and they contain fields on the native grid. Although the UFS Weather Model can run without quilting, the regional grid requires the use of the write component. Therefore, QUILTING should be set to true when running the SRW App. If ``QUILTING`` is set to false, the ``RUN_POST`` (meta)task cannot run because the :term:`UPP` code called by this task cannot process fields on the native grid. In that case, the ``RUN_POST`` (meta)task will be automatically removed from the Rocoto workflow XML. The :ref:`INLINE POST ` option also requires ``QUILTING`` to be set to true in the SRW App. Valid values: ``True`` | ``False`` ``PRINT_ESMF``: (Default: false) Flag that determines whether to output extra (debugging) information from :term:`ESMF` routines. Note that the write component uses ESMF library routines to interpolate from the native forecast model grid to the user-specified output grid (which is defined in the model configuration file ``model_configure`` in the forecast run directory). Valid values: ``True`` | ``False`` @@ -1170,9 +1094,6 @@ Basic Task Parameters For each workflow task, certain parameter values must be passed to the job scheduler (e.g., Slurm), which submits a job for the task. -``TN_RUN_POST``: (Default: "run_post") - Set the name of this Rocoto workflow task. Users typically do not need to change this value. - ``NNODES_RUN_POST``: (Default: 2) Number of nodes to use for the job. @@ -1234,9 +1155,6 @@ Basic Task Parameters --------------------------------- For each workflow task, certain parameter values must be passed to the job scheduler (e.g., Slurm), which submits a job for the task. -``TN_RUN_PRDGEN``: (Default: "run_prdgen") - Set the name of this Rocoto workflow task. Users typically do not need to change this value. - ``NNODES_RUN_PRDGEN``: (Default: 1) Number of nodes to use for the job. @@ -1275,9 +1193,6 @@ GET_OBS_CCPA Configuration Parameters Non-default parameters for the ``get_obs_ccpa`` task are set in the ``task_get_obs_ccpa:`` section of the ``config.yaml`` file. -``TN_GET_OBS_CCPA``: (Default: "get_obs_ccpa") - Set the name of this Rocoto workflow task. Users typically do not need to change this value. See :numref:`Section %s ` for more information about the verification tasks. - ``NNODES_GET_OBS_CCPA``: (Default: 1) Number of nodes to use for the job. @@ -1297,9 +1212,6 @@ GET_OBS_MRMS Configuration Parameters Non-default parameters for the ``get_obs_mrms`` task are set in the ``task_get_obs_mrms:`` section of the ``config.yaml`` file. See :numref:`Section %s ` for more information about the verification tasks. -``TN_GET_OBS_MRMS``: (Default: "get_obs_mrms") - Set the name of this Rocoto workflow task. Users typically do not need to change this value. - ``NNODES_GET_OBS_MRMS``: (Default: 1) Number of nodes to use for the job. @@ -1319,9 +1231,6 @@ GET_OBS_NDAS Configuration Parameters Non-default parameters for the ``get_obs_ndas`` task are set in the ``task_get_obs_ndas:`` section of the ``config.yaml`` file. See :numref:`Section %s ` for more information about the verification tasks. -``TN_GET_OBS_NDAS``: (Default: "get_obs_ndas") - Set the name of this Rocoto workflow task. Users typically do not need to change this value. - ``NNODES_GET_OBS_NDAS``: (Default: 1) Number of nodes to use for the job. @@ -1342,9 +1251,6 @@ VX_GRIDSTAT Configuration Parameters Non-default parameters for the ``run_gridstatvx`` task are set in the ``task_run_vx_gridstat:`` section of the ``config.yaml`` file. -``TN_VX_GRIDSTAT``: (Default: "run_gridstatvx") - Set the name of this Rocoto workflow task. Users typically do not need to change this value. - ``NNODES_VX_GRIDSTAT``: (Default: 1) Number of nodes to use for the job. @@ -1363,9 +1269,6 @@ VX_GRIDSTAT_REFC Configuration Parameters Non-default parameters for the ``run_gridstatvx_refc`` task are set in the ``task_run_vx_gridstat_refc:`` section of the ``config.yaml`` file. -``TN_VX_GRIDSTAT_REFC``: (Default: "run_gridstatvx_refc") - Set the name of this Rocoto workflow task. Users typically do not need to change this value. - ``NNODES_VX_GRIDSTAT``: (Default: 1) Number of nodes to use for the job. @@ -1384,9 +1287,6 @@ VX_GRIDSTAT_RETOP Configuration Parameters Non-default parameters for the ``run_gridstatvx_retop`` task are set in the ``task_run_vx_gridstat_retop:`` section of the ``config.yaml`` file. -``TN_VX_GRIDSTAT_RETOP``: (Default: "run_gridstatvx_retop") - Set the name of this Rocoto workflow task. Users typically do not need to change this value. - ``NNODES_VX_GRIDSTAT``: (Default: 1) Number of nodes to use for the job. @@ -1405,9 +1305,6 @@ VX_GRIDSTAT_03h Configuration Parameters Non-default parameters for the ``run_gridstatvx_03h`` task are set in the ``task_run_vx_gridstat_03h:`` section of the ``config.yaml`` file. -``TN_VX_GRIDSTAT_03h``: (Default: "run_gridstatvx_03h") - Set the name of this Rocoto workflow task. Users typically do not need to change this value. - ``NNODES_VX_GRIDSTAT``: (Default: 1) Number of nodes to use for the job. @@ -1426,9 +1323,6 @@ VX_GRIDSTAT_06h Configuration Parameters Non-default parameters for the ``run_gridstatvx_06h`` task are set in the ``task_run_vx_gridstat_06h:`` section of the ``config.yaml`` file. -``TN_VX_GRIDSTAT_06h``: (Default: "run_gridstatvx_06h") - Set the name of this Rocoto workflow task. Users typically do not need to change this value. - ``NNODES_VX_GRIDSTAT``: (Default: 1) Number of nodes to use for the job. @@ -1447,9 +1341,6 @@ VX_GRIDSTAT_24h Configuration Parameters Non-default parameters for the ``run_gridstatvx_24h`` task are set in the ``task_run_vx_gridstat_24h:`` section of the ``config.yaml`` file. -``TN_VX_GRIDSTAT_24h``: (Default: "run_gridstatvx_24h") - Set the name of this Rocoto workflow task. Users typically do not need to change this value. - ``NNODES_VX_GRIDSTAT``: (Default: 1) Number of nodes to use for the job. @@ -1469,9 +1360,6 @@ VX_POINTSTAT Configuration Parameters Non-default parameters for the ``run_pointstatvx`` task are set in the ``task_run_vx_pointstat:`` section of the ``config.yaml`` file. -``TN_VX_POINTSTAT``: (Default: "run_pointstatvx") - Set the name of this Rocoto workflow task. Users typically do not need to change this value. - ``NNODES_VX_POINTSTAT``: (Default: 1) Number of nodes to use for the job. @@ -1491,33 +1379,18 @@ VX_ENSGRID Configuration Parameters Non-default parameters for the ``run_ensgridvx_*`` tasks are set in the ``task_run_vx_ensgrid:`` section of the ``config.yaml`` file. -``TN_VX_ENSGRID_03h``: (Default: "run_ensgridvx_03h") - Set the name of this Rocoto workflow task. Users typically do not need to change this value. - ``MAXTRIES_VX_ENSGRID_03h``: (Default: 1) Maximum number of times to attempt the task. -``TN_VX_ENSGRID_06h``: (Default: "run_ensgridvx_06h") - Set the name of this Rocoto workflow task. Users typically do not need to change this value. - ``MAXTRIES_VX_ENSGRID_06h``: (Default: 1) Maximum number of times to attempt the task. -``TN_VX_ENSGRID_24h``: (Default: "run_ensgridvx_24h") - Set the name of this Rocoto workflow task. Users typically do not need to change this value. - ``MAXTRIES_VX_ENSGRID_24h``: (Default: 1) Maximum number of times to attempt the task. -``TN_VX_ENSGRID_RETOP``: (Default: "run_ensgridvx_retop") - Set the name of this Rocoto workflow task. Users typically do not need to change this value. - ``MAXTRIES_VX_ENSGRID_RETOP``: (Default: 1) Maximum number of times to attempt the task. -``TN_VX_ENSGRID_PROB_RETOP``: (Default: "run_ensgridvx_prob_retop") - Set the name of this Rocoto workflow task. Users typically do not need to change this value. - ``MAXTRIES_VX_ENSGRID_PROB_RETOP``: (Default: 1) Maximum number of times to attempt the task. @@ -1539,9 +1412,6 @@ VX_ENSGRID_REFC Configuration Parameters Non-default parameters for the ``run_ensgridvx_refc`` task are set in the ``task_run_vx_ensgrid_refc:`` section of the ``config.yaml`` file. -``TN_VX_ENSGRID_REFC``: (Default: "run_ensgridvx_refc") - Set the name of this Rocoto workflow task. Users typically do not need to change this value. - ``NNODES_VX_ENSGRID``: (Default: 1) Number of nodes to use for the job. @@ -1560,9 +1430,6 @@ VX_ENSGRID_MEAN Configuration Parameters Non-default parameters for the ``run_ensgridvx_mean`` task are set in the ``task_run_vx_ensgrid_mean:`` section of the ``config.yaml`` file. -``TN_VX_ENSGRID_MEAN``: (Default: "run_ensgridvx_mean") - Set the name of this Rocoto workflow task. Users typically do not need to change this value. - ``NNODES_VX_ENSGRID_MEAN``: (Default: 1) Number of nodes to use for the job. @@ -1581,9 +1448,6 @@ VX_ENSGRID_MEAN_03h Configuration Parameters Non-default parameters for the ``run_ensgridvx_mean_03h`` task are set in the ``task_run_vx_ensgrid_mean_03h:`` section of the ``config.yaml`` file. -``TN_VX_ENSGRID_MEAN_03h``: (Default: "run_ensgridvx_mean_03h") - Set the name of this Rocoto workflow task. Users typically do not need to change this value. - ``NNODES_VX_ENSGRID_MEAN``: (Default: 1) Number of nodes to use for the job. @@ -1602,9 +1466,6 @@ VX_ENSGRID_MEAN_06h Configuration Parameters Non-default parameters for the ``run_ensgridvx_mean_06h`` task are set in the ``task_run_vx_ensgrid_mean_06h:`` section of the ``config.yaml`` file. -``TN_VX_ENSGRID_MEAN_06h``: (Default: "run_ensgridvx_mean_06h") - Set the name of this Rocoto workflow task. Users typically do not need to change this value. - ``NNODES_VX_ENSGRID_MEAN``: (Default: 1) Number of nodes to use for the job. @@ -1623,9 +1484,6 @@ VX_ENSGRID_MEAN_24h Configuration Parameters Non-default parameters for the ``run_ensgridvx_mean_24h`` task are set in the ``task_run_vx_ensgrid_mean_24h:`` section of the ``config.yaml`` file. -``TN_VX_ENSGRID_MEAN_24h``: (Default: "run_ensgridvx_mean_24h") - Set the name of this Rocoto workflow task. Users typically do not need to change this value. - ``NNODES_VX_ENSGRID_MEAN``: (Default: 1) Number of nodes to use for the job. @@ -1644,9 +1502,6 @@ VX_ENSGRID_PROB Configuration Parameters Non-default parameters for the ``run_ensgridvx_prob`` task are set in the ``task_run_vx_ensgrid_prob:`` section of the ``config.yaml`` file. -``TN_VX_ENSGRID_PROB``: (Default: "run_ensgridvx_prob") - Set the name of this Rocoto workflow task. Users typically do not need to change this value. - ``NNODES_VX_ENSGRID_PROB``: (Default: 1) Number of nodes to use for the job. @@ -1665,9 +1520,6 @@ VX_ENSGRID_PROB_03h Configuration Parameters Non-default parameters for the ``run_ensgridvx_prob_03h`` task are set in the ``task_run_vx_ensgrid_prob_03h:`` section of the ``config.yaml`` file. -``TN_VX_ENSGRID_PROB_03h``: (Default: "run_ensgridvx_prob_03h") - Set the name of this Rocoto workflow task. Users typically do not need to change this value. - ``NNODES_VX_ENSGRID_PROB``: (Default: 1) Number of nodes to use for the job. @@ -1686,9 +1538,6 @@ VX_ENSGRID_PROB_06h Configuration Parameters Non-default parameters for the ``run_ensgridvx_prob_06h`` task are set in the ``task_run_vx_ensgrid_prob_06h:`` section of the ``config.yaml`` file. -``TN_VX_ENSGRID_PROB_06h``: (Default: "run_ensgridvx_prob_06h") - Set the name of this Rocoto workflow task. Users typically do not need to change this value. - ``NNODES_VX_ENSGRID_PROB``: (Default: 1) Number of nodes to use for the job. @@ -1707,9 +1556,6 @@ VX_ENSGRID_PROB_24h Configuration Parameters Non-default parameters for the ``run_ensgridvx_prob_24h`` task are set in the ``task_run_vx_ensgrid_prob_24h:`` section of the ``config.yaml`` file. -``TN_VX_ENSGRID_PROB_24h``: (Default: "run_ensgridvx_prob_24h") - Set the name of this Rocoto workflow task. Users typically do not need to change this value. - ``NNODES_VX_ENSGRID_PROB``: (Default: 1) Number of nodes to use for the job. @@ -1729,9 +1575,6 @@ VX_ENSPOINT Configuration Parameters Non-default parameters for the ``run_enspointvx`` task are set in the ``task_run_vx_enspoint:`` section of the ``config.yaml`` file. -``TN_VX_ENSPOINT``: (Default: "run_enspointvx") - Set the name of this Rocoto workflow task. Users typically do not need to change this value. - ``NNODES_VX_ENSPOINT``: (Default: 1) Number of nodes to use for the job. @@ -1750,9 +1593,6 @@ VX_ENSPOINT_MEAN Configuration Parameters Non-default parameters for the ``run_enspointvx_mean`` task are set in the ``task_run_vx_enspoint_mean:`` section of the ``config.yaml`` file. -``TN_VX_ENSPOINT_MEAN``: (Default: "run_enspointvx_mean") - Set the name of this Rocoto workflow task. Users typically do not need to change this value. - ``NNODES_VX_ENSPOINT_MEAN``: (Default: 1) Number of nodes to use for the job. @@ -1771,9 +1611,6 @@ VX_ENSPOINT_PROB Configuration Parameters Non-default parameters for the ``run_enspointvx_prob`` task are set in the ``task_run_vx_enspoint_prob:`` section of the ``config.yaml`` file. -``TN_VX_ENSPOINT_PROB``: (Default: "run_enspointvx_prob") - Set the name of this Rocoto workflow task. Users typically do not need to change this value. - ``NNODES_VX_ENSPOINT_PROB``: (Default: 1) Number of nodes to use for the job. @@ -1798,9 +1635,6 @@ Basic Task Parameters For each workflow task, certain parameter values must be passed to the job scheduler (e.g., Slurm), which submits a job for the task. Typically, users do not need to adjust the default values. -``TN_PLOT_ALLVARS``: (Default: "plot_allvars") - Set the name of this Rocoto workflow task. Users typically do not need to change this value. - ``NNODES_PLOT_ALLVARS``: (Default: 1) Number of nodes to use for the job. diff --git a/docs/UsersGuide/source/RunSRW.rst b/docs/UsersGuide/source/RunSRW.rst index 2164a5c52e..500a1e259a 100644 --- a/docs/UsersGuide/source/RunSRW.rst +++ b/docs/UsersGuide/source/RunSRW.rst @@ -181,8 +181,7 @@ Configuration parameters in the ``config_defaults.yaml`` file appear in :numref: | | FCST_MODEL, WFLOW_XML_FN, GLOBAL_VAR_DEFNS_FN, | | | EXTRN_MDL_VAR_DEFNS_FN, WFLOW_LAUNCH_SCRIPT_FN, WFLOW_LAUNCH_LOG_FN, | | | CCPP_PHYS_SUITE, GRID_GEN_METHOD, DATE_FIRST_CYCL, DATE_LAST_CYCL, | - | | INCR_CYCL_FREQ, FCST_LEN_HRS, GET_OBS, TN_VX, TN_VX_ENSGRID, | - | | TN_VX_ENSGRID_PROB_REFC, MAXTRIES_VX_ENSGRID_PROB_REFC, | + | | INCR_CYCL_FREQ, FCST_LEN_HRS, GET_OBS, MAXTRIES_VX_ENSGRID_PROB_REFC, | | | PREEXISTING_DIR_METHOD, VERBOSE, DEBUG, COMPILER | +-----------------------------+-----------------------------------------------------------------------+ | NCO | envir, NET, model_ver, RUN, OPSROOT | @@ -194,7 +193,7 @@ Configuration parameters in the ``config_defaults.yaml`` file appear in :numref: | | RUN_TASK_VX_GRIDSTAT, RUN_TASK_VX_POINTSTAT, RUN_TASK_VX_ENSGRID, | | | RUN_TASK_VX_ENSPOINT | +-----------------------------+-----------------------------------------------------------------------+ - | task_make_grid | TN_MAKE_GRID, NNODES_MAKE_GRID, PPN_MAKE_GRID, WTIME_MAKE_GRID, | + | task_make_grid | NNODES_MAKE_GRID, PPN_MAKE_GRID, WTIME_MAKE_GRID, | | | MAXTRIES_MAKE_GRID, GRID_DIR, ESGgrid_LON_CTR, ESGgrid_LAT_CTR, | | | ESGgrid_DELX, ESGgrid_DELY, ESGgrid_NX, ESGgrid_NY, ESGgrid_PAZI, | | | ESGgrid_WIDE_HALO_WIDTH, GFDLgrid_LON_T6_CTR, GFDLgrid_LAT_T6_CTR, | @@ -203,16 +202,16 @@ Configuration parameters in the ``config_defaults.yaml`` file appear in :numref: | | GFDLgrid_JSTART_OF_RGNL_DOM_ON_T6G, GFDLgrid_JEND_OF_RGNL_DOM_ON_T6G, | | | GFDLgrid_USE_NUM_CELLS_IN_FILENAMES | +-----------------------------+-----------------------------------------------------------------------+ - | task_make_orog | TN_MAKE_OROG, NNODES_MAKE_OROG, PPN_MAKE_OROG, WTIME_MAKE_OROG, | + | task_make_orog | NNODES_MAKE_OROG, PPN_MAKE_OROG, WTIME_MAKE_OROG, | | | MAXTRIES_MAKE_OROG, KMP_AFFINITY_MAKE_OROG, OMP_NUM_THREADS_MAKE_OROG | | | OMP_STACKSIZE_MAKE_OROG, OROG_DIR | +-----------------------------+-----------------------------------------------------------------------+ - | task_make_sfc_climo | TN_MAKE_SFC_CLIMO, NNODES_MAKE_SFC_CLIMO, PPN_MAKE_SFC_CLIMO, | + | task_make_sfc_climo | NNODES_MAKE_SFC_CLIMO, PPN_MAKE_SFC_CLIMO, | | | WTIME_MAKE_SFC_CLIMO, MAXTRIES_MAKE_SFC_CLIMO, | | | KMP_AFFINITY_MAKE_SFC_CLIMO, OMP_NUM_THREADS_MAKE_SFC_CLIMO, | | | OMP_STACKSIZE_MAKE_SFC_CLIMO, SFC_CLIMO_DIR | +-----------------------------+-----------------------------------------------------------------------+ - | task_get_extrn_ics | TN_GET_EXTRN_ICS, NNODES_GET_EXTRN_ICS, PPN_GET_EXTRN_ICS, | + | task_get_extrn_ics | NNODES_GET_EXTRN_ICS, PPN_GET_EXTRN_ICS, | | | WTIME_GET_EXTRN_ICS, MAXTRIES_GET_EXTRN_ICS, EXTRN_MDL_NAME_ICS, | | | EXTRN_MDL_ICS_OFFSET_HRS, FV3GFS_FILE_FMT_ICS, | | | EXTRN_MDL_SYSBASEDIR_ICS, USE_USER_STAGED_EXTRN_FILES, | @@ -220,23 +219,23 @@ Configuration parameters in the ``config_defaults.yaml`` file appear in :numref: | | EXTRN_MDL_FILES_ICS, EXTRN_MDL_FILES_ICS, EXTRN_MDL_DATA_STORES, | | | NOMADS, NOMADS_file_type | +-----------------------------+-----------------------------------------------------------------------+ - | task_get_extrn_lbcs | TN_GET_EXTRN_LBCS, NNODES_GET_EXTRN_LBCS, PPN_GET_EXTRN_LBCS, | + | task_get_extrn_lbcs | NNODES_GET_EXTRN_LBCS, PPN_GET_EXTRN_LBCS, | | | WTIME_GET_EXTRN_LBCS, MAXTRIES_GET_EXTRN_LBCS, EXTRN_MDL_NAME_LBCS, | | | LBC_SPEC_INTVL_HRS, EXTRN_MDL_LBCS_OFFSET_HRS, FV3GFS_FILE_FMT_LBCS, | | | EXTRN_MDL_SYSBASEDIR_LBCS, USE_USER_STAGED_EXTRN_FILES, | | | EXTRN_MDL_SOURCE_BASEDIR_LBCS, EXTRN_MDL_FILES_LBCS, | | | EXTRN_MDL_DATA_STORE, NOMADS, NOMADS_file_type | +-----------------------------+-----------------------------------------------------------------------+ - | task_make_ics | TN_MAKE_ICS, NNODES_MAKE_ICS, PPN_MAKE_ICS, WTIME_MAKE_ICS, | + | task_make_ics | NNODES_MAKE_ICS, PPN_MAKE_ICS, WTIME_MAKE_ICS, | | | MAXTRIES_MAKE_ICS, KMP_AFFINITY_MAKE_ICS, OMP_NUM_THREADS_MAKE_ICS, | | | OMP_STACKSIZE_MAKE_ICS, USE_FVCOM, FVCOM_WCSTART, FVCOM_DIR, | | | FVCOM_FILE | +-----------------------------+-----------------------------------------------------------------------+ - | task_make_lbcs | TN_MAKE_LBCS, NNODES_MAKE_LBCS, PPN_MAKE_LBCS, WTIME_MAKE_LBCS, | + | task_make_lbcs | NNODES_MAKE_LBCS, PPN_MAKE_LBCS, WTIME_MAKE_LBCS, | | | MAXTRIES_MAKE_LBCS, KMP_AFFINITY_MAKE_LBCS, OMP_NUM_THREADS_MAKE_LBCS,| | | OMP_STACKSIZE_MAKE_LBCS | +-----------------------------+-----------------------------------------------------------------------+ - | task_run_fcst | TN_RUN_FCST, NNODES_RUN_FCST, PPN_RUN_FCST, WTIME_RUN_FCST, | + | task_run_fcst | NNODES_RUN_FCST, PPN_RUN_FCST, WTIME_RUN_FCST, | | | MAXTRIES_RUN_FCST, KMP_AFFINITY_RUN_FCST, OMP_NUM_THREADS_RUN_FCST, | | | OMP_STACKSIZE_RUN_FCST, DT_ATMOS, RESTART_INTERVAL, WRITE_DOPOST, | | | LAYOUT_X, LAYOUT_Y, BLOCKSIZE, QUILTING, PRINT_ESMF, | @@ -252,7 +251,7 @@ Configuration parameters in the ``config_defaults.yaml`` file appear in :numref: | | FV3_NML_VARNAME_TO_SFC_CLIMO_FIELD_MAPPING, | | | CYCLEDIR_LINKS_TO_FIXam_FILES_MAPPING | +-----------------------------+-----------------------------------------------------------------------+ - | task_run_post | TN_RUN_POST, NNODES_RUN_POST, PPN_RUN_POST, WTIME_RUN_POST, | + | task_run_post | NNODES_RUN_POST, PPN_RUN_POST, WTIME_RUN_POST, | | | MAXTRIES_RUN_POST, KMP_AFFINITY_RUN_POST, OMP_NUM_THREADS_RUN_POST, | | | OMP_STACKSIZE_RUN_POST, SUB_HOURLY_POST, DT_SUB_HOURLY_POST_MNTS, | | | USE_CUSTOM_POST_CONFIG_FILE, CUSTOM_POST_CONFIG_FP, | @@ -268,77 +267,77 @@ Configuration parameters in the ``config_defaults.yaml`` file appear in :numref: | | LSM_SPP_TSCALE, LSM_SPP_LSCALE, ISEED_LSM_SPP, LSM_SPP_VAR_LIST, | | | LSM_SPP_MAG_LIST, HALO_BLEND | +-----------------------------+-----------------------------------------------------------------------+ - | task_get_obs_ccpa | TN_GET_OBS_CCPA, NNODES_GET_OBS_CCPA, PPN_GET_OBS_CCPA, | + | task_get_obs_ccpa | NNODES_GET_OBS_CCPA, PPN_GET_OBS_CCPA, | | | WTIME_GET_OBS_CCPA, MAXTRIES_GET_OBS_CCPA | +-----------------------------+-----------------------------------------------------------------------+ - | task_get_obs_mrms | TN_GET_OBS_MRMS, NNODES_GET_OBS_MRMS, PPN_GET_OBS_MRMS, | + | task_get_obs_mrms | NNODES_GET_OBS_MRMS, PPN_GET_OBS_MRMS, | | | WTIME_GET_OBS_MRMS, MAXTRIES_GET_OBS_MRMS | +-----------------------------+-----------------------------------------------------------------------+ - | task_get_obs_ndas | TN_GET_OBS_NDAS, NNODES_GET_OBS_NDAS, PPN_GET_OBS_NDAS, | + | task_get_obs_ndas | NNODES_GET_OBS_NDAS, PPN_GET_OBS_NDAS, | | | WTIME_GET_OBS_NDAS, MAXTRIES_GET_OBS_NDAS | +-----------------------------+-----------------------------------------------------------------------+ - | task_run_vx_gridstat | TN_VX_GRIDSTAT, NNODES_VX_GRIDSTAT, PPN_VX_GRIDSTAT, | + | task_run_vx_gridstat | NNODES_VX_GRIDSTAT, PPN_VX_GRIDSTAT, | | | WTIME_VX_GRIDSTAT, MAXTRIES_VX_GRIDSTAT | +-----------------------------+-----------------------------------------------------------------------+ - | task_run_vx_gridstat_refc | TN_VX_GRIDSTAT_REFC, NNODES_VX_GRIDSTAT, PPN_VX_GRIDSTAT, | + | task_run_vx_gridstat_refc | NNODES_VX_GRIDSTAT, PPN_VX_GRIDSTAT, | | | WTIME_VX_GRIDSTAT, MAXTRIES_VX_GRIDSTAT_REFC | +-----------------------------+-----------------------------------------------------------------------+ - | task_run_vx_gridstat_retop | TN_VX_GRIDSTAT_RETOP, NNODES_VX_GRIDSTAT, PPN_VX_GRIDSTAT, | + | task_run_vx_gridstat_retop | NNODES_VX_GRIDSTAT, PPN_VX_GRIDSTAT, | | | WTIME_VX_GRIDSTAT, MAXTRIES_VX_GRIDSTAT_RETOP | +-----------------------------+-----------------------------------------------------------------------+ - | task_run_vx_gridstat_03h | TN_VX_GRIDSTAT_03h, NNODES_VX_GRIDSTAT, PPN_VX_GRIDSTAT, | + | task_run_vx_gridstat_03h | NNODES_VX_GRIDSTAT, PPN_VX_GRIDSTAT, | | | WTIME_VX_GRIDSTAT, MAXTRIES_VX_GRIDSTAT_03h | +-----------------------------+-----------------------------------------------------------------------+ - | task_run_vx_gridstat_06h | TN_VX_GRIDSTAT_06h, NNODES_VX_GRIDSTAT, PPN_VX_GRIDSTAT, | + | task_run_vx_gridstat_06h | NNODES_VX_GRIDSTAT, PPN_VX_GRIDSTAT, | | | WTIME_VX_GRIDSTAT, MAXTRIES_VX_GRIDSTAT_06h | +-----------------------------+-----------------------------------------------------------------------+ - | task_run_vx_gridstat_24h | TN_VX_GRIDSTAT_24h, NNODES_VX_GRIDSTAT, PPN_VX_GRIDSTAT, | + | task_run_vx_gridstat_24h | NNODES_VX_GRIDSTAT, PPN_VX_GRIDSTAT, | | | WTIME_VX_GRIDSTAT, MAXTRIES_VX_GRIDSTAT_24h | +-----------------------------+-----------------------------------------------------------------------+ - | task_run_vx_pointstat | TN_VX_POINTSTAT, NNODES_VX_POINTSTAT, PPN_VX_POINTSTAT, | + | task_run_vx_pointstat | NNODES_VX_POINTSTAT, PPN_VX_POINTSTAT, | | | WTIME_VX_POINTSTAT, MAXTRIES_VX_POINTSTAT | +-----------------------------+-----------------------------------------------------------------------+ - | task_run_vx_ensgrid | TN_VX_ENSGRID_03h, MAXTRIES_VX_ENSGRID_03h, TN_VX_ENSGRID_06h, | - | | MAXTRIES_VX_ENSGRID_06h, TN_VX_ENSGRID_24h, MAXTRIES_VX_ENSGRID_24h, | - | | TN_VX_ENSGRID_RETOP, MAXTRIES_VX_ENSGRID_RETOP, | - | | TN_VX_ENSGRID_PROB_RETOP, MAXTRIES_VX_ENSGRID_PROB_RETOP, | + | task_run_vx_ensgrid | MAXTRIES_VX_ENSGRID_03h, | + | | MAXTRIES_VX_ENSGRID_06h, MAXTRIES_VX_ENSGRID_24h, | + | | MAXTRIES_VX_ENSGRID_RETOP, | + | | MAXTRIES_VX_ENSGRID_PROB_RETOP, | | | NNODES_VX_ENSGRID, PPN_VX_ENSGRID, WTIME_VX_ENSGRID, | | | MAXTRIES_VX_ENSGRID | +-----------------------------+-----------------------------------------------------------------------+ - | task_run_vx_ensgrid_refc | TN_VX_ENSGRID_REFC, NNODES_VX_ENSGRID, PPN_VX_ENSGRID, | + | task_run_vx_ensgrid_refc | NNODES_VX_ENSGRID, PPN_VX_ENSGRID, | | | WTIME_VX_ENSGRID, MAXTRIES_VX_ENSGRID_REFC | +-----------------------------+-----------------------------------------------------------------------+ - | task_run_vx_ensgrid_mean | TN_VX_ENSGRID_MEAN, NNODES_VX_ENSGRID_MEAN, PPN_VX_ENSGRID_MEAN, | + | task_run_vx_ensgrid_mean | NNODES_VX_ENSGRID_MEAN, PPN_VX_ENSGRID_MEAN, | | | WTIME_VX_ENSGRID_MEAN, MAXTRIES_VX_ENSGRID_MEAN | +-----------------------------+-----------------------------------------------------------------------+ - | task_run_vx_ensgrid_mean_03h| TN_VX_ENSGRID_MEAN_03h, NNODES_VX_ENSGRID_MEAN, PPN_VX_ENSGRID_MEAN, | + | task_run_vx_ensgrid_mean_03h| NNODES_VX_ENSGRID_MEAN, PPN_VX_ENSGRID_MEAN, | | | WTIME_VX_ENSGRID_MEAN, MAXTRIES_VX_ENSGRID_MEAN_03h | +-----------------------------+-----------------------------------------------------------------------+ - | task_run_vx_ensgrid_mean_06h| TN_VX_ENSGRID_MEAN_06h, NNODES_VX_ENSGRID_MEAN, PPN_VX_ENSGRID_MEAN, | + | task_run_vx_ensgrid_mean_06h| NNODES_VX_ENSGRID_MEAN, PPN_VX_ENSGRID_MEAN, | | | WTIME_VX_ENSGRID_MEAN, MAXTRIES_VX_ENSGRID_MEAN_06h | +-----------------------------+-----------------------------------------------------------------------+ - | task_run_vx_ensgrid_mean_24h| TN_VX_ENSGRID_MEAN_24h, NNODES_VX_ENSGRID_MEAN, PPN_VX_ENSGRID_MEAN, | + | task_run_vx_ensgrid_mean_24h| NNODES_VX_ENSGRID_MEAN, PPN_VX_ENSGRID_MEAN, | | | WTIME_VX_ENSGRID_MEAN, MAXTRIES_VX_ENSGRID_MEAN_24h | +-----------------------------+-----------------------------------------------------------------------+ - | task_run_vx_ensgrid_prob | TN_VX_ENSGRID_PROB, NNODES_VX_ENSGRID_PROB, PPN_VX_ENSGRID_PROB, | + | task_run_vx_ensgrid_prob | NNODES_VX_ENSGRID_PROB, PPN_VX_ENSGRID_PROB, | | | WTIME_VX_ENSGRID_PROB, MAXTRIES_VX_ENSGRID_PROB | +-----------------------------+-----------------------------------------------------------------------+ - | task_run_vx_ensgrid_prob_03h| TN_VX_ENSGRID_PROB_03h, NNODES_VX_ENSGRID_PROB, PPN_VX_ENSGRID_PROB, | + | task_run_vx_ensgrid_prob_03h| NNODES_VX_ENSGRID_PROB, PPN_VX_ENSGRID_PROB, | | | WTIME_VX_ENSGRID_PROB, MAXTRIES_VX_ENSGRID_PROB_03h | +-----------------------------+-----------------------------------------------------------------------+ - | task_run_vx_ensgrid_prob_06h| TN_VX_ENSGRID_PROB_06h, NNODES_VX_ENSGRID_PROB, PPN_VX_ENSGRID_PROB, | + | task_run_vx_ensgrid_prob_06h| NNODES_VX_ENSGRID_PROB, PPN_VX_ENSGRID_PROB, | | | WTIME_VX_ENSGRID_PROB, MAXTRIES_VX_ENSGRID_PROB_06h | +-----------------------------+-----------------------------------------------------------------------+ - | task_run_vx_ensgrid_prob_24h| TN_VX_ENSGRID_PROB_24h, NNODES_VX_ENSGRID_PROB, PPN_VX_ENSGRID_PROB, | + | task_run_vx_ensgrid_prob_24h| NNODES_VX_ENSGRID_PROB, PPN_VX_ENSGRID_PROB, | | | WTIME_VX_ENSGRID_PROB, MAXTRIES_VX_ENSGRID_PROB_24h | +-----------------------------+-----------------------------------------------------------------------+ - | task_run_vx_enspoint | TN_VX_ENSPOINT, NNODES_VX_ENSPOINT, PPN_VX_ENSPOINT, | + | task_run_vx_enspoint | NNODES_VX_ENSPOINT, PPN_VX_ENSPOINT, | | | WTIME_VX_ENSPOINT, MAXTRIES_VX_ENSPOINT | +-----------------------------+-----------------------------------------------------------------------+ - | task_run_vx_enspoint_mean | TN_VX_ENSPOINT_MEAN, NNODES_VX_ENSPOINT_MEAN, PPN_VX_ENSPOINT_MEAN, | + | task_run_vx_enspoint_mean | NNODES_VX_ENSPOINT_MEAN, PPN_VX_ENSPOINT_MEAN, | | | WTIME_VX_ENSPOINT_MEAN, MAXTRIES_VX_ENSPOINT_MEAN | +-----------------------------+-----------------------------------------------------------------------+ - | task_run_vx_enspoint_prob | TN_VX_ENSPOINT_PROB, NNODES_VX_ENSPOINT_PROB, PPN_VX_ENSPOINT_PROB, | + | task_run_vx_enspoint_prob | NNODES_VX_ENSPOINT_PROB, PPN_VX_ENSPOINT_PROB, | | | WTIME_VX_ENSPOINT_PROB, MAXTRIES_VX_ENSPOINT_PROB | +-----------------------------+-----------------------------------------------------------------------+ From ac9601527fd71ba9dc49af04abd2b05a1ac06ed2 Mon Sep 17 00:00:00 2001 From: Christina Holt Date: Tue, 18 Apr 2023 17:56:09 -0600 Subject: [PATCH 02/10] Remove references to NNODES, PPN, MAXTRIES, WTIME vars --- docs/UsersGuide/source/ConfigWorkflow.rst | 581 ---------------------- docs/UsersGuide/source/RunSRW.rst | 111 +---- 2 files changed, 10 insertions(+), 682 deletions(-) diff --git a/docs/UsersGuide/source/ConfigWorkflow.rst b/docs/UsersGuide/source/ConfigWorkflow.rst index 5ee5884096..1222e077a6 100644 --- a/docs/UsersGuide/source/ConfigWorkflow.rst +++ b/docs/UsersGuide/source/ConfigWorkflow.rst @@ -459,18 +459,6 @@ Basic Task Parameters For each workflow task, certain parameter values must be passed to the job scheduler (e.g., Slurm), which submits a job for the task. Typically, users do not need to adjust the default values. - ``NNODES_MAKE_GRID``: (Default: 1) - Number of nodes to use for the job. - - ``PPN_MAKE_GRID``: (Default: 24) - Number of :term:`MPI` processes per node. - - ``WTIME_MAKE_GRID``: (Default: 00:20:00) - Maximum time for the task to complete. - - ``MAXTRIES_MAKE_GRID``: (Default: 2) - Maximum number of times to attempt the task. - ``GRID_DIR``: (Default: "") The directory containing pre-generated grid files when ``RUN_TASK_MAKE_GRID`` is set to false. @@ -576,18 +564,6 @@ MAKE_OROG Configuration Parameters Non-default parameters for the ``make_orog`` task are set in the ``task_make_orog:`` section of the ``config.yaml`` file. -``NNODES_MAKE_OROG``: (Default: 1) - Number of nodes to use for the job. - -``PPN_MAKE_OROG``: (Default: 24) - Number of :term:`MPI` processes per node. - -``WTIME_MAKE_OROG``: (Default: 00:20:00) - Maximum time for the task to complete. - -``MAXTRIES_MAKE_OROG``: (Default: 2) - Maximum number of times to attempt the task. - ``KMP_AFFINITY_MAKE_OROG``: (Default: "disabled") Intel Thread Affinity Interface for the ``make_orog`` task. See :ref:`this note ` for more information on thread affinity. Settings for the ``make_orog`` task is disabled because this task does not use parallelized code. @@ -609,18 +585,6 @@ MAKE_SFC_CLIMO Configuration Parameters Non-default parameters for the ``make_sfc_climo`` task are set in the ``task_make_sfc_climo:`` section of the ``config.yaml`` file. -``NNODES_MAKE_SFC_CLIMO``: (Default: 2) - Number of nodes to use for the job. - -``PPN_MAKE_SFC_CLIMO``: (Default: 24) - Number of :term:`MPI` processes per node. - -``WTIME_MAKE_SFC_CLIMO``: (Default: 00:20:00) - Maximum time for the task to complete. - -``MAXTRIES_MAKE_SFC_CLIMO``: (Default: 2) - Maximum number of times to attempt the task. - ``KMP_AFFINITY_MAKE_SFC_CLIMO``: (Default: "scatter") Intel Thread Affinity Interface for the ``make_sfc_climo`` task. See :ref:`this note ` for more information on thread affinity. @@ -647,18 +611,6 @@ Basic Task Parameters For each workflow task, certain parameter values must be passed to the job scheduler (e.g., Slurm), which submits a job for the task. -``NNODES_GET_EXTRN_ICS``: (Default: 1) - Number of nodes to use for the job. - -``PPN_GET_EXTRN_ICS``: (Default: 1) - Number of :term:`MPI` processes per node. - -``WTIME_GET_EXTRN_ICS``: (Default: 00:45:00) - Maximum time for the task to complete. - -``MAXTRIES_GET_EXTRN_ICS``: (Default: 1) - Maximum number of times to attempt the task. - ``EXTRN_MDL_NAME_ICS``: (Default: "FV3GFS") The name of the external model that will provide fields from which initial condition (IC) files, surface files, and 0-th hour boundary condition files will be generated for input into the forecast model. Valid values: ``"GSMGFS"`` | ``"FV3GFS"`` | ``"GEFS"`` | ``"GDAS"`` | ``"RAP"`` | ``"HRRR"`` | ``"NAM"`` @@ -727,18 +679,6 @@ Basic Task Parameters For each workflow task, certain parameter values must be passed to the job scheduler (e.g., Slurm), which submits a job for the task. -``NNODES_GET_EXTRN_LBCS``: (Default: 1) - Number of nodes to use for the job. - -``PPN_GET_EXTRN_LBCS``: (Default: 1) - Number of :term:`MPI` processes per node. - -``WTIME_GET_EXTRN_LBCS``: (Default: 00:45:00) - Maximum time for the task to complete. - -``MAXTRIES_GET_EXTRN_LBCS``: (Default: 1) - Maximum number of times to attempt the task. - ``EXTRN_MDL_NAME_LBCS``: (Default: "FV3GFS") The name of the external model that will provide fields from which lateral boundary condition (LBC) files (except for the 0-th hour LBC file) will be generated for input into the forecast model. Valid values: ``"GSMGFS"`` | ``"FV3GFS"`` | ``"GEFS"`` | ``"GDAS"`` | ``"RAP"`` | ``"HRRR"`` | ``"NAM"`` @@ -795,18 +735,6 @@ Basic Task Parameters For each workflow task, certain parameter values must be passed to the job scheduler (e.g., Slurm), which submits a job for the task. -``NNODES_MAKE_ICS``: (Default: 4) - Number of nodes to use for the job. - -``PPN_MAKE_ICS``: (Default: 12) - Number of :term:`MPI` processes per node. - -``WTIME_MAKE_ICS``: (Default: 00:30:00) - Maximum time for the task to complete. - -``MAXTRIES_MAKE_ICS``: (Default: 1) - Maximum number of times to attempt the task. - ``KMP_AFFINITY_MAKE_ICS``: (Default: "scatter") Intel Thread Affinity Interface for the ``make_ics`` task. See :ref:`this note ` for more information on thread affinity. @@ -836,18 +764,6 @@ MAKE_LBCS Configuration Parameters Non-default parameters for the ``make_lbcs`` task are set in the ``task_make_lbcs:`` section of the ``config.yaml`` file. -``NNODES_MAKE_LBCS``: (Default: 4) - Number of nodes to use for the job. - -``PPN_MAKE_LBCS``: (Default: 12) - Number of :term:`MPI` processes per node. - -``WTIME_MAKE_LBCS``: (Default: 00:30:00) - Maximum time for the task to complete. - -``MAXTRIES_MAKE_LBCS``: (Default: 1) - Maximum number of times to attempt the task. - ``KMP_AFFINITY_MAKE_LBCS``: (Default: "scatter") Intel Thread Affinity Interface for the ``make_lbcs`` task. See :ref:`this note ` for more information on thread affinity. @@ -869,18 +785,6 @@ Basic Task Parameters For each workflow task, certain parameter values must be passed to the job scheduler (e.g., Slurm), which submits a job for the task. -``NNODES_RUN_FCST``: (Default: "") - Number of nodes to use for the job. This is calculated in the workflow generation scripts, so there is no need to set it in the configuration file. - -``PPN_RUN_FCST``: (Default: "") - Number of :term:`MPI` processes per node. It will be calculated from ``NCORES_PER_NODE`` and ``OMP_NUM_THREADS`` in ``setup.py``. - -``WTIME_RUN_FCST``: (Default: 04:30:00) - Maximum time for the task to complete. - -``MAXTRIES_RUN_FCST``: (Default: 1) - Maximum number of times to attempt the task. - ``KMP_AFFINITY_RUN_FCST``: (Default: "scatter") Intel Thread Affinity Interface for the ``run_fcst`` task. @@ -1094,18 +998,6 @@ Basic Task Parameters For each workflow task, certain parameter values must be passed to the job scheduler (e.g., Slurm), which submits a job for the task. -``NNODES_RUN_POST``: (Default: 2) - Number of nodes to use for the job. - -``PPN_RUN_POST``: (Default: 24) - Number of :term:`MPI` processes per node. - -``WTIME_RUN_POST``: (Default: 00:15:00) - Maximum time for the task to complete. - -``MAXTRIES_RUN_POST``: (Default: 2) - Maximum number of times to attempt the task. - ``KMP_AFFINITY_RUN_POST``: (Default: "scatter") Intel Thread Affinity Interface for the ``run_post`` task. See :ref:`this note ` for more information on thread affinity. @@ -1155,18 +1047,6 @@ Basic Task Parameters --------------------------------- For each workflow task, certain parameter values must be passed to the job scheduler (e.g., Slurm), which submits a job for the task. -``NNODES_RUN_PRDGEN``: (Default: 1) - Number of nodes to use for the job. - -``PPN_RUN_PRDGEN``: (Default: 22) - Number of :term:`MPI` processes per node. - -``WTIME_RUN_PRDGEN``: (Default: 00:30:00) - Maximum time for the task to complete. - -``MAXTRIES_RUN_PRDGEN``: (Default: 2) - Maximum number of times to attempt the task. - ``KMP_AFFINITY_RUN_PRDGEN``: (Default: "scatter") Intel Thread Affinity Interface for the ``run_prdgen`` task. See :ref:`this note ` for more information on thread affinity. @@ -1186,467 +1066,6 @@ For each workflow task, certain parameter values must be passed to the job sched The file which lists grib2 fields to be extracted for testbed files. Empty string means no need to generate testbed files. -.. _get-obs-ccpa: - -GET_OBS_CCPA Configuration Parameters -======================================== - -Non-default parameters for the ``get_obs_ccpa`` task are set in the ``task_get_obs_ccpa:`` section of the ``config.yaml`` file. - -``NNODES_GET_OBS_CCPA``: (Default: 1) - Number of nodes to use for the job. - -``PPN_GET_OBS_CCPA``: (Default: 1) - Number of :term:`MPI` processes per node. - -``WTIME_GET_OBS_CCPA``: (Default: 00:45:00) - Maximum time for the task to complete. - -``MAXTRIES_GET_OBS_CCPA``: (Default: 1) - Maximum number of times to attempt the task. - -.. _get-obs-mrms: - -GET_OBS_MRMS Configuration Parameters -======================================== - -Non-default parameters for the ``get_obs_mrms`` task are set in the ``task_get_obs_mrms:`` section of the ``config.yaml`` file. See :numref:`Section %s ` for more information about the verification tasks. - -``NNODES_GET_OBS_MRMS``: (Default: 1) - Number of nodes to use for the job. - -``PPN_GET_OBS_MRMS``: (Default: 1) - Number of :term:`MPI` processes per node. - -``WTIME_GET_OBS_MRMS``: (Default: 00:45:00) - Maximum time for the task to complete. - -``MAXTRIES_GET_OBS_MRMS``: (Default: 1) - Maximum number of times to attempt the task. - -.. _get-obs-ndas: - -GET_OBS_NDAS Configuration Parameters -======================================== - -Non-default parameters for the ``get_obs_ndas`` task are set in the ``task_get_obs_ndas:`` section of the ``config.yaml`` file. See :numref:`Section %s ` for more information about the verification tasks. - -``NNODES_GET_OBS_NDAS``: (Default: 1) - Number of nodes to use for the job. - -``PPN_GET_OBS_NDAS``: (Default: 1) - Number of :term:`MPI` processes per node. - -``WTIME_GET_OBS_NDAS``: (Default: 02:00:00) - Maximum time for the task to complete. - -``MAXTRIES_GET_OBS_NDAS``: (Default: 1) - Maximum number of times to attempt the task. - - -.. _VX-gridstat: - -VX_GRIDSTAT Configuration Parameters -======================================== - -Non-default parameters for the ``run_gridstatvx`` task are set in the ``task_run_vx_gridstat:`` section of the ``config.yaml`` file. - -``NNODES_VX_GRIDSTAT``: (Default: 1) - Number of nodes to use for the job. - -``PPN_VX_GRIDSTAT``: (Default: 1) - Number of :term:`MPI` processes per node. - -``WTIME_VX_GRIDSTAT``: (Default: 02:00:00) - Maximum time for the task to complete. - -``MAXTRIES_VX_GRIDSTAT``: (Default: 1) - Maximum number of times to attempt the task. - - -VX_GRIDSTAT_REFC Configuration Parameters -============================================= - -Non-default parameters for the ``run_gridstatvx_refc`` task are set in the ``task_run_vx_gridstat_refc:`` section of the ``config.yaml`` file. - -``NNODES_VX_GRIDSTAT``: (Default: 1) - Number of nodes to use for the job. - -``PPN_VX_GRIDSTAT``: (Default: 1) - Number of :term:`MPI` processes per node. - -``WTIME_VX_GRIDSTAT``: (Default: 02:00:00) - Maximum time for the task to complete. - -``MAXTRIES_VX_GRIDSTAT_REFC``: (Default: 1) - Maximum number of times to attempt the task. - - -VX_GRIDSTAT_RETOP Configuration Parameters -============================================= - -Non-default parameters for the ``run_gridstatvx_retop`` task are set in the ``task_run_vx_gridstat_retop:`` section of the ``config.yaml`` file. - -``NNODES_VX_GRIDSTAT``: (Default: 1) - Number of nodes to use for the job. - -``PPN_VX_GRIDSTAT``: (Default: 1) - Number of :term:`MPI` processes per node. - -``WTIME_VX_GRIDSTAT``: (Default: 02:00:00) - Maximum time for the task to complete. - -``MAXTRIES_VX_GRIDSTAT_RETOP``: (Default: 1) - Maximum number of times to attempt the task. - - -VX_GRIDSTAT_03h Configuration Parameters -============================================= - -Non-default parameters for the ``run_gridstatvx_03h`` task are set in the ``task_run_vx_gridstat_03h:`` section of the ``config.yaml`` file. - -``NNODES_VX_GRIDSTAT``: (Default: 1) - Number of nodes to use for the job. - -``PPN_VX_GRIDSTAT``: (Default: 1) - Number of :term:`MPI` processes per node. - -``WTIME_VX_GRIDSTAT``: (Default: 02:00:00) - Maximum time for the task to complete. - -``MAXTRIES_VX_GRIDSTAT_03h``: (Default: 1) - Maximum number of times to attempt the task. - - -VX_GRIDSTAT_06h Configuration Parameters -============================================= - -Non-default parameters for the ``run_gridstatvx_06h`` task are set in the ``task_run_vx_gridstat_06h:`` section of the ``config.yaml`` file. - -``NNODES_VX_GRIDSTAT``: (Default: 1) - Number of nodes to use for the job. - -``PPN_VX_GRIDSTAT``: (Default: 1) - Number of :term:`MPI` processes per node. - -``WTIME_VX_GRIDSTAT``: (Default: 02:00:00) - Maximum time for the task to complete. - -``MAXTRIES_VX_GRIDSTAT_06h``: (Default: 1) - Maximum number of times to attempt the task. - - -VX_GRIDSTAT_24h Configuration Parameters -============================================= - -Non-default parameters for the ``run_gridstatvx_24h`` task are set in the ``task_run_vx_gridstat_24h:`` section of the ``config.yaml`` file. - -``NNODES_VX_GRIDSTAT``: (Default: 1) - Number of nodes to use for the job. - -``PPN_VX_GRIDSTAT``: (Default: 1) - Number of :term:`MPI` processes per node. - -``WTIME_VX_GRIDSTAT``: (Default: 02:00:00) - Maximum time for the task to complete. - -``MAXTRIES_VX_GRIDSTAT_24h``: (Default: 1) - Maximum number of times to attempt the task. - -.. _VX-pointstat: - -VX_POINTSTAT Configuration Parameters -============================================= - -Non-default parameters for the ``run_pointstatvx`` task are set in the ``task_run_vx_pointstat:`` section of the ``config.yaml`` file. - -``NNODES_VX_POINTSTAT``: (Default: 1) - Number of nodes to use for the job. - -``PPN_VX_POINTSTAT``: (Default: 1) - Number of :term:`MPI` processes per node. - -``WTIME_VX_POINTSTAT``: (Default: 01:00:00) - Maximum time for the task to complete. - -``MAXTRIES_VX_POINTSTAT``: (Default: 1) - Maximum number of times to attempt the task. - -.. _VX-ensgrid: - -VX_ENSGRID Configuration Parameters -============================================= - -Non-default parameters for the ``run_ensgridvx_*`` tasks are set in the ``task_run_vx_ensgrid:`` section of the ``config.yaml`` file. - -``MAXTRIES_VX_ENSGRID_03h``: (Default: 1) - Maximum number of times to attempt the task. - -``MAXTRIES_VX_ENSGRID_06h``: (Default: 1) - Maximum number of times to attempt the task. - -``MAXTRIES_VX_ENSGRID_24h``: (Default: 1) - Maximum number of times to attempt the task. - -``MAXTRIES_VX_ENSGRID_RETOP``: (Default: 1) - Maximum number of times to attempt the task. - -``MAXTRIES_VX_ENSGRID_PROB_RETOP``: (Default: 1) - Maximum number of times to attempt the task. - -``NNODES_VX_ENSGRID``: (Default: 1) - Number of nodes to use for the job. - -``PPN_VX_ENSGRID``: (Default: 1) - Number of :term:`MPI` processes per node. - -``WTIME_VX_ENSGRID``: (Default: 01:00:00) - Maximum time for the task to complete. - -``MAXTRIES_VX_ENSGRID``: (Default: 1) - Maximum number of times to attempt the task. - - -VX_ENSGRID_REFC Configuration Parameters -============================================= - -Non-default parameters for the ``run_ensgridvx_refc`` task are set in the ``task_run_vx_ensgrid_refc:`` section of the ``config.yaml`` file. - -``NNODES_VX_ENSGRID``: (Default: 1) - Number of nodes to use for the job. - -``PPN_VX_ENSGRID``: (Default: 1) - Number of :term:`MPI` processes per node. - -``WTIME_VX_ENSGRID``: (Default: 01:00:00) - Maximum time for the task to complete. - -``MAXTRIES_VX_ENSGRID_REFC``: (Default: 1) - Maximum number of times to attempt the task. - - -VX_ENSGRID_MEAN Configuration Parameters -============================================= - -Non-default parameters for the ``run_ensgridvx_mean`` task are set in the ``task_run_vx_ensgrid_mean:`` section of the ``config.yaml`` file. - -``NNODES_VX_ENSGRID_MEAN``: (Default: 1) - Number of nodes to use for the job. - -``PPN_VX_ENSGRID_MEAN``: (Default: 1) - Number of :term:`MPI` processes per node. - -``WTIME_VX_ENSGRID_MEAN``: (Default: 01:00:00) - Maximum time for the task to complete. - -``MAXTRIES_VX_ENSGRID_MEAN``: (Default: 1) - Maximum number of times to attempt the task. - - -VX_ENSGRID_MEAN_03h Configuration Parameters -=============================================== - -Non-default parameters for the ``run_ensgridvx_mean_03h`` task are set in the ``task_run_vx_ensgrid_mean_03h:`` section of the ``config.yaml`` file. - -``NNODES_VX_ENSGRID_MEAN``: (Default: 1) - Number of nodes to use for the job. - -``PPN_VX_ENSGRID_MEAN``: (Default: 1) - Number of :term:`MPI` processes per node. - -``WTIME_VX_ENSGRID_MEAN``: (Default: 01:00:00) - Maximum time for the task to complete. - -``MAXTRIES_VX_ENSGRID_MEAN_03h``: (Default: 1) - Maximum number of times to attempt the task. - - -VX_ENSGRID_MEAN_06h Configuration Parameters -=============================================== - -Non-default parameters for the ``run_ensgridvx_mean_06h`` task are set in the ``task_run_vx_ensgrid_mean_06h:`` section of the ``config.yaml`` file. - -``NNODES_VX_ENSGRID_MEAN``: (Default: 1) - Number of nodes to use for the job. - -``PPN_VX_ENSGRID_MEAN``: (Default: 1) - Number of :term:`MPI` processes per node. - -``WTIME_VX_ENSGRID_MEAN``: (Default: 01:00:00) - Maximum time for the task to complete. - -``MAXTRIES_VX_ENSGRID_MEAN_06h``: (Default: 1) - Maximum number of times to attempt the task. - - -VX_ENSGRID_MEAN_24h Configuration Parameters -=============================================== - -Non-default parameters for the ``run_ensgridvx_mean_24h`` task are set in the ``task_run_vx_ensgrid_mean_24h:`` section of the ``config.yaml`` file. - -``NNODES_VX_ENSGRID_MEAN``: (Default: 1) - Number of nodes to use for the job. - -``PPN_VX_ENSGRID_MEAN``: (Default: 1) - Number of :term:`MPI` processes per node. - -``WTIME_VX_ENSGRID_MEAN``: (Default: 01:00:00) - Maximum time for the task to complete. - -``MAXTRIES_VX_ENSGRID_MEAN_24h``: (Default: 1) - Maximum number of times to attempt the task. - - -VX_ENSGRID_PROB Configuration Parameters -============================================ - -Non-default parameters for the ``run_ensgridvx_prob`` task are set in the ``task_run_vx_ensgrid_prob:`` section of the ``config.yaml`` file. - -``NNODES_VX_ENSGRID_PROB``: (Default: 1) - Number of nodes to use for the job. - -``PPN_VX_ENSGRID_PROB``: (Default: 1) - Number of :term:`MPI` processes per node. - -``WTIME_VX_ENSGRID_PROB``: (Default: 01:00:00) - Maximum time for the task to complete. - -``MAXTRIES_VX_ENSGRID_PROB``: (Default: 1) - Maximum number of times to attempt the task. - - -VX_ENSGRID_PROB_03h Configuration Parameters -================================================ - -Non-default parameters for the ``run_ensgridvx_prob_03h`` task are set in the ``task_run_vx_ensgrid_prob_03h:`` section of the ``config.yaml`` file. - -``NNODES_VX_ENSGRID_PROB``: (Default: 1) - Number of nodes to use for the job. - -``PPN_VX_ENSGRID_PROB``: (Default: 1) - Number of :term:`MPI` processes per node. - -``WTIME_VX_ENSGRID_PROB``: (Default: 01:00:00) - Maximum time for the task to complete. - -``MAXTRIES_VX_ENSGRID_PROB_03h``: (Default: 1) - Maximum number of times to attempt the task. - - -VX_ENSGRID_PROB_06h Configuration Parameters -================================================ - -Non-default parameters for the ``run_ensgridvx_prob_06h`` task are set in the ``task_run_vx_ensgrid_prob_06h:`` section of the ``config.yaml`` file. - -``NNODES_VX_ENSGRID_PROB``: (Default: 1) - Number of nodes to use for the job. - -``PPN_VX_ENSGRID_PROB``: (Default: 1) - Number of :term:`MPI` processes per node. - -``WTIME_VX_ENSGRID_PROB``: (Default: 01:00:00) - Maximum time for the task to complete. - -``MAXTRIES_VX_ENSGRID_PROB_06h``: (Default: 1) - Maximum number of times to attempt the task. - - -VX_ENSGRID_PROB_24h Configuration Parameters -================================================ - -Non-default parameters for the ``run_ensgridvx_prob_24h`` task are set in the ``task_run_vx_ensgrid_prob_24h:`` section of the ``config.yaml`` file. - -``NNODES_VX_ENSGRID_PROB``: (Default: 1) - Number of nodes to use for the job. - -``PPN_VX_ENSGRID_PROB``: (Default: 1) - Number of :term:`MPI` processes per node. - -``WTIME_VX_ENSGRID_PROB``: (Default: 01:00:00) - Maximum time for the task to complete. - -``MAXTRIES_VX_ENSGRID_PROB_24h``: (Default: 1) - Maximum number of times to attempt the task. - -.. _VX-enspoint: - -VX_ENSPOINT Configuration Parameters -======================================== - -Non-default parameters for the ``run_enspointvx`` task are set in the ``task_run_vx_enspoint:`` section of the ``config.yaml`` file. - -``NNODES_VX_ENSPOINT``: (Default: 1) - Number of nodes to use for the job. - -``PPN_VX_ENSPOINT``: (Default: 1) - Number of :term:`MPI` processes per node. - -``WTIME_VX_ENSPOINT``: (Default: 01:00:00) - Maximum time for the task to complete. - -``MAXTRIES_VX_ENSPOINT``: (Default: 1) - Maximum number of times to attempt the task. - - -VX_ENSPOINT_MEAN Configuration Parameters -============================================== - -Non-default parameters for the ``run_enspointvx_mean`` task are set in the ``task_run_vx_enspoint_mean:`` section of the ``config.yaml`` file. - -``NNODES_VX_ENSPOINT_MEAN``: (Default: 1) - Number of nodes to use for the job. - -``PPN_VX_ENSPOINT_MEAN``: (Default: 1) - Number of :term:`MPI` processes per node. - -``WTIME_VX_ENSPOINT_MEAN``: (Default: 01:00:00) - Maximum time for the task to complete. - -``MAXTRIES_VX_ENSPOINT_MEAN``: (Default: 1) - Maximum number of times to attempt the task. - - -VX_ENSPOINT_PROB Configuration Parameters -============================================== - -Non-default parameters for the ``run_enspointvx_prob`` task are set in the ``task_run_vx_enspoint_prob:`` section of the ``config.yaml`` file. - -``NNODES_VX_ENSPOINT_PROB``: (Default: 1) - Number of nodes to use for the job. - -``PPN_VX_ENSPOINT_PROB``: (Default: 1) - Number of :term:`MPI` processes per node. - -``WTIME_VX_ENSPOINT_PROB``: (Default: 01:00:00) - Maximum time for the task to complete. - -``MAXTRIES_VX_ENSPOINT_PROB``: (Default: 1) - Maximum number of times to attempt the task. - -.. _PlotVars: - -PLOT_ALLVARS Configuration Parameters -======================================== - -Non-default parameters for the ``plot_allvars`` task are set in the ``task_plot_allvars:`` section of the ``config.yaml`` file. - -Basic Task Parameters --------------------------- - -For each workflow task, certain parameter values must be passed to the job scheduler (e.g., Slurm), which submits a job for the task. Typically, users do not need to adjust the default values. - -``NNODES_PLOT_ALLVARS``: (Default: 1) - Number of nodes to use for the job. - -``PPN_PLOT_ALLVARS``: (Default: 24) - Number of :term:`MPI` processes per node. - -``WTIME_PLOT_ALLVARS``: (Default: 01:00:00) - Maximum time for the task to complete. - -``MAXTRIES_PLOT_ALLVARS``: (Default: 1) - Maximum number of times to attempt the task. - Additional Parameters ------------------------ diff --git a/docs/UsersGuide/source/RunSRW.rst b/docs/UsersGuide/source/RunSRW.rst index 500a1e259a..0a323964f6 100644 --- a/docs/UsersGuide/source/RunSRW.rst +++ b/docs/UsersGuide/source/RunSRW.rst @@ -186,15 +186,7 @@ Configuration parameters in the ``config_defaults.yaml`` file appear in :numref: +-----------------------------+-----------------------------------------------------------------------+ | NCO | envir, NET, model_ver, RUN, OPSROOT | +-----------------------------+-----------------------------------------------------------------------+ - | Workflow Switches | RUN_TASK_MAKE_GRID, RUN_TASK_MAKE_OROG, RUN_TASK_MAKE_SFC_CLIMO, | - | | RUN_TASK_GET_EXTRN_ICS, RUN_TASK_GET_EXTRN_LBCS, RUN_TASK_MAKE_ICS, | - | | RUN_TASK_MAKE_LBCS, RUN_TASK_RUN_FCST, RUN_TASK_RUN_POST, | - | | RUN_TASK_GET_OBS_CCPA, RUN_TASK_GET_OBS_MRMS, RUN_TASK_GET_OBS_NDAS, | - | | RUN_TASK_VX_GRIDSTAT, RUN_TASK_VX_POINTSTAT, RUN_TASK_VX_ENSGRID, | - | | RUN_TASK_VX_ENSPOINT | - +-----------------------------+-----------------------------------------------------------------------+ - | task_make_grid | NNODES_MAKE_GRID, PPN_MAKE_GRID, WTIME_MAKE_GRID, | - | | MAXTRIES_MAKE_GRID, GRID_DIR, ESGgrid_LON_CTR, ESGgrid_LAT_CTR, | + | task_make_grid | GRID_DIR, ESGgrid_LON_CTR, ESGgrid_LAT_CTR, | | | ESGgrid_DELX, ESGgrid_DELY, ESGgrid_NX, ESGgrid_NY, ESGgrid_PAZI, | | | ESGgrid_WIDE_HALO_WIDTH, GFDLgrid_LON_T6_CTR, GFDLgrid_LAT_T6_CTR, | | | GFDLgrid_NUM_CELLS, GFDLgrid_STRETCH_FAC, GFDLgrid_REFINE_RATIO, | @@ -202,41 +194,32 @@ Configuration parameters in the ``config_defaults.yaml`` file appear in :numref: | | GFDLgrid_JSTART_OF_RGNL_DOM_ON_T6G, GFDLgrid_JEND_OF_RGNL_DOM_ON_T6G, | | | GFDLgrid_USE_NUM_CELLS_IN_FILENAMES | +-----------------------------+-----------------------------------------------------------------------+ - | task_make_orog | NNODES_MAKE_OROG, PPN_MAKE_OROG, WTIME_MAKE_OROG, | - | | MAXTRIES_MAKE_OROG, KMP_AFFINITY_MAKE_OROG, OMP_NUM_THREADS_MAKE_OROG | + | task_make_orog | KMP_AFFINITY_MAKE_OROG, OMP_NUM_THREADS_MAKE_OROG | | | OMP_STACKSIZE_MAKE_OROG, OROG_DIR | +-----------------------------+-----------------------------------------------------------------------+ - | task_make_sfc_climo | NNODES_MAKE_SFC_CLIMO, PPN_MAKE_SFC_CLIMO, | - | | WTIME_MAKE_SFC_CLIMO, MAXTRIES_MAKE_SFC_CLIMO, | - | | KMP_AFFINITY_MAKE_SFC_CLIMO, OMP_NUM_THREADS_MAKE_SFC_CLIMO, | + | task_make_sfc_climo | KMP_AFFINITY_MAKE_SFC_CLIMO, OMP_NUM_THREADS_MAKE_SFC_CLIMO, | | | OMP_STACKSIZE_MAKE_SFC_CLIMO, SFC_CLIMO_DIR | +-----------------------------+-----------------------------------------------------------------------+ - | task_get_extrn_ics | NNODES_GET_EXTRN_ICS, PPN_GET_EXTRN_ICS, | - | | WTIME_GET_EXTRN_ICS, MAXTRIES_GET_EXTRN_ICS, EXTRN_MDL_NAME_ICS, | - | | EXTRN_MDL_ICS_OFFSET_HRS, FV3GFS_FILE_FMT_ICS, | + | task_get_extrn_ics | EXTRN_MDL_NAME_ICS, EXTRN_MDL_ICS_OFFSET_HRS, FV3GFS_FILE_FMT_ICS, | | | EXTRN_MDL_SYSBASEDIR_ICS, USE_USER_STAGED_EXTRN_FILES, | | | EXTRN_MDL_SOURCE_BASEDIR_ICS, EXTRN_MDL_FILES_ICS, | | | EXTRN_MDL_FILES_ICS, EXTRN_MDL_FILES_ICS, EXTRN_MDL_DATA_STORES, | | | NOMADS, NOMADS_file_type | +-----------------------------+-----------------------------------------------------------------------+ - | task_get_extrn_lbcs | NNODES_GET_EXTRN_LBCS, PPN_GET_EXTRN_LBCS, | - | | WTIME_GET_EXTRN_LBCS, MAXTRIES_GET_EXTRN_LBCS, EXTRN_MDL_NAME_LBCS, | + | task_get_extrn_lbcs | EXTRN_MDL_NAME_LBCS, | | | LBC_SPEC_INTVL_HRS, EXTRN_MDL_LBCS_OFFSET_HRS, FV3GFS_FILE_FMT_LBCS, | | | EXTRN_MDL_SYSBASEDIR_LBCS, USE_USER_STAGED_EXTRN_FILES, | | | EXTRN_MDL_SOURCE_BASEDIR_LBCS, EXTRN_MDL_FILES_LBCS, | | | EXTRN_MDL_DATA_STORE, NOMADS, NOMADS_file_type | +-----------------------------+-----------------------------------------------------------------------+ - | task_make_ics | NNODES_MAKE_ICS, PPN_MAKE_ICS, WTIME_MAKE_ICS, | - | | MAXTRIES_MAKE_ICS, KMP_AFFINITY_MAKE_ICS, OMP_NUM_THREADS_MAKE_ICS, | + | task_make_ics | KMP_AFFINITY_MAKE_ICS, OMP_NUM_THREADS_MAKE_ICS, | | | OMP_STACKSIZE_MAKE_ICS, USE_FVCOM, FVCOM_WCSTART, FVCOM_DIR, | | | FVCOM_FILE | +-----------------------------+-----------------------------------------------------------------------+ - | task_make_lbcs | NNODES_MAKE_LBCS, PPN_MAKE_LBCS, WTIME_MAKE_LBCS, | - | | MAXTRIES_MAKE_LBCS, KMP_AFFINITY_MAKE_LBCS, OMP_NUM_THREADS_MAKE_LBCS,| + | task_make_lbcs | KMP_AFFINITY_MAKE_LBCS, OMP_NUM_THREADS_MAKE_LBCS, | | | OMP_STACKSIZE_MAKE_LBCS | +-----------------------------+-----------------------------------------------------------------------+ - | task_run_fcst | NNODES_RUN_FCST, PPN_RUN_FCST, WTIME_RUN_FCST, | - | | MAXTRIES_RUN_FCST, KMP_AFFINITY_RUN_FCST, OMP_NUM_THREADS_RUN_FCST, | + | task_run_fcst | KMP_AFFINITY_RUN_FCST, OMP_NUM_THREADS_RUN_FCST, | | | OMP_STACKSIZE_RUN_FCST, DT_ATMOS, RESTART_INTERVAL, WRITE_DOPOST, | | | LAYOUT_X, LAYOUT_Y, BLOCKSIZE, QUILTING, PRINT_ESMF, | | | WRTCMP_write_groups, WRTCMP_write_tasks_per_group, | @@ -251,8 +234,7 @@ Configuration parameters in the ``config_defaults.yaml`` file appear in :numref: | | FV3_NML_VARNAME_TO_SFC_CLIMO_FIELD_MAPPING, | | | CYCLEDIR_LINKS_TO_FIXam_FILES_MAPPING | +-----------------------------+-----------------------------------------------------------------------+ - | task_run_post | NNODES_RUN_POST, PPN_RUN_POST, WTIME_RUN_POST, | - | | MAXTRIES_RUN_POST, KMP_AFFINITY_RUN_POST, OMP_NUM_THREADS_RUN_POST, | + | task_run_post | KMP_AFFINITY_RUN_POST, OMP_NUM_THREADS_RUN_POST, | | | OMP_STACKSIZE_RUN_POST, SUB_HOURLY_POST, DT_SUB_HOURLY_POST_MNTS, | | | USE_CUSTOM_POST_CONFIG_FILE, CUSTOM_POST_CONFIG_FP, | | | POST_OUTPUT_DOMAIN_NAME | @@ -267,80 +249,7 @@ Configuration parameters in the ``config_defaults.yaml`` file appear in :numref: | | LSM_SPP_TSCALE, LSM_SPP_LSCALE, ISEED_LSM_SPP, LSM_SPP_VAR_LIST, | | | LSM_SPP_MAG_LIST, HALO_BLEND | +-----------------------------+-----------------------------------------------------------------------+ - | task_get_obs_ccpa | NNODES_GET_OBS_CCPA, PPN_GET_OBS_CCPA, | - | | WTIME_GET_OBS_CCPA, MAXTRIES_GET_OBS_CCPA | - +-----------------------------+-----------------------------------------------------------------------+ - | task_get_obs_mrms | NNODES_GET_OBS_MRMS, PPN_GET_OBS_MRMS, | - | | WTIME_GET_OBS_MRMS, MAXTRIES_GET_OBS_MRMS | - +-----------------------------+-----------------------------------------------------------------------+ - | task_get_obs_ndas | NNODES_GET_OBS_NDAS, PPN_GET_OBS_NDAS, | - | | WTIME_GET_OBS_NDAS, MAXTRIES_GET_OBS_NDAS | - +-----------------------------+-----------------------------------------------------------------------+ - | task_run_vx_gridstat | NNODES_VX_GRIDSTAT, PPN_VX_GRIDSTAT, | - | | WTIME_VX_GRIDSTAT, MAXTRIES_VX_GRIDSTAT | - +-----------------------------+-----------------------------------------------------------------------+ - | task_run_vx_gridstat_refc | NNODES_VX_GRIDSTAT, PPN_VX_GRIDSTAT, | - | | WTIME_VX_GRIDSTAT, MAXTRIES_VX_GRIDSTAT_REFC | - +-----------------------------+-----------------------------------------------------------------------+ - | task_run_vx_gridstat_retop | NNODES_VX_GRIDSTAT, PPN_VX_GRIDSTAT, | - | | WTIME_VX_GRIDSTAT, MAXTRIES_VX_GRIDSTAT_RETOP | - +-----------------------------+-----------------------------------------------------------------------+ - | task_run_vx_gridstat_03h | NNODES_VX_GRIDSTAT, PPN_VX_GRIDSTAT, | - | | WTIME_VX_GRIDSTAT, MAXTRIES_VX_GRIDSTAT_03h | - +-----------------------------+-----------------------------------------------------------------------+ - | task_run_vx_gridstat_06h | NNODES_VX_GRIDSTAT, PPN_VX_GRIDSTAT, | - | | WTIME_VX_GRIDSTAT, MAXTRIES_VX_GRIDSTAT_06h | - +-----------------------------+-----------------------------------------------------------------------+ - | task_run_vx_gridstat_24h | NNODES_VX_GRIDSTAT, PPN_VX_GRIDSTAT, | - | | WTIME_VX_GRIDSTAT, MAXTRIES_VX_GRIDSTAT_24h | - +-----------------------------+-----------------------------------------------------------------------+ - | task_run_vx_pointstat | NNODES_VX_POINTSTAT, PPN_VX_POINTSTAT, | - | | WTIME_VX_POINTSTAT, MAXTRIES_VX_POINTSTAT | - +-----------------------------+-----------------------------------------------------------------------+ - | task_run_vx_ensgrid | MAXTRIES_VX_ENSGRID_03h, | - | | MAXTRIES_VX_ENSGRID_06h, MAXTRIES_VX_ENSGRID_24h, | - | | MAXTRIES_VX_ENSGRID_RETOP, | - | | MAXTRIES_VX_ENSGRID_PROB_RETOP, | - | | NNODES_VX_ENSGRID, PPN_VX_ENSGRID, WTIME_VX_ENSGRID, | - | | MAXTRIES_VX_ENSGRID | - +-----------------------------+-----------------------------------------------------------------------+ - | task_run_vx_ensgrid_refc | NNODES_VX_ENSGRID, PPN_VX_ENSGRID, | - | | WTIME_VX_ENSGRID, MAXTRIES_VX_ENSGRID_REFC | - +-----------------------------+-----------------------------------------------------------------------+ - | task_run_vx_ensgrid_mean | NNODES_VX_ENSGRID_MEAN, PPN_VX_ENSGRID_MEAN, | - | | WTIME_VX_ENSGRID_MEAN, MAXTRIES_VX_ENSGRID_MEAN | - +-----------------------------+-----------------------------------------------------------------------+ - | task_run_vx_ensgrid_mean_03h| NNODES_VX_ENSGRID_MEAN, PPN_VX_ENSGRID_MEAN, | - | | WTIME_VX_ENSGRID_MEAN, MAXTRIES_VX_ENSGRID_MEAN_03h | - +-----------------------------+-----------------------------------------------------------------------+ - | task_run_vx_ensgrid_mean_06h| NNODES_VX_ENSGRID_MEAN, PPN_VX_ENSGRID_MEAN, | - | | WTIME_VX_ENSGRID_MEAN, MAXTRIES_VX_ENSGRID_MEAN_06h | - +-----------------------------+-----------------------------------------------------------------------+ - | task_run_vx_ensgrid_mean_24h| NNODES_VX_ENSGRID_MEAN, PPN_VX_ENSGRID_MEAN, | - | | WTIME_VX_ENSGRID_MEAN, MAXTRIES_VX_ENSGRID_MEAN_24h | - +-----------------------------+-----------------------------------------------------------------------+ - | task_run_vx_ensgrid_prob | NNODES_VX_ENSGRID_PROB, PPN_VX_ENSGRID_PROB, | - | | WTIME_VX_ENSGRID_PROB, MAXTRIES_VX_ENSGRID_PROB | - +-----------------------------+-----------------------------------------------------------------------+ - | task_run_vx_ensgrid_prob_03h| NNODES_VX_ENSGRID_PROB, PPN_VX_ENSGRID_PROB, | - | | WTIME_VX_ENSGRID_PROB, MAXTRIES_VX_ENSGRID_PROB_03h | - +-----------------------------+-----------------------------------------------------------------------+ - | task_run_vx_ensgrid_prob_06h| NNODES_VX_ENSGRID_PROB, PPN_VX_ENSGRID_PROB, | - | | WTIME_VX_ENSGRID_PROB, MAXTRIES_VX_ENSGRID_PROB_06h | - +-----------------------------+-----------------------------------------------------------------------+ - | task_run_vx_ensgrid_prob_24h| NNODES_VX_ENSGRID_PROB, PPN_VX_ENSGRID_PROB, | - | | WTIME_VX_ENSGRID_PROB, MAXTRIES_VX_ENSGRID_PROB_24h | - +-----------------------------+-----------------------------------------------------------------------+ - | task_run_vx_enspoint | NNODES_VX_ENSPOINT, PPN_VX_ENSPOINT, | - | | WTIME_VX_ENSPOINT, MAXTRIES_VX_ENSPOINT | - +-----------------------------+-----------------------------------------------------------------------+ - | task_run_vx_enspoint_mean | NNODES_VX_ENSPOINT_MEAN, PPN_VX_ENSPOINT_MEAN, | - | | WTIME_VX_ENSPOINT_MEAN, MAXTRIES_VX_ENSPOINT_MEAN | - +-----------------------------+-----------------------------------------------------------------------+ - | task_run_vx_enspoint_prob | NNODES_VX_ENSPOINT_PROB, PPN_VX_ENSPOINT_PROB, | - | | WTIME_VX_ENSPOINT_PROB, MAXTRIES_VX_ENSPOINT_PROB | - +-----------------------------+-----------------------------------------------------------------------+ - + .. _UserSpecificConfig: User-specific configuration: ``config.yaml`` From e4c65705e6d2d4c7705e7fa0ab6e45376013f775 Mon Sep 17 00:00:00 2001 From: Christina Holt Date: Tue, 18 Apr 2023 18:51:28 -0600 Subject: [PATCH 03/10] Remove RUN_TASK references. --- docs/UsersGuide/source/ConfigWorkflow.rst | 46 +----- docs/UsersGuide/source/RunSRW.rst | 180 ++++++++++++---------- 2 files changed, 106 insertions(+), 120 deletions(-) diff --git a/docs/UsersGuide/source/ConfigWorkflow.rst b/docs/UsersGuide/source/ConfigWorkflow.rst index 1222e077a6..07cfdbc9cf 100644 --- a/docs/UsersGuide/source/ConfigWorkflow.rst +++ b/docs/UsersGuide/source/ConfigWorkflow.rst @@ -110,7 +110,7 @@ These parameters vary depending on machine. On `Level 1 and 2 `. When pulling observations from NOAA HPSS, the data retrieved will be placed in the ``CCPA_OBS_DIR`` directory. This path must be defind as ``//ccpa/proc``. METplus is configured to verify 01-, 03-, 06-, and 24-h accumulated precipitation using hourly CCPA files. @@ -159,7 +159,7 @@ METplus Parameters There is a problem with the valid time in the metadata for files valid from 19 - 00 UTC (i.e., files under the "00" directory). The script to pull the CCPA data from the NOAA HPSS (``scripts/exregional_get_obs_ccpa.sh``) has an example of how to account for this and organize the data into a more intuitive format. When a fix is provided, it will be accounted for in the ``exregional_get_obs_ccpa.sh`` script. ``MRMS_OBS_DIR``: (Default: "") - User-specified location of top-level directory where MRMS composite reflectivity files used by METplus are located. This parameter needs to be set for both user-provided observations and for observations that are retrieved from the NOAA :term:`HPSS` (if the user has access) via the ``GET_OBS_MRMS`` task (activated in the workflow by setting ``RUN_TASK_GET_OBS_MRMS: true``). When pulling observations directly from NOAA HPSS, the data retrieved will be placed in this directory. Please note, this path must be defind as ``//mrms/proc``. + User-specified location of top-level directory where MRMS composite reflectivity files used by METplus are located. This parameter needs to be set for both user-provided observations and for observations that are retrieved from the NOAA :term:`HPSS` (if the user has access) via the ``GET_OBS_MRMS`` task (activated in the workflow automatically when using the taskgroup file ``parm/wflow/verify.yaml``). When pulling observations directly from NOAA HPSS, the data retrieved will be placed in this directory. Please note, this path must be defind as ``//mrms/proc``. METplus configuration files require the use of a predetermined directory structure and file names. Therefore, if the MRMS files are user-provided, they need to follow the anticipated naming structure: ``{YYYYMMDD}/MergedReflectivityQCComposite_00.50_{YYYYMMDD}-{HH}{mm}{SS}.grib2``, where YYYYMMDD and {HH}{mm}{SS} are as described in the note :ref:`above `. @@ -167,7 +167,7 @@ METplus Parameters METplus is configured to look for a MRMS composite reflectivity file for the valid time of the forecast being verified; since MRMS composite reflectivity files do not always exactly match the valid time, a script (within the main script that retrieves MRMS data from the NOAA HPSS) is used to identify and rename the MRMS composite reflectivity file to match the valid time of the forecast. The script to pull the MRMS data from the NOAA HPSS has an example of the expected file-naming structure: ``scripts/exregional_get_obs_mrms.sh``. This script calls the script used to identify the MRMS file closest to the valid time: ``ush/mrms_pull_topofhour.py``. ``NDAS_OBS_DIR``: (Default: "") - User-specified location of the top-level directory where NDAS prepbufr files used by METplus are located. This parameter needs to be set for both user-provided observations and for observations that are retrieved from the NOAA :term:`HPSS` (if the user has access) via the ``GET_OBS_NDAS`` task (activated in the workflow by setting ``RUN_TASK_GET_OBS_NDAS: true``). When pulling observations directly from NOAA HPSS, the data retrieved will be placed in this directory. Please note, this path must be defined as ``//ndas/proc``. METplus is configured to verify near-surface variables hourly and upper-air variables at 00 and 12 UTC with NDAS prepbufr files. + User-specified location of the top-level directory where NDAS prepbufr files used by METplus are located. This parameter needs to be set for both user-provided observations and for observations that are retrieved from the NOAA :term:`HPSS` (if the user has access) via the ``GET_OBS_NDAS`` task (activated in the workflow by automatically when using the taskgroup file ``parm/wflow/verify.yaml``). When pulling observations directly from NOAA HPSS, the data retrieved will be placed in this directory. Please note, this path must be defined as ``//ndas/proc``. METplus is configured to verify near-surface variables hourly and upper-air variables at 00 and 12 UTC with NDAS prepbufr files. METplus configuration files require the use of predetermined file names. Therefore, if the NDAS files are user-provided, they need to follow the anticipated naming structure: ``prepbufr.ndas.{YYYYMMDDHH}``, where YYYYMMDDHH is as described in the note :ref:`above `. The script to pull the NDAS data from the NOAA HPSS (``scripts/exregional_get_obs_ndas.sh``) has an example of how to rename the NDAS data into a more intuitive format with the valid time listed in the file name. @@ -415,38 +415,6 @@ A standard set of environment variables has been established for *nco* mode to s ``OPSROOT``: (Default: "") The operations root directory in *nco* mode. -Verification Tasks --------------------- - -``RUN_TASK_GET_OBS_CCPA``: (Default: false) - Flag that determines whether to run the ``GET_OBS_CCPA`` task, which retrieves the :term:`CCPA` hourly precipitation files used by METplus from NOAA :term:`HPSS`. See :numref:`Section %s ` for additional parameters related to this task. - -``RUN_TASK_GET_OBS_MRMS``: (Default: false) - Flag that determines whether to run the ``GET_OBS_MRMS`` task, which retrieves the :term:`MRMS` composite reflectivity files used by METplus from NOAA HPSS. See :numref:`Section %s ` for additional parameters related to this task. - -``RUN_TASK_GET_OBS_NDAS``: (Default: false) - Flag that determines whether to run the ``GET_OBS_NDAS`` task, which retrieves the :term:`NDAS` PrepBufr files used by METplus from NOAA HPSS. See :numref:`Section %s ` for additional parameters related to this task. - -``RUN_TASK_VX_GRIDSTAT``: (Default: false) - Flag that determines whether to run the grid-stat verification task. The :ref:`MET Grid-Stat tool ` provides verification statistics for a matched forecast and observation grid. See :numref:`Section %s ` for additional parameters related to this task. Valid values: ``True`` | ``False`` - -``RUN_TASK_VX_POINTSTAT``: (Default: false) - Flag that determines whether to run the point-stat verification task. The :ref:`MET Point-Stat tool ` provides verification statistics for forecasts at observation points (as opposed to over gridded analyses). See :numref:`Section %s ` for additional parameters related to this task. Valid values: ``True`` | ``False`` - -``RUN_TASK_VX_ENSGRID``: (Default: false) - Flag that determines whether to run the ensemble-stat verification for gridded data task. The :ref:`MET Ensemble-Stat tool ` provides verification statistics for ensemble forecasts and can be used in conjunction with the :ref:`MET Grid-Stat tool `. See :numref:`Section %s ` for additional parameters related to this task. Valid values: ``True`` | ``False`` - -``RUN_TASK_VX_ENSPOINT``: (Default: false) - Flag that determines whether to run the ensemble point verification task. If this flag is set, both ensemble-stat point verification and point verification of ensemble-stat output is computed. The :ref:`MET Ensemble-Stat tool ` provides verification statistics for ensemble forecasts and can be used in conjunction with the :ref:`MET Point-Stat tool `. See :numref:`Section %s ` for additional parameters related to this task. Valid values: ``True`` | ``False`` - -.. COMMENT: COMMENT: Define "ensemble-stat verification for gridded data," "ensemble point verification," "ensemble-stat point verification," and "point verification of ensemble-stat output"? - -Plotting Task ----------------- - -``RUN_TASK_PLOT_ALLVARS:`` (Default: false) - Flag that determines whether to run python plotting scripts. - .. _make-grid: MAKE_GRID Configuration Parameters @@ -460,7 +428,7 @@ Basic Task Parameters For each workflow task, certain parameter values must be passed to the job scheduler (e.g., Slurm), which submits a job for the task. Typically, users do not need to adjust the default values. ``GRID_DIR``: (Default: "") - The directory containing pre-generated grid files when ``RUN_TASK_MAKE_GRID`` is set to false. + The directory containing pre-generated grid files when the ``MAKE_GRID`` task is not meant to run. .. _ESGgrid: @@ -820,7 +788,7 @@ These parameters set values in the Weather Model's ``model_configure`` file. .. _InlinePost: ``WRITE_DOPOST``: (Default: false) - Flag that determines whether to use the inline post option. The default ``WRITE_DOPOST: false`` does not use the inline post functionality, and the ``run_post`` tasks are called from outside of the Weather Model. If ``WRITE_DOPOST: true``, the ``WRITE_DOPOST`` flag in the ``model_configure`` file will be set to true, and the post-processing (:term:`UPP`) tasks will be called from within the Weather Model. This means that the post-processed files (in :term:`grib2` format) are output by the Weather Model at the same time that it outputs the ``dynf###.nc`` and ``phyf###.nc`` files. Setting ``WRITE_DOPOST: true`` turns off the separate ``run_post`` task (i.e., ``RUN_TASK_RUN_POST`` is set to false) in ``setup.py`` to avoid unnecessary computations. Valid values: ``True`` | ``False`` + Flag that determines whether to use the inline post option. The default ``WRITE_DOPOST: false`` does not use the inline post functionality, and the ``run_post`` tasks are called from outside of the Weather Model. If ``WRITE_DOPOST: true``, the ``WRITE_DOPOST`` flag in the ``model_configure`` file will be set to true, and the post-processing (:term:`UPP`) tasks will be called from within the Weather Model. This means that the post-processed files (in :term:`grib2` format) are output by the Weather Model at the same time that it outputs the ``dynf###.nc`` and ``phyf###.nc`` files. Setting ``WRITE_DOPOST: true`` turns off the separate ``run_post`` task in ``setup.py`` to avoid unnecessary computations. Valid values: ``True`` | ``False`` Computational Parameters ---------------------------- @@ -983,7 +951,7 @@ These parameters are associated with the fixed (i.e., static) files. On `Level 1 The location on disk of the static input files used by the ``make_orog`` task (i.e., ``orog.x`` and ``shave.x``). Can be the same as ``FIXgsm``. ``SFC_CLIMO_INPUT_DIR``: (Default: "") - The location on disk of the static surface climatology input fields, used by ``sfc_climo_gen``. These files are only used if ``RUN_TASK_MAKE_SFC_CLIMO: true``. + The location on disk of the static surface climatology input fields, used by ``sfc_climo_gen``. These files are only used if the ``MAKE_SFC_CLIMO`` is meant to run. ``SYMLINK_FIX_FILES``: (Default: true) Flag that indicates whether to symlink or copy fix files to the experiment directory. diff --git a/docs/UsersGuide/source/RunSRW.rst b/docs/UsersGuide/source/RunSRW.rst index 0a323964f6..7f1a41bee3 100644 --- a/docs/UsersGuide/source/RunSRW.rst +++ b/docs/UsersGuide/source/RunSRW.rst @@ -298,26 +298,6 @@ The user must specify certain basic experiment configuration information in a `` +--------------------------------+-------------------+------------------------------------+ | COMPILER | "intel" | "intel" | +--------------------------------+-------------------+------------------------------------+ - | RUN_TASK_MAKE_GRID | true | true | - +--------------------------------+-------------------+------------------------------------+ - | RUN_TASK_MAKE_OROG | true | true | - +--------------------------------+-------------------+------------------------------------+ - | RUN_TASK_MAKE_SFC_CLIMO | true | true | - +--------------------------------+-------------------+------------------------------------+ - | RUN_TASK_GET_OBS_CCPA | false | false | - +--------------------------------+-------------------+------------------------------------+ - | RUN_TASK_GET_OBS_MRMS | false | false | - +--------------------------------+-------------------+------------------------------------+ - | RUN_TASK_GET_OBS_NDAS | false | false | - +--------------------------------+-------------------+------------------------------------+ - | RUN_TASK_VX_GRIDSTAT | false | false | - +--------------------------------+-------------------+------------------------------------+ - | RUN_TASK_VX_POINTSTAT | false | false | - +--------------------------------+-------------------+------------------------------------+ - | RUN_TASK_VX_ENSGRID | false | false | - +--------------------------------+-------------------+------------------------------------+ - | RUN_TASK_VX_ENSPOINT | false | false | - +--------------------------------+-------------------+------------------------------------+ | EXTRN_MDL_NAME_ICS | "FV3GFS" | "FV3GFS" | +--------------------------------+-------------------+------------------------------------+ | FV3GFS_FILE_FMT_ICS | "nemsio" | "grib2" | @@ -476,12 +456,13 @@ The Python plotting tasks require a path to the directory where the Cartopy Natu Task Configuration ````````````````````` -Users will need to add or modify certain variables in ``config.yaml`` to run the plotting task(s). At a minimum, users must set ``RUN_TASK_PLOT_ALLVARS`` to true in the ``workflow_switches:`` section: +Users will need to add or modify certain variables in ``config.yaml`` to run the plotting task(s). At a minimum, to activate the ``plot_allvars`` tasks, users must add it to the default list of ``taskgroups`` under the ``rocoto: tasks:`` section. .. code-block:: console - workflow_switches: - RUN_TASK_PLOT_ALLVARS: true + rocoto: + tasks: + taskgroups: '{{ ["parm/wflow/prep.yaml", "parm/wflow/coldstart.yaml", "parm/wflow/post.yaml", "parm/wflow/plot.yaml"]|include }}' Users may also wish to adjust the start, end, and increment value for the plotting task. For example: @@ -706,34 +687,49 @@ To use METplus verification, the path to the MET and METplus directories must be METPLUS_PATH: MET_INSTALL_DIR: -Users who have already staged the observation data needed for METplus (i.e., the :term:`CCPA`, :term:`MRMS`, and :term:`NDAS` data) on their system should set the path to this data and set the corresponding ``RUN_TASK_GET_OBS_*`` parameters to false in ``config.yaml``. +To turn on verification tasks in the workflow, include the ``parm/wflow/verify.yaml`` file in the ``rocoto: tasks: taskgroups:`` section of ``config.yaml``. .. code-block:: console - platform: - CCPA_OBS_DIR: /path/to/UFS_SRW_App/develop/obs_data/ccpa/proc - MRMS_OBS_DIR: /path/to/UFS_SRW_App/develop/obs_data/mrms/proc - NDAS_OBS_DIR: /path/to/UFS_SRW_App/develop/obs_data/ndas/proc - workflow_switches: - RUN_TASK_GET_OBS_CCPA: false - RUN_TASK_GET_OBS_MRMS: false - RUN_TASK_GET_OBS_NDAS: false + rocoto: + tasks: + taskgroups: '{{ ["parm/wflow/prep.yaml", "parm/wflow/coldstart.yaml", "parm/wflow/post.yaml", "parm/wflow/verify.yaml"]|include }}' -If users have access to NOAA :term:`HPSS` but have not pre-staged the data, they can simply set the ``RUN_TASK_GET_OBS_*`` tasks to true, and the machine will attempt to download the appropriate data from NOAA HPSS. In this case, the ``*_OBS_DIR`` paths must be set to the location where users want the downloaded data to reside. +The ``verify.yaml`` file includes the definitions of several common verification tasks by default. They are independent of each other, so users may want to turn some off depending on the needs of their experiment. Note that the ENSGRID and ENSPOINT tasks apply only to ensemble model verification. Additional verification tasks appear in :numref:`Table %s `. More details on all of the parameters in this section are available in :numref:`Section %s `. + +To turn off a task, simply include its entry from ``verify.yaml`` as an empty YAML entry. For example, to turn off PointStat tasks: + +.. code-block:: console + + rocoto: + tasks: + taskgroups: '{{ ["parm/wflow/prep.yaml", "parm/wflow/coldstart.yaml", "parm/wflow/post.yaml", "parm/wflow/verify.yaml"]|include }}' + metatask_vx_ens_member: + metatask_PointStat_mem#mem#: + + +More information about configuring the ``rocoto:`` section can be found here. + +If users have access to NOAA :term:`HPSS` but have not pre-staged the data, the default ``verify.yaml`` taskgroup will activate the tasks, and the workflow will attempt to download the appropriate data from NOAA HPSS. In this case, the ``*_OBS_DIR`` paths must be set to the location where users want the downloaded data to reside. Users who do not have access to NOAA HPSS and do not have the data on their system will need to download :term:`CCPA`, :term:`MRMS`, and :term:`NDAS` data manually from collections of publicly available data, such as the ones listed `here `__. -Next, the verification tasks must be turned on according to the user's needs. Users should add some or all of the following tasks to ``config.yaml``, depending on the verification procedure(s) they have in mind: +Users who have already staged the observation data needed for METplus (i.e., the :term:`CCPA`, :term:`MRMS`, and :term:`NDAS` data) on their system should set the path to this data and turn off the corresponding task by including them with no entry in ``config.yaml``. .. code-block:: console - workflow_switches: - RUN_TASK_VX_GRIDSTAT: true - RUN_TASK_VX_POINTSTAT: true - RUN_TASK_VX_ENSGRID: true - RUN_TASK_VX_ENSPOINT: true + platform: + CCPA_OBS_DIR: /path/to/UFS_SRW_App/develop/obs_data/ccpa/proc + MRMS_OBS_DIR: /path/to/UFS_SRW_App/develop/obs_data/mrms/proc + NDAS_OBS_DIR: /path/to/UFS_SRW_App/develop/obs_data/ndas/proc + rocoto: + tasks: + taskgroups: '{{ ["parm/wflow/prep.yaml", "parm/wflow/coldstart.yaml", "parm/wflow/post.yaml", "parm/wflow/verify.yaml"]|include }}' + task_get_obs_ccpa: + task_get_obs_mrms: + task_get_obs_ndas: + -These tasks are independent, so users may set some values to true and others to false depending on the needs of their experiment. Note that the ENSGRID and ENSPOINT tasks apply only to ensemble model verification. Additional verification tasks appear in :numref:`Table %s `. More details on all of the parameters in this section are available in :numref:`Section %s `. .. _GenerateWorkflow: @@ -769,15 +765,13 @@ Description of Workflow Tasks .. note:: This section gives a general overview of workflow tasks. To begin running the workflow, skip to :numref:`Step %s ` -:numref:`Figure %s ` illustrates the overall workflow. Individual tasks that make up the workflow are specified in the ``FV3LAM_wflow.xml`` file. :numref:`Table %s ` describes the function of each baseline task. The first three pre-processing tasks; ``MAKE_GRID``, ``MAKE_OROG``, and ``MAKE_SFC_CLIMO``; are optional. If the user stages pre-generated grid, orography, and surface climatology fix files, these three tasks can be skipped by adding the following lines to the ``config.yaml`` file before running the ``generate_FV3LAM_wflow.py`` script: +:numref:`Figure %s ` illustrates the overall workflow. Individual tasks that make up the workflow are specified in the ``FV3LAM_wflow.xml`` file. :numref:`Table %s ` describes the function of each baseline task. The first three pre-processing tasks; ``MAKE_GRID``, ``MAKE_OROG``, and ``MAKE_SFC_CLIMO``; are optional. If the user stages pre-generated grid, orography, and surface climatology fix files, these three tasks can be skipped by removing the ``prep.yaml`` file from the default ``taskgroups`` entry in the ``config.yaml`` file before running the ``generate_FV3LAM_wflow.py`` script: .. code-block:: console - workflow_switches: - RUN_TASK_MAKE_GRID: false - RUN_TASK_MAKE_OROG: false - RUN_TASK_MAKE_SFC_CLIMO: false - + rocoto: + tasks: + taskgroups: '{{ ["parm/wflow/coldstart.yaml", "parm/wflow/post.yaml"]|include }}' .. _WorkflowTasksFig: @@ -831,17 +825,20 @@ In addition to the baseline tasks described in :numref:`Table %s Date: Tue, 18 Apr 2023 21:11:50 -0600 Subject: [PATCH 04/10] Adding new docs for YAML interface to XML. --- docs/UsersGuide/source/ConfigWorkflow.rst | 2 +- docs/UsersGuide/source/DefineWorkflow.rst | 242 ++++++++++++++++++++++ docs/UsersGuide/source/index.rst | 1 + 3 files changed, 244 insertions(+), 1 deletion(-) create mode 100644 docs/UsersGuide/source/DefineWorkflow.rst diff --git a/docs/UsersGuide/source/ConfigWorkflow.rst b/docs/UsersGuide/source/ConfigWorkflow.rst index 07cfdbc9cf..b4f453dc59 100644 --- a/docs/UsersGuide/source/ConfigWorkflow.rst +++ b/docs/UsersGuide/source/ConfigWorkflow.rst @@ -8,7 +8,7 @@ To create the experiment directory and workflow when running the SRW Application There is an extensive list of experiment parameters that a user can set when configuring the experiment. Not all of these parameters need to be set explicitly by the user in ``config.yaml``. If a user does not define a variable in the ``config.yaml`` script, its value in ``config_defaults.yaml`` will be used, or the value will be reset depending on other parameters, such as the platform (``MACHINE``) selected for the experiment. .. note:: - The ``config_defaults.yaml`` file contains the full list of experiment parameters that a user may set in ``config.yaml``. The user cannot set parameters in ``config.yaml`` that are not initialized in ``config_defaults.yaml``. + The ``config_defaults.yaml`` file contains the full list of experiment parameters that a user may set in ``config.yaml``. The user cannot set parameters in ``config.yaml`` that are not initialized in ``config_defaults.yaml``, with the notable exception in the ``rocoto`` section, described in :numref:`Chapter %s . The following is a list of the parameters in the ``config_defaults.yaml`` file. For each parameter, the default value and a brief description is provided. diff --git a/docs/UsersGuide/source/DefineWorkflow.rst b/docs/UsersGuide/source/DefineWorkflow.rst new file mode 100644 index 0000000000..2366bdcddf --- /dev/null +++ b/docs/UsersGuide/source/DefineWorkflow.rst @@ -0,0 +1,242 @@ +.. _DefineWorkflow: + +======================= +Defining a SRW Workflow +======================= + + +Rocoto is the primary workflow manager software used by the UFS SRW App. Because the SRW App supports a variety of research and operational configurations, it is important to incorporate the ability to build a static Rocoto XML from scratch. This means a user will be able to specify exactly which tasks are included in the workflow definition from a YAML configuration file. While many predefined workflows will exist with optional variants in the SRW App, additional tasks may be added by individuals for their scientific exploration needs. + +This guide explains how the Rocoto XML is built using a Jinja2 template (`Jinja docs here `__) and structured YAML files. The YAML follows the requirements in the `Rocoto documentation `__ with a few exceptions or additions outlined in this documentation. + +The Jinja2 Template +=================== + +In previous versions of the SRW App, the Jinja2 template to create the Rocoto XML was tightly coupled to specific configuration settings of the SRW App. It was built from a limited, pre-defined set of specific tasks, defining switches for those tasks to be included or not in the rendered XML. + +Now, the Jinja2 template is entirely agnostic to Short-Range Weather Application decisions and has been developed to wrap the features of Rocoto in an extensible, configurable format. + + +The ``rocoto`` section of ``config.py`` +======================================= +The structured YAML file that defines the Rocoto XML is meant to reflect the sections required by any Rocoto XML. That structure looks like this, with some example values filled in: + +.. code-block:: console + + rocoto: + attrs: + realtime: F + scheduler: slurm + cyclethrottle: 5 + corethrottle: + taskthrottle: + cycledefs: + groupname: + !startstopfreq ['2022102000', ‘2023102018’, ‘06:00:00’] + groupname2: + !startstopfreq ['2022102000', ‘2023102018’, ‘24:00:00’] + !startstopfreq ['2022102006', ‘2023102018’, ‘24:00:00’] + entities: + foo: 1 + bar: “/some/path” + log: "" + tasks: + taskgroups: '{{ ["parm/wflow/prep.yaml", "parm/wflow/coldstart.yaml", "parm/wflow/post.yaml"]|include }}' + task_*: + metatask_*: + +In addition to the structured YAML itself, the SRW App enables additional functionality in defining a YAML file. Often, PyYAML features are introduced and documented `here `__. In the above example, the ``!startstopfreq`` is an example of a PyYAML constructor. Supported constructors will be outlined :ref:`below `. There are also examples of using PyYAML anchors and aliases in the definition of groups of tasks in the SRW App. Please see `this documentation `__ for the behavior of PyYAML anchors and aliases. + +The use of Jinja2 templates inside the values of entries allows for the reference to other keys, mathematical operations, Jinja2 control structures, and the use of user-defined filters. Here, the ``include`` filter in the ``taskgroups`` entry is a user-defined filter. The supported filters are described in a section :ref:`below `. + +Under the Rocoto section, several subentries are required. They are described here using similar language as in the Rocoto documentation. + +``attrs``: Any of the attributes to the ``workflow`` tag in Rocoto. This is meant to contain a nested dictionary defining any of the Rocoto-supported attributes, where the key is the name of the attribute, and the value is what Rocoto expects. + +``cycledefs``: A dictionary with each key defining a group name for a ``cycledef`` tag, where the key’s value is a list of ``cycledef`` accepted strings. The PyYAML constructor ``!startstopfreq`` has been included here to help with the automated construction of the tag of that nature. The preferred option for the SRW App is to use the “start, stop, step” method. + +``entities``: A dictionary with each key defining the name of a Rocoto entity, and its value. These variables are referenceable throughout the workflow with the ‘&foo;’ notation. + +``log``: The path to the log file. This corresponds to the ```` tag. + +``tasks``: This section is where the defined tasks and metatasks are created. This is the main portion of the workflow that will most commonly differ from experiment to experiment with different configurations. + +.. _tasks: + +The ``tasks`` Subsection +======================== + +``taskgroups``: This entry is not a standard Rocoto entry. It defines a set of files that will be included to build a workflow from predefined groups of tasks. The supported groups are included under parm/wflow for the SRW App, but the paths can point to any location on your local disk. The resulting order of the tasks will be in the same order as defined in this list. The syntax for the “include” is included as a Jinja2 filter. + +``task_*``: This is a section header to add a task. The task name will be whatever the section key has defined after the first underscore. For example “task_run_fcst” will be named “run_fcst” in the resulting workflow. More information about defining a task is included :ref:`below `. + +``metatask_*``: This is a section header to add a metatask. The metatask name will be whatever the section key has defined after the first underscore. For example “metatask_run_ensemble” will be named “run_ensemble” in the resulting workflow. More information about defining a metatask is included :ref:`below `. + +.. _defining_tasks: + +Defining a Task +=============== +Each task supports any of the tags that are defined in the Rocoto documentation. Here’s an example of a task: + +.. code-block:: console + + task_make_grid: + account: '&ACCOUNT;' + command: '&LOAD_MODULES_RUN_TASK_FP; "make_grid" + attrs: + cycledefs: at_start + maxtries: '2' + envars: &default_envars + GLOBAL_VAR_DEFNS_FP: '&GLOBAL_VAR_DEFNS_FP;' + USHdir: '&USHdir;' + PDY: !cycstr "@Y@m@d" + cyc: !cycstr "@H" + subcyc: !cycstr "@M" + LOGDIR: !cycstr "&LOGDIR;" + nprocs: '{{ parent.nnodes * parent.ppn }}' + native: '{{ platform.SCHED_NATIVE_CMD }}' + nodes: '{{ nnodes }}:ppn={{ ppn }}' + nnodes: 1 + nodesize: "&NCORES_PER_NODE;" + ppn: 24 + partition: '{% if platform.get("PARTITION_DEFAULT") %}&PARTITION_DEFAULT;{% else %}None{% endif %}' + queue: '&QUEUE_DEFAULT;' + walltime: 00:20:00 + dependency: + + +The following sections are constructs of the interface, while all others are direct translations to tags available in Rocoto. Any tag that allows for attributes to the XML tag will take an attrs nested dictionary entry. + +``attrs``: Any of the attributes to the task tag in Rocoto. This is meant to be a subdictionary defining any of the Rocoto-supported attributes, where the key is the name of the attribute, and the value is what Rocoto expects. This might include any combination of the following: cycledefs, maxtries, throttle, or final. + +``envars``: A dictionary of keys that map to variable names that will be exported for the job. These will show up as the set of ```` tags in the XML. The value will be the value of the defined variable when it is exported. + + +If the command entry is not provided, the task won’t show up in the resulting workflow. + +Defining Dependencies +===================== + +The dependency entry will be an arbitrarily deep nested dictionary of key, value pairs. Each level represents entries that must come below it in priority. This is especially relevant for logic files. If an “and” tag must apply to multiple dependencies, those dependencies are all included as a nested dictionary of dependencies. + +Because we are representing these entries as a dictionary, which requires hashable keys (no repeats at the same level), some tags may need to be differentiated where XML may not differentiate at all. In these instances, it’s best practice to name them something descriptive. For example, you might have multiple “or” dependencies at the same level that could be named “or_files_exist” and “or_task_ran”. This style can be adopted whether or not differentiation is needed. + +The text entry on some dependencies is for those dependency tags that need the information to come between two flags, as in a data dependency. + +Otherwise, all dependencies follow the same naming conventions as defined in Rocoto with ``attrs`` dictionaries included to define any of the tag attributes that may be accepted by Rocoto. + +Here is an example of a complex dependency that relies on logic, task dependencies, and data dependencies: + +.. code-block:: console + + dependency: + and: + or_get_obs: # Ensure get_obs task is complete if it's turned on + not: + taskvalid: + attrs: + task: get_obs_mrms + and: + taskvalid: + attrs: + task: get_obs_mrms + taskdep: + attrs: + task: get_obs_mrms + or_do_post: &post_files_exist + and_run_post: # If post was meant to run, wait on the whole post metatask + taskvalid: + attrs: + task: run_post_mem#mem#_f000 + metataskdep: + attrs: + metatask: run_ens_post + and_inline_post: # If inline post ran, wait on the forecast task to complete + not: + taskvalid: + attrs: + task: run_post_mem#mem#_f000 + taskdep: + attrs: + task: run_fcst_mem#mem# + +Notice the use of a PyYAML anchor under the ``or_do_post`` section. If other tasks need this same section of the dependency, it can be included like this to reduce the extensive replication: + +.. code-block:: console + + dependency: + or_do_post: + <<: *post_files_exist + datadep: + text: "&CCPA_OBS_DIR;" + +The use of ``#mem#`` here is a Rocoto construct that identifies this task as a part of a metatask that is looping over ensemble members (more on metatasks below). + +.. _defining_metatasks: + +Defining a Metatask +=================== + +A metatask groups together similar tasks and allows for the definition over entries defined by ``var`` tags. To define a metatask, the ``var`` entry with a nested dictionary of keys representing the names of the metatask variables and values indicating the list of values for each iteration, is required. + +Multiple var entries may be included, but each entry must have the same number of items. + +The metatask section must include at least one entry defining another metatask or a task. + +Here’s an example of a metatask section (without the task definition): + +.. code-block:: console + + metatask_run_ensemble: + var: + mem: '{% if global.DO_ENSEMBLE %}{%- for m in range(1, global.NUM_ENS_MEMBERS+1) -%}{{ "%03d "%m }}{%- endfor -%} {% else %}{{ "000"|string }}{% endif %}' + task_make_ics_mem#mem#: + +This metatask will be named “run_ensemble” and will loop over all ensemble members, or just the deterministic member (“000”) if no ensemble of forecasts is meant to run. + +The ``var`` section defines the metatask variables, here only “mem”. The name of the task represents that variable using ``#mem#`` to indicate that the resulting task name might be ``make_ics_mem000`` if only a deterministic forecast is configured to run. + +When the task or the metatask is referenced in a dependency later on, do not include the ``task_`` or ```metatask_`` portions of the name. The reference to ``#mem#`` can be included if the dependency is included in a metatask that defines the variable, e.g., ``make_ics_mem#mem#``. Otherwise, you can reference a task that includes the value of the metatask var, e.g., ``make_ics_mem000``. More on this distinction is included in the Rocoto documentation. + +.. _J2filters: + +SRW-Defined Jinja2 Filters Used by YAML Interface +================================================= + +``include()`` – given a list of files to other YAML files, load their contents as a nested dictionary under the entry. + +.. _YAMLconstructors: + +SRW-Defined PyYAML Constructors Used by YAML Interface +====================================================== + +``!cycstr`` - Returns a ```` element for use in Rocoto. It does not support the “offset” attribute. + +``!startstopfreq`` – Creates a Rocoto XML-formatted string given a start, stop, and freq value in a list. + +Order of Precedence +=================== +There is a specific order of precedence imposed when the SRW App loads configuration files. + +#. Load config_defaults.yaml file. +#. Load the user’s config.yaml file. +#. Load the workflow defaults YAML file. +#. At this point, all anchors and references will be resolved. +#. All PyYAML constructors will also be called for the data provided in that entry. +#. Call update_dict function to remove any null entries from default tasks using the PyYAML anchors. +#. Load all files from the ``taskgroups:`` entry from the user’s config, or default if not overridden. This is achieved with a call to the ``extend_yaml()`` function. +#. Add the contents of the files to the ``task:`` section. +#. Update the existing workflow configuration with any user-specified entries (removing the ones that are null entries). +#. Add a ``jobname:`` entry to every task in the workflow definition section. + +#. Incorporate other default configuration settings from machine files, constants, etc into the default config dictionary in memory. +#. Apply all user settings last to take highest precedence. +#. Call ``extend_yaml()`` to render templates that are available. + #. NOTE: This is the one that is likely to trip up any settings that setup.py will make. References to other defaults that get changed during the course of validation may be rendered here earlier than desired. + +At this point, validation and updates for many other configuration settings will be made for a variety of sections. Once complete, ``extend_yaml()`` is called repeatedly, stopping only when all possible Jinja2-templated values have been rendered. + +Just before the ``rocoto:`` section is written to its own file in the experiment directory, ``clean_rocoto_dict()`` is called on that section to remove invalid dictionaries, i.e., metatasks with no tasks, tasks with no associated commands, etc. + +The ``rocoto:`` section is not included in the ``var_defns.sh`` since that file is used primarily to store settings needed at run-time. + diff --git a/docs/UsersGuide/source/index.rst b/docs/UsersGuide/source/index.rst index e5cdbc3b2f..1346d0c025 100644 --- a/docs/UsersGuide/source/index.rst +++ b/docs/UsersGuide/source/index.rst @@ -22,6 +22,7 @@ UFS Short-Range Weather App Users Guide InputOutputFiles LAMGrids ConfigWorkflow + DefineWorkflow RocotoInfo WE2Etests TemplateVars From b504dbf51b4b59567a5922069f7bfc29315005e2 Mon Sep 17 00:00:00 2001 From: Christina Holt Date: Tue, 18 Apr 2023 21:33:29 -0600 Subject: [PATCH 05/10] Building errors addressed. --- docs/UsersGuide/source/ConfigWorkflow.rst | 11 ++++++----- docs/UsersGuide/source/DefineWorkflow.rst | 2 +- docs/UsersGuide/source/RunSRW.rst | 4 ++-- 3 files changed, 9 insertions(+), 8 deletions(-) diff --git a/docs/UsersGuide/source/ConfigWorkflow.rst b/docs/UsersGuide/source/ConfigWorkflow.rst index b4f453dc59..086c06db11 100644 --- a/docs/UsersGuide/source/ConfigWorkflow.rst +++ b/docs/UsersGuide/source/ConfigWorkflow.rst @@ -7,8 +7,8 @@ To create the experiment directory and workflow when running the SRW Application There is an extensive list of experiment parameters that a user can set when configuring the experiment. Not all of these parameters need to be set explicitly by the user in ``config.yaml``. If a user does not define a variable in the ``config.yaml`` script, its value in ``config_defaults.yaml`` will be used, or the value will be reset depending on other parameters, such as the platform (``MACHINE``) selected for the experiment. -.. note:: - The ``config_defaults.yaml`` file contains the full list of experiment parameters that a user may set in ``config.yaml``. The user cannot set parameters in ``config.yaml`` that are not initialized in ``config_defaults.yaml``, with the notable exception in the ``rocoto`` section, described in :numref:`Chapter %s . +.. note:: + The ``config_defaults.yaml`` file contains the full list of experiment parameters that a user may set in ``config.yaml``. The user cannot set parameters in ``config.yaml`` that are not initialized in ``config_defaults.yaml``, with the notable exception in the ``rocoto`` section, described in :numref:`Chapter %s `. The following is a list of the parameters in the ``config_defaults.yaml`` file. For each parameter, the default value and a brief description is provided. @@ -545,7 +545,7 @@ Non-default parameters for the ``make_orog`` task are set in the ``task_make_oro ``OROG_DIR``: (Default: "") The directory containing pre-generated orography files to use when the ``MAKE_OROG`` task is not meant to run. -f + .. _make-sfc-climo: MAKE_SFC_CLIMO Configuration Parameters @@ -1033,9 +1033,10 @@ For each workflow task, certain parameter values must be passed to the job sched ``TESTBED_FIELDS_FN``: (Default: "") The file which lists grib2 fields to be extracted for testbed files. Empty string means no need to generate testbed files. +.. _PlotVars: -Additional Parameters ------------------------- +PLOT_ALLVARS Configuration Parameters +======================================== Typically, the following parameters must be set explicitly by the user in the configuration file (``config.yaml``) when executing the plotting tasks. diff --git a/docs/UsersGuide/source/DefineWorkflow.rst b/docs/UsersGuide/source/DefineWorkflow.rst index 2366bdcddf..a2562494b4 100644 --- a/docs/UsersGuide/source/DefineWorkflow.rst +++ b/docs/UsersGuide/source/DefineWorkflow.rst @@ -232,7 +232,7 @@ There is a specific order of precedence imposed when the SRW App loads configura #. Incorporate other default configuration settings from machine files, constants, etc into the default config dictionary in memory. #. Apply all user settings last to take highest precedence. #. Call ``extend_yaml()`` to render templates that are available. - #. NOTE: This is the one that is likely to trip up any settings that setup.py will make. References to other defaults that get changed during the course of validation may be rendered here earlier than desired. + NOTE: This is the one that is likely to trip up any settings that setup.py will make. References to other defaults that get changed during the course of validation may be rendered here earlier than desired. At this point, validation and updates for many other configuration settings will be made for a variety of sections. Once complete, ``extend_yaml()`` is called repeatedly, stopping only when all possible Jinja2-templated values have been rendered. diff --git a/docs/UsersGuide/source/RunSRW.rst b/docs/UsersGuide/source/RunSRW.rst index 7f1a41bee3..76625a3c3e 100644 --- a/docs/UsersGuide/source/RunSRW.rst +++ b/docs/UsersGuide/source/RunSRW.rst @@ -194,7 +194,7 @@ Configuration parameters in the ``config_defaults.yaml`` file appear in :numref: | | GFDLgrid_JSTART_OF_RGNL_DOM_ON_T6G, GFDLgrid_JEND_OF_RGNL_DOM_ON_T6G, | | | GFDLgrid_USE_NUM_CELLS_IN_FILENAMES | +-----------------------------+-----------------------------------------------------------------------+ - | task_make_orog | KMP_AFFINITY_MAKE_OROG, OMP_NUM_THREADS_MAKE_OROG | + | task_make_orog | KMP_AFFINITY_MAKE_OROG, OMP_NUM_THREADS_MAKE_OROG | | | OMP_STACKSIZE_MAKE_OROG, OROG_DIR | +-----------------------------+-----------------------------------------------------------------------+ | task_make_sfc_climo | KMP_AFFINITY_MAKE_SFC_CLIMO, OMP_NUM_THREADS_MAKE_SFC_CLIMO, | @@ -695,7 +695,7 @@ To turn on verification tasks in the workflow, include the ``parm/wflow/verify.y tasks: taskgroups: '{{ ["parm/wflow/prep.yaml", "parm/wflow/coldstart.yaml", "parm/wflow/post.yaml", "parm/wflow/verify.yaml"]|include }}' -The ``verify.yaml`` file includes the definitions of several common verification tasks by default. They are independent of each other, so users may want to turn some off depending on the needs of their experiment. Note that the ENSGRID and ENSPOINT tasks apply only to ensemble model verification. Additional verification tasks appear in :numref:`Table %s `. More details on all of the parameters in this section are available in :numref:`Section %s `. +The ``verify.yaml`` file includes the definitions of several common verification tasks by default. They are independent of each other, so users may want to turn some off depending on the needs of their experiment. Note that the ENSGRID and ENSPOINT tasks apply only to ensemble model verification. Additional verification tasks appear in :numref:`Table %s `. To turn off a task, simply include its entry from ``verify.yaml`` as an empty YAML entry. For example, to turn off PointStat tasks: From 4e17da813285a9daa2a7835f6754df299f38e6ca Mon Sep 17 00:00:00 2001 From: Christina Holt <56881914+christinaholtNOAA@users.noreply.github.com> Date: Wed, 19 Apr 2023 18:47:17 -0600 Subject: [PATCH 06/10] Update docs/UsersGuide/source/ConfigWorkflow.rst Co-authored-by: Michael Lueken <63728921+MichaelLueken@users.noreply.github.com> --- docs/UsersGuide/source/ConfigWorkflow.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/UsersGuide/source/ConfigWorkflow.rst b/docs/UsersGuide/source/ConfigWorkflow.rst index 086c06db11..988c2eb701 100644 --- a/docs/UsersGuide/source/ConfigWorkflow.rst +++ b/docs/UsersGuide/source/ConfigWorkflow.rst @@ -167,7 +167,7 @@ METplus Parameters METplus is configured to look for a MRMS composite reflectivity file for the valid time of the forecast being verified; since MRMS composite reflectivity files do not always exactly match the valid time, a script (within the main script that retrieves MRMS data from the NOAA HPSS) is used to identify and rename the MRMS composite reflectivity file to match the valid time of the forecast. The script to pull the MRMS data from the NOAA HPSS has an example of the expected file-naming structure: ``scripts/exregional_get_obs_mrms.sh``. This script calls the script used to identify the MRMS file closest to the valid time: ``ush/mrms_pull_topofhour.py``. ``NDAS_OBS_DIR``: (Default: "") - User-specified location of the top-level directory where NDAS prepbufr files used by METplus are located. This parameter needs to be set for both user-provided observations and for observations that are retrieved from the NOAA :term:`HPSS` (if the user has access) via the ``GET_OBS_NDAS`` task (activated in the workflow by automatically when using the taskgroup file ``parm/wflow/verify.yaml``). When pulling observations directly from NOAA HPSS, the data retrieved will be placed in this directory. Please note, this path must be defined as ``//ndas/proc``. METplus is configured to verify near-surface variables hourly and upper-air variables at 00 and 12 UTC with NDAS prepbufr files. + User-specified location of the top-level directory where NDAS prepbufr files used by METplus are located. This parameter needs to be set for both user-provided observations and for observations that are retrieved from the NOAA :term:`HPSS` (if the user has access) via the ``GET_OBS_NDAS`` task (activated in the workflow automatically when using the taskgroup file ``parm/wflow/verify.yaml``). When pulling observations directly from NOAA HPSS, the data retrieved will be placed in this directory. Please note, this path must be defined as ``//ndas/proc``. METplus is configured to verify near-surface variables hourly and upper-air variables at 00 and 12 UTC with NDAS prepbufr files. METplus configuration files require the use of predetermined file names. Therefore, if the NDAS files are user-provided, they need to follow the anticipated naming structure: ``prepbufr.ndas.{YYYYMMDDHH}``, where YYYYMMDDHH is as described in the note :ref:`above `. The script to pull the NDAS data from the NOAA HPSS (``scripts/exregional_get_obs_ndas.sh``) has an example of how to rename the NDAS data into a more intuitive format with the valid time listed in the file name. From 55ea12aba20144531ca96c00cbe70ac9b41f6ed1 Mon Sep 17 00:00:00 2001 From: Christina Holt Date: Wed, 19 Apr 2023 18:47:34 -0600 Subject: [PATCH 07/10] Add reference to rocoto section. --- docs/UsersGuide/source/RunSRW.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/UsersGuide/source/RunSRW.rst b/docs/UsersGuide/source/RunSRW.rst index 76625a3c3e..11ed8f360c 100644 --- a/docs/UsersGuide/source/RunSRW.rst +++ b/docs/UsersGuide/source/RunSRW.rst @@ -708,7 +708,7 @@ To turn off a task, simply include its entry from ``verify.yaml`` as an empty YA metatask_PointStat_mem#mem#: -More information about configuring the ``rocoto:`` section can be found here. +More information about configuring the ``rocoto:`` section can be found in :numref:`Section %s `. If users have access to NOAA :term:`HPSS` but have not pre-staged the data, the default ``verify.yaml`` taskgroup will activate the tasks, and the workflow will attempt to download the appropriate data from NOAA HPSS. In this case, the ``*_OBS_DIR`` paths must be set to the location where users want the downloaded data to reside. From c331647f9016920464a79c45217c9c02369471bf Mon Sep 17 00:00:00 2001 From: Gillian Petro <96886803+gspetro-NOAA@users.noreply.github.com> Date: Fri, 21 Apr 2023 10:00:38 -0400 Subject: [PATCH 08/10] Add "task" after task name --- docs/UsersGuide/source/ConfigWorkflow.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/UsersGuide/source/ConfigWorkflow.rst b/docs/UsersGuide/source/ConfigWorkflow.rst index 988c2eb701..4610c18f32 100644 --- a/docs/UsersGuide/source/ConfigWorkflow.rst +++ b/docs/UsersGuide/source/ConfigWorkflow.rst @@ -563,7 +563,7 @@ Non-default parameters for the ``make_sfc_climo`` task are set in the ``task_mak Controls the size of the stack for threads created by the OpenMP implementation. ``SFC_CLIMO_DIR``: (Default: "") - The directory containing pre-generated surface climatology files to use when the ``MAKE_SFC_CLIMO`` is not meant to run. + The directory containing pre-generated surface climatology files to use when the ``MAKE_SFC_CLIMO`` task is not meant to run. .. _task_get_extrn_ics: @@ -951,7 +951,7 @@ These parameters are associated with the fixed (i.e., static) files. On `Level 1 The location on disk of the static input files used by the ``make_orog`` task (i.e., ``orog.x`` and ``shave.x``). Can be the same as ``FIXgsm``. ``SFC_CLIMO_INPUT_DIR``: (Default: "") - The location on disk of the static surface climatology input fields, used by ``sfc_climo_gen``. These files are only used if the ``MAKE_SFC_CLIMO`` is meant to run. + The location on disk of the static surface climatology input fields, used by ``sfc_climo_gen``. These files are only used if the ``MAKE_SFC_CLIMO`` task is meant to run. ``SYMLINK_FIX_FILES``: (Default: true) Flag that indicates whether to symlink or copy fix files to the experiment directory. From e147319af222504a151ab3b73d7502172b6f30d7 Mon Sep 17 00:00:00 2001 From: Gillian Petro <96886803+gspetro-NOAA@users.noreply.github.com> Date: Fri, 21 Apr 2023 20:49:29 -0400 Subject: [PATCH 09/10] Minor edits for DefineWorkflow.rst --- docs/UsersGuide/source/DefineWorkflow.rst | 38 +++++++++++------------ 1 file changed, 19 insertions(+), 19 deletions(-) diff --git a/docs/UsersGuide/source/DefineWorkflow.rst b/docs/UsersGuide/source/DefineWorkflow.rst index a2562494b4..470d04ea95 100644 --- a/docs/UsersGuide/source/DefineWorkflow.rst +++ b/docs/UsersGuide/source/DefineWorkflow.rst @@ -1,8 +1,8 @@ .. _DefineWorkflow: -======================= -Defining a SRW Workflow -======================= +============================= +Defining an SRW App Workflow +============================= Rocoto is the primary workflow manager software used by the UFS SRW App. Because the SRW App supports a variety of research and operational configurations, it is important to incorporate the ability to build a static Rocoto XML from scratch. This means a user will be able to specify exactly which tasks are included in the workflow definition from a YAML configuration file. While many predefined workflows will exist with optional variants in the SRW App, additional tasks may be added by individuals for their scientific exploration needs. @@ -53,9 +53,9 @@ Under the Rocoto section, several subentries are required. They are described he ``attrs``: Any of the attributes to the ``workflow`` tag in Rocoto. This is meant to contain a nested dictionary defining any of the Rocoto-supported attributes, where the key is the name of the attribute, and the value is what Rocoto expects. -``cycledefs``: A dictionary with each key defining a group name for a ``cycledef`` tag, where the key’s value is a list of ``cycledef`` accepted strings. The PyYAML constructor ``!startstopfreq`` has been included here to help with the automated construction of the tag of that nature. The preferred option for the SRW App is to use the “start, stop, step” method. +``cycledefs``: A dictionary with each key defining a group name for a ``cycledef`` tag, where the key’s value is a list of acceptable ``cycledef`` formatted strings. The PyYAML constructor ``!startstopfreq`` has been included here to help with the automated construction of a tag of that nature. The preferred option for the SRW App is to use the “start, stop, step” method. -``entities``: A dictionary with each key defining the name of a Rocoto entity, and its value. These variables are referenceable throughout the workflow with the ‘&foo;’ notation. +``entities``: A dictionary with each key defining the name of a Rocoto entity and its value. These variables are referenceable throughout the workflow with the ‘&foo;’ notation. ``log``: The path to the log file. This corresponds to the ```` tag. @@ -66,9 +66,9 @@ Under the Rocoto section, several subentries are required. They are described he The ``tasks`` Subsection ======================== -``taskgroups``: This entry is not a standard Rocoto entry. It defines a set of files that will be included to build a workflow from predefined groups of tasks. The supported groups are included under parm/wflow for the SRW App, but the paths can point to any location on your local disk. The resulting order of the tasks will be in the same order as defined in this list. The syntax for the “include” is included as a Jinja2 filter. +``taskgroups``: This entry is not a standard Rocoto entry. It defines a set of files that will be included to build a workflow from predefined groups of tasks. The supported groups are included under ``parm/wflow`` for the SRW App, but the paths can point to any location on your local disk. The resulting order of the tasks will be in the same order as defined in this list. The syntax for the “include” is included as a Jinja2 filter. -``task_*``: This is a section header to add a task. The task name will be whatever the section key has defined after the first underscore. For example “task_run_fcst” will be named “run_fcst” in the resulting workflow. More information about defining a task is included :ref:`below `. +``task_*``: This is a section header to add a task. The task name will be whatever the section key has defined after the first underscore. For example, “task_run_fcst” will be named “run_fcst” in the resulting workflow. More information about defining a task is included :ref:`below `. ``metatask_*``: This is a section header to add a metatask. The metatask name will be whatever the section key has defined after the first underscore. For example “metatask_run_ensemble” will be named “run_ensemble” in the resulting workflow. More information about defining a metatask is included :ref:`below `. @@ -105,21 +105,21 @@ Each task supports any of the tags that are defined in the Rocoto documentation. dependency: -The following sections are constructs of the interface, while all others are direct translations to tags available in Rocoto. Any tag that allows for attributes to the XML tag will take an attrs nested dictionary entry. +The following sections are constructs of the interface, while all others are direct translations to tags available in Rocoto. Any tag that allows for attributes to the XML tag will take an ``attrs`` nested dictionary entry. ``attrs``: Any of the attributes to the task tag in Rocoto. This is meant to be a subdictionary defining any of the Rocoto-supported attributes, where the key is the name of the attribute, and the value is what Rocoto expects. This might include any combination of the following: cycledefs, maxtries, throttle, or final. ``envars``: A dictionary of keys that map to variable names that will be exported for the job. These will show up as the set of ```` tags in the XML. The value will be the value of the defined variable when it is exported. -If the command entry is not provided, the task won’t show up in the resulting workflow. +If the ``command`` entry is not provided, the task won’t show up in the resulting workflow. Defining Dependencies ===================== The dependency entry will be an arbitrarily deep nested dictionary of key, value pairs. Each level represents entries that must come below it in priority. This is especially relevant for logic files. If an “and” tag must apply to multiple dependencies, those dependencies are all included as a nested dictionary of dependencies. -Because we are representing these entries as a dictionary, which requires hashable keys (no repeats at the same level), some tags may need to be differentiated where XML may not differentiate at all. In these instances, it’s best practice to name them something descriptive. For example, you might have multiple “or” dependencies at the same level that could be named “or_files_exist” and “or_task_ran”. This style can be adopted whether or not differentiation is needed. +Because we are representing these entries as a dictionary, which requires hashable keys (no repeats at the same level), some tags may need to be differentiated where XML may not differentiate at all. In these instances, it is best practice to name them something descriptive. For example, you might have multiple “or” dependencies at the same level that could be named “or_files_exist” and “or_task_ran”. This style can be adopted whether or not differentiation is needed. The text entry on some dependencies is for those dependency tags that need the information to come between two flags, as in a data dependency. @@ -177,7 +177,7 @@ The use of ``#mem#`` here is a Rocoto construct that identifies this task as a p Defining a Metatask =================== -A metatask groups together similar tasks and allows for the definition over entries defined by ``var`` tags. To define a metatask, the ``var`` entry with a nested dictionary of keys representing the names of the metatask variables and values indicating the list of values for each iteration, is required. +A metatask groups together similar tasks and allows for the definition over entries defined by ``var`` tags. To define a metatask, the ``var`` entry with a nested dictionary of keys representing the names of the metatask variables and values indicating the list of values for each iteration is required. Multiple var entries may be included, but each entry must have the same number of items. @@ -192,11 +192,11 @@ Here’s an example of a metatask section (without the task definition): mem: '{% if global.DO_ENSEMBLE %}{%- for m in range(1, global.NUM_ENS_MEMBERS+1) -%}{{ "%03d "%m }}{%- endfor -%} {% else %}{{ "000"|string }}{% endif %}' task_make_ics_mem#mem#: -This metatask will be named “run_ensemble” and will loop over all ensemble members, or just the deterministic member (“000”) if no ensemble of forecasts is meant to run. +This metatask will be named “run_ensemble” and will loop over all ensemble members or just the deterministic member (“000”) if no ensemble of forecasts is meant to run. The ``var`` section defines the metatask variables, here only “mem”. The name of the task represents that variable using ``#mem#`` to indicate that the resulting task name might be ``make_ics_mem000`` if only a deterministic forecast is configured to run. -When the task or the metatask is referenced in a dependency later on, do not include the ``task_`` or ```metatask_`` portions of the name. The reference to ``#mem#`` can be included if the dependency is included in a metatask that defines the variable, e.g., ``make_ics_mem#mem#``. Otherwise, you can reference a task that includes the value of the metatask var, e.g., ``make_ics_mem000``. More on this distinction is included in the Rocoto documentation. +When the task or the metatask is referenced in a dependency later on, do not include the ``task_`` or ``metatask_`` portions of the name. The reference to ``#mem#`` can be included if the dependency is included in a metatask that defines the variable, e.g., ``make_ics_mem#mem#``. Otherwise, you can reference a task that includes the value of the metatask var, e.g., ``make_ics_mem000``. More on this distinction is included in the Rocoto documentation. .. _J2filters: @@ -218,21 +218,21 @@ Order of Precedence =================== There is a specific order of precedence imposed when the SRW App loads configuration files. -#. Load config_defaults.yaml file. -#. Load the user’s config.yaml file. +#. Load ``config_defaults.yaml`` file. +#. Load the user’s ``config.yaml`` file. #. Load the workflow defaults YAML file. #. At this point, all anchors and references will be resolved. #. All PyYAML constructors will also be called for the data provided in that entry. -#. Call update_dict function to remove any null entries from default tasks using the PyYAML anchors. -#. Load all files from the ``taskgroups:`` entry from the user’s config, or default if not overridden. This is achieved with a call to the ``extend_yaml()`` function. +#. Call ``update_dict`` function to remove any null entries from default tasks using the PyYAML anchors. +#. Load all files from the ``taskgroups:`` entry from the user’s config or from the default if not overridden. This is achieved with a call to the ``extend_yaml()`` function. #. Add the contents of the files to the ``task:`` section. #. Update the existing workflow configuration with any user-specified entries (removing the ones that are null entries). #. Add a ``jobname:`` entry to every task in the workflow definition section. -#. Incorporate other default configuration settings from machine files, constants, etc into the default config dictionary in memory. +#. Incorporate other default configuration settings from machine files, constants, etc. into the default config dictionary in memory. #. Apply all user settings last to take highest precedence. #. Call ``extend_yaml()`` to render templates that are available. - NOTE: This is the one that is likely to trip up any settings that setup.py will make. References to other defaults that get changed during the course of validation may be rendered here earlier than desired. + NOTE: This is the one that is likely to trip up any settings that ``setup.py`` will make. References to other defaults that get changed during the course of validation may be rendered here earlier than desired. At this point, validation and updates for many other configuration settings will be made for a variety of sections. Once complete, ``extend_yaml()`` is called repeatedly, stopping only when all possible Jinja2-templated values have been rendered. From 708a4bbd70e0c4a125afe7ac322d9aca4a46f3ae Mon Sep 17 00:00:00 2001 From: Christina Holt Date: Fri, 12 May 2023 15:11:02 -0600 Subject: [PATCH 10/10] Address Gillian's comments. I _think_ this got them. --- docs/UsersGuide/source/Tutorial.rst | 24 +++++++++--------------- docs/UsersGuide/source/index.rst | 2 +- 2 files changed, 10 insertions(+), 16 deletions(-) diff --git a/docs/UsersGuide/source/Tutorial.rst b/docs/UsersGuide/source/Tutorial.rst index bf3711ca87..dafcae7025 100644 --- a/docs/UsersGuide/source/Tutorial.rst +++ b/docs/UsersGuide/source/Tutorial.rst @@ -148,24 +148,18 @@ In the ``workflow:`` section of ``config.yaml``, update ``EXPT_SUBDIR`` and ``PR For a detailed description of other ``workflow:`` variables, see :numref:`Section %s `. -In the ``workflow_switches:`` section, turn on the plotting task by setting ``RUN_TASK_PLOT_ALLVARS`` to true. All other variables should remain as they are. +To turn on the plotting for the experiment, the YAML configuration file +should be included in the ``rocoto:taskgroups:`` section, like this: .. code-block:: console - workflow_switches: - RUN_TASK_MAKE_GRID: true - RUN_TASK_MAKE_OROG: true - RUN_TASK_MAKE_SFC_CLIMO: true - RUN_TASK_GET_OBS_CCPA: false - RUN_TASK_GET_OBS_MRMS: false - RUN_TASK_GET_OBS_NDAS: false - RUN_TASK_VX_GRIDSTAT: false - RUN_TASK_VX_POINTSTAT: false - RUN_TASK_VX_ENSGRID: false - RUN_TASK_VX_ENSPOINT: false - RUN_TASK_PLOT_ALLVARS: true - -For a detailed description of the ``workflow-switches:`` variables, see :numref:`Section %s `. + rocoto: + tasks: + taskgroups: '{{ ["parm/wflow/prep.yaml", "parm/wflow/coldstart.yaml", "parm/wflow/post.yaml", "parm/wflow/plot.yaml"]|include }}' + + +For more information on how to turn on/off tasks in the worklfow, please +see :numref:`Section %s ` In the ``task_get_extrn_ics:`` section, add ``USE_USER_STAGED_EXTRN_FILES`` and ``EXTRN_MDL_SOURCE_BASEDIR_ICS``. Users will need to adjust the file path to reflect the location of data on their system (see :numref:`Section %s ` for locations on `Level 1 `__ systems). diff --git a/docs/UsersGuide/source/index.rst b/docs/UsersGuide/source/index.rst index 1346d0c025..1a51b34552 100644 --- a/docs/UsersGuide/source/index.rst +++ b/docs/UsersGuide/source/index.rst @@ -22,8 +22,8 @@ UFS Short-Range Weather App Users Guide InputOutputFiles LAMGrids ConfigWorkflow - DefineWorkflow RocotoInfo + DefineWorkflow WE2Etests TemplateVars VXCases