diff --git a/docs/UsersGuide/source/AQM.rst b/docs/UsersGuide/source/AQM.rst deleted file mode 100644 index 4c810f0d69..0000000000 --- a/docs/UsersGuide/source/AQM.rst +++ /dev/null @@ -1,259 +0,0 @@ -.. _AQM: - -===================================== -Air Quality Modeling (SRW-AQM) -===================================== - -The standard SRW App distribution uses the uncoupled version of the UFS Weather Model (atmosphere-only). However, users have the option to use a coupled version of the SRW App that includes the standard distribution (atmospheric model) plus the Air Quality Model (AQM). - -The AQM is a UFS Application that dynamically couples the Community Multiscale Air Quality (:term:`CMAQ`) model with the UFS Weather Model (WM) through the :term:`NUOPC` Layer to simulate temporal and spatial variations of atmospheric compositions (e.g., ozone and aerosol compositions). The CMAQ model, treated as a column chemistry model, updates concentrations of chemical species (e.g., ozone and aerosol compositions) at each integration time step. The transport terms (e.g., :term:`advection` and diffusion) of all chemical species are handled by the UFS WM as tracers. - -.. note:: - - Although this chapter is the primary documentation resource for running the AQM configuration, users may need to refer to :numref:`Chapter %s ` and :numref:`Chapter %s ` for additional information on building and running the SRW App, respectively. - -.. attention:: - - These instructions should work smoothly on Hera and WCOSS2, but users on other systems may need to make additional adjustments. - -Quick Start Guide (SRW-AQM) -===================================== - -Download the Code -------------------- - -Clone the ``develop`` branch of the authoritative SRW App repository: - -.. code-block:: console - - git clone -b develop https://github.com/ufs-community/ufs-srweather-app - cd ufs-srweather-app - -Checkout Externals ---------------------- - -Users must run the ``checkout_externals`` script to collect (or "check out") the individual components of the SRW App (AQM version) from their respective GitHub repositories. - -.. code-block:: console - - ./manage_externals/checkout_externals - -Build the SRW App with AQM ------------------------------ - -On Hera and WCOSS2, users can build the SRW App AQM binaries with the following command: - -.. code-block:: console - - ./devbuild.sh -p= -a=ATMAQ - -where ```` is ``hera``, or ``wcoss2``. The ``-a`` argument indicates the configuration/version of the application to build. - -Building the SRW App with AQM on other machines, including other `Level 1 `__ platforms, is not currently guaranteed to work, and users may have to make adjustments to the modulefiles for their system. - -Load the ``workflow_tools`` Environment --------------------------------------------- - -Load the python environment for the workflow: - -.. code-block:: console - - # On WCOSS2 (do not run on other systems): - source ../versions/run.ver.wcoss2 - # On all systems (including WCOSS2): - module use /path/to/ufs-srweather-app/modulefiles - module load wflow_ - -where ```` is ``hera`` or ``wcoss2``. The workflow should load on other platforms listed under the ``MACHINE`` variable in :numref:`Section %s `, but users may need to adjust other elements of the process when running on those platforms. - -If the console outputs a message, the user should run the commands specified in the message. For example, if the output says: - -.. code-block:: console - - Please do the following to activate conda: - > conda activate workflow_tools - -then the user should run ``conda activate workflow_tools``. Otherwise, the user can continue with configuring the workflow. - -.. _AQMConfig: - -Configure and Experiment ---------------------------- - -Users will need to configure their experiment by setting parameters in the ``config.yaml`` file. To start, users can copy a default experiment setting into ``config.yaml``: - -.. code-block:: console - - cd ush - cp config.aqm.community.yaml config.yaml - -Users may prefer to copy the ``config.aqm.nco.realtime.yaml`` for a default "nco" mode experiment instead. - -Users will need to change the ``MACHINE`` and ``ACCOUNT`` variables in ``config.yaml`` to match their system. They may also wish to adjust other experiment settings. For more information on each task and variable, see :numref:`Chapter %s `. - -Users may also wish to change :term:`cron`-related parameters in ``config.yaml``. In the ``config.aqm.community.yaml`` file, which was copied into ``config.yaml``, cron is used for automatic submission and resubmission of the workflow: - -.. code-block:: console - - workflow: - USE_CRON_TO_RELAUNCH: true - CRON_RELAUNCH_INTVL_MNTS: 3 - -This means that cron will submit the launch script every 3 minutes. Users may choose not to submit using cron or to submit at a different frequency. Note that users should create a crontab by running ``crontab -e`` the first time they use cron. - -Generate the Workflow ------------------------- - -Generate the workflow: - -.. code-block:: console - - ./generate_FV3LAM_wflow.py - -Run the Workflow ------------------- - -If ``USE_CRON_TO_RELAUNCH`` is set to true in ``config.yaml`` (see :numref:`Section %s `), the workflow will run automatically. If it was set to false, users must submit the workflow manually from the experiment directory: - -.. code-block:: console - - cd / - ./launch_FV3LAM_wflow.sh - -Repeat the launch command regularly until a SUCCESS or FAILURE message appears on the terminal window. See :numref:`Section %s ` for more on the ```` and ```` variables. - -Users may check experiment status from the experiment directory with either of the following commands: - -.. code-block:: console - - # Check the experiment status (best for cron jobs) - rocotostat -w FV3LAM_wflow.xml -d FV3LAM_wflow.db -v 10 - - # Check the experiment status and relaunch the workflow (for manual jobs) - ./launch_FV3LAM_wflow.sh; tail -n 40 log.launch_FV3LAM_wflow - - -WE2E Test for AQM -======================= - -Build the app for AQM: - -.. code-block:: console - - ./devbuild.sh -p=hera -a=ATMAQ - - -Add the WE2E test for AQM to the list file: - -.. code-block:: console - - echo "custom_ESGgrid" > my_tests.txt - echo "aqm_grid_AQM_NA13km_suite_GFS_v16" >> my_tests.txt - - -Run the WE2E test: - -.. code-block:: console - - $ ./run_WE2E_tests.py -t my_tests.txt -m hera -a gsd-fv3 -q - - - -Additional Tasks for AQM -=============================== - -Structure of SRW-AQM -------------------------- - -The flowchart of the non-DA (data assimilation) SRW-AQM (Air Quality Modeling) is illustrated in :numref:`Figure %s `. Compared to the non-coupled (ATM stand-alone) FV3-LAM, SRW-AQM has additional tasks for pre- and post-processing. For pre-processing, multiple emission data such as NEXUS, fire, and point-source emission are retrieved or created for air quality modeling. Moreover, the chemical initial conditions (ICs) are extracted from the restart files of the previous cycle and added to the existing IC files. The chemical lateral boundary conditions (LBCs) and the GEFS aerosol data are also adeded to the existing LBC files. For post-processing, air quality forecast products for O3 and PM2.5 are generated and the bias-correction technique is applied to improve the accuracy of the results. - -.. _FlowProcAQM: - -.. figure:: https://github.com/ufs-community/ufs-srweather-app/wiki/SRW-AQM_workflow.png - :alt: Flowchart of the SRW-AQM tasks. - - *Workflow structure of SRW-AQM (non-DA)* - - - -Pre-processing Tasks of SRW-AQM ------------------------------------- - -The pre-processing tasks for air quality modeling (AQM) are shown in :numref:`Table %s `. - -.. _TasksPrepAQM: - -.. table:: Tasks for pre-processing of AQM - - +-----------------------+--------------------------------------------------------------------+ - | **Task name** | **Description** | - +=======================+====================================================================+ - | nexus_gfs_sfc | This task retrieves the GFS surface files from the previous cycle | - | | in NRT (Near-Real-Time) or current cycle in retrospective cases. | - | | The surface radiation, soil moisture and temperature fields are | - | | needed for the MEGAN biogenics emissions within nexus_emission. | - +-----------------------+--------------------------------------------------------------------+ - | nexus_emission | This task prepares the run directory with gridded emission inputs, | - | | run nexus to create model ready emission for the given simulation | - | | day, and post processes nexus output to make it more readable. The | - | | task will also split the task into multiple jobs set by the user. | - +-----------------------+--------------------------------------------------------------------+ - | nexus_post_split | This task combines the nexus_emission outputs into a single job. | - +-----------------------+--------------------------------------------------------------------+ - | fire_emission | This tasks is used to convert both satellite-retrieved gas and | - | | aerosol species emissions (RAVE) from mass (kg) to emission rates | - | | (kg/m2/s) and create 3-day hourly model-ready fire emission input | - | | files. | - +-----------------------+--------------------------------------------------------------------+ - | point_source | This task aggregates the anthropogenic point source sectors of the | - | | National Emission Inventory(NEI) into a ready-to-input point-source| - | | emission file based on the weekday/weekend/holiday patterns of each| - | | sector and date/time of the simulation. | - +-----------------------+--------------------------------------------------------------------+ - | aqm_ics | This task creates a chemical initial condition file by using the | - | | previous cycle restart files. | - +-----------------------+--------------------------------------------------------------------+ - | aqm_lbcs | This task adds the chemical lateral boundary condition (LBC) upon | - | | the meteorological lateral boundary condition to form the full-set | - | | ready-to-input LBC for the simulation. It includes two sub-tasks: | - | | the gaseous species LBC and dynamic aerosol LBC. The former adds | - | | static gaseous LBC using monthly mean global data. The latter is | - | | the parallel job, which extracts the GEFS-Aerosol Model's output | - | | along the regional domain, and performs the species conversion | - | | from GOCART aerosols to CMAQ aerosols. | - +-----------------------+--------------------------------------------------------------------+ - - -Post-processing Tasks of SRW-AQM ------------------------------------- - -The post-processing tasks for air quality modeling (AQM) are shown in :numref:`Table %s `. Since the module required to run these tasks is available only on WCOSS2, these tasks should not be defined in the configuration file ``config.yaml`` on other platforms. - -.. _TasksPostAQM: - -.. table:: Tasks for post-processing of AQM - - +-----------------------+--------------------------------------------------------------------+ - | **Task name** | **Description** | - +=======================+====================================================================+ - | pre_post_stat | This task creates surface (i.e., model 1st level) meteorological | - | | and chemical files to support air quality product generation and | - | | generate training data to support bias correction tasks. | - +-----------------------+--------------------------------------------------------------------+ - | post_stat_o3 | This task generates air quality forecast products including hourly | - | | -average and statistical products for O3 (e.g., daily 8-hour | - | | average maximum O3). | - +-----------------------+--------------------------------------------------------------------+ - | post_stat_pm25 | This task generates air quality forecast products including hourly | - | | -average and statistical products for PM2.5 (e.g., 24-hour average | - | | PM2.5). | - +-----------------------+--------------------------------------------------------------------+ - | bias_correction_o3 | This task applies a bias-correction technique (e.g., analog | - | | ensemble) to improve model raw forecast for O3 and generates the | - | | bias-corrected O3 products. | - +-----------------------+--------------------------------------------------------------------+ - | bias_correction_pm25 | This task applies a bias-correction technique (e.g., analog | - | | ensemble) to improve model raw forecast for PM2.5 and generates the| - | | bias-corrected PM2.5 products. | - +-----------------------+--------------------------------------------------------------------+ - diff --git a/docs/UsersGuide/source/BackgroundInfo/Components.rst b/docs/UsersGuide/source/BackgroundInfo/Components.rst new file mode 100644 index 0000000000..fd362ec691 --- /dev/null +++ b/docs/UsersGuide/source/BackgroundInfo/Components.rst @@ -0,0 +1,116 @@ +.. _Components: + +============================ +SRW Application Components +============================ + +The SRW Application assembles a variety of components, including: + +* Pre-processor Utilities & Initial Conditions +* UFS Weather Forecast Model +* Unified Post Processor +* METplus Verification Suite +* Unified Workflow Tools +* Build System and Workflow + +These components are documented within this User's Guide and supported through the `GitHub Discussions `__ forum. + +.. _Utils: + +UFS Preprocessing Utilities (UFS_UTILS) +========================================== + +The SRW Application includes a number of pre-processing utilities (UFS_UTILS) that initialize and prepare the model. Since the SRW App provides forecast predictions over a limited area (rather than globally), these utilities generate a regional grid (``regional_esg_grid/make_hgrid``) along with :term:`orography` (``orog``) and surface climatology (``sfc_climo_gen``) files on that grid. Grids include a strip, or "halo," of six cells that surround the regional grid and feed in lateral boundary condition data. Since different grid and orography files require different numbers of :term:`halo` cells, additional utilities handle topography filtering and shave the number of halo points (based on downstream workflow component requirements). The pre-processing software :term:`chgres_cube` is used to convert the raw external model data into initial and lateral boundary condition files in :term:`netCDF` format. These are needed as input to the :term:`FV3`-:term:`LAM`. Additional information about the UFS pre-processor utilities can be found in the `UFS_UTILS Technical Documentation `__ and in the `UFS_UTILS Scientific Documentation `__. + +The SRW Application can be initialized from a range of operational initial condition files. It is possible to initialize the model from the Global Forecast System (:term:`GFS`), North American Mesoscale (:term:`NAM`) Forecast System, Rapid Refresh (:term:`RAP`), and High-Resolution Rapid Refresh (:term:`HRRR`) files in Gridded Binary v2 (:term:`GRIB2`) format. GFS files also come in :term:`NEMSIO` format for past dates. + +.. WARNING:: + For GFS data, dates prior to 1 January 2018 may work but are not guaranteed. Public archives of model data can be accessed through the `NOAA Operational Model Archive and Distribution System `__ (NOMADS). Raw external model data may be pre-staged on disk by the user. + + +Forecast Model +============== + +The prognostic atmospheric model in the UFS SRW Application is the Finite-Volume Cubed-Sphere (:term:`FV3`) dynamical core configured with a Limited Area Model (:term:`LAM`) capability (:cite:t:`BlackEtAl2021`). The :term:`dynamical core` is the computational part of a model that solves the equations of fluid motion. A User's Guide for the UFS :term:`Weather Model` can be accessed `here `__. + +Supported model resolutions in this release include 3-, 13-, and 25-km predefined contiguous U.S. (:term:`CONUS`) domains, each with 127 vertical levels. Preliminary tools for users to define their own domain are also available in the release with full, formal support of these tools to be provided in future releases. The Extended Schmidt Gnomonic (ESG) grid is used with the FV3-LAM, which features relatively uniform grid cells across the entirety of the domain. Additional information about the FV3 dynamical core can be found in the `scientific documentation `__, the `technical documentation `__, and on the `NOAA Geophysical Fluid Dynamics Laboratory website `__. + +Model Physics +--------------- + +The Common Community Physics Package (CCPP), described `here `__, supports interoperable atmospheric physics and land surface model options. Atmospheric physics are a set of numerical methods describing small-scale processes such as clouds, turbulence, radiation, and their interactions. The most recent SRW App release (v2.1.0) included four supported physics suites, and a fifth has since been added: FV3_RRFS_v1beta, FV3_GFS_v16, FV3_WoFS_v0, FV3_HRRR, and FV3_RAP (new!). The FV3_RRFS_v1beta physics suite is being tested for use in the future operational implementation of the Rapid Refresh Forecast System (:term:`RRFS`) planned for 2023-2024, and the FV3_GFS_v16 is an updated version of the physics suite used in the operational Global Forecast System (GFS) v16. A detailed list of CCPP updates since the SRW App v2.0.0 release is available :ref:`here `. A full scientific description of CCPP parameterizations and suites can be found in the `CCPP Scientific Documentation `__, and CCPP technical aspects are described in the `CCPP Technical Documentation `__. The model namelist has many settings beyond the physics options that can optimize various aspects of the model for use with each of the supported suites. Additional information on Stochastic Physics options is available `here `__. Additionally, a CCPP single-column model (`CCPP-SCM `__) option has also been developed as a child repository. Users can refer to the `CCPP Single Column Model User and Technical Guide `__ for more details. This CCPP-SCM user guide contains a Quick Start Guide with instructions for obtaining the code, compiling, and running test cases, which include five standard test cases and two additional FV3 replay cases (refer to section 5.2 in the CCPP-SCM user guide for more details). Moreover, the CCPP-SCM supports a precompiled version in a docker container, allowing it to be easily executed on NOAA's cloud computing platforms without any issues (see section 2.5 in the CCPP-SCM user guide for more details). + +.. note:: + SPP is currently only available for specific physics schemes used in the RAP/HRRR physics suite. Users need to be aware of which physics suite definition file (:term:`SDF`) is chosen when turning this option on. Among the supported physics suites, the full set of parameterizations can only be used with the ``FV3_HRRR`` option for ``CCPP_PHYS_SUITE``. + +The SRW App supports the use of both :term:`GRIB2` and :term:`NEMSIO` input data. The UFS Weather Model ingests initial and lateral boundary condition files produced by :term:`chgres_cube` and outputs files in netCDF format on a specific projection (e.g., Lambert Conformal) in the horizontal direction and model levels in the vertical direction. + +Unified Post Processor (UPP) +============================== + +The Unified Post Processor (:term:`UPP`) processes raw output from a variety of numerical weather prediction (:term:`NWP`) models. In the SRW App, the UPP converts model output data from the model's native :term:`netCDF` format to :term:`GRIB2` format on standard isobaric vertical coordinates. The UPP can also be used to compute a variety of useful diagnostic fields, as described in the `UPP User's Guide `__. Output from UPP can be used with visualization, plotting, and verification packages or in further downstream post-processing (e.g., statistical post-processing techniques). + +.. _MetplusComponent: + +METplus Verification Suite +============================= + +The Model Evaluation Tools (MET) package is a set of statistical verification tools developed by the `Developmental Testbed Center `__ (DTC) for use by the :term:`NWP` community to help them assess and evaluate the performance of numerical weather predictions. MET is the core component of the enhanced `METplus `__ verification framework; the suite also includes the associated database and display systems called METviewer and METexpress. + +The METplus verification framework has been integrated into the SRW App to facilitate forecast evaluation. METplus is a verification framework that spans a wide range of temporal scales (warn-on-forecast to climate) and spatial scales (storm to global). It is supported by the `Developmental Testbed Center (DTC) `__. + +METplus *installation* is not included as part of the build process for the most recent release of the SRW App. However, METplus is preinstalled on many `Level 1 & 2 `__ systems; existing builds can be viewed `here `__. + +METplus can be installed on other systems individually or as part of :term:`HPC-Stack` installation. Users on systems without a previous installation of METplus can follow the `MET Installation Guide `__ and `METplus Installation Guide `__ for individual installation. Currently, METplus *installation* is not a supported feature for this release of the SRW App. However, METplus *use* is supported on systems with a functioning METplus installation. + +The core components of the METplus framework include the statistical driver, MET, the associated database and display systems known as METviewer and METexpress, and a suite of Python wrappers to provide low-level automation and examples, also called use cases. MET is a set of verification tools developed for use by the :term:`NWP` community. It matches up grids with either gridded analyses or point observations and applies configurable methods to compute statistics and diagnostics. Extensive documentation is available in the `METplus User's Guide `__ and `MET User's Guide `__. Documentation for all other components of the framework can be found at the Documentation link for each component on the METplus `downloads `__ page. + +Among other techniques, MET provides the capability to compute standard verification scores for comparing deterministic gridded model data to point-based and gridded observations. It also provides ensemble and probabilistic verification methods for comparing gridded model data to point-based or gridded observations. Verification tasks to accomplish these comparisons are defined in the SRW App in :numref:`Table %s `. Currently, the SRW App supports the use of :term:`NDAS` observation files (which include conventional point-based surface and upper-air data) in `prepBUFR format `__ for point-based verification. It also supports gridded Climatology-Calibrated Precipitation Analysis (:term:`CCPA`) data for accumulated precipitation evaluation and Multi-Radar/Multi-Sensor (:term:`MRMS`) gridded analysis data for composite reflectivity and :term:`echo top` verification. + +METplus is being actively developed by :term:`NCAR`/Research Applications Laboratory (RAL), NOAA/Earth Systems Research Laboratories (ESRL), and NOAA/Environmental Modeling Center (:term:`EMC`), and it is open to community contributions. More details about METplus can be found in :numref:`Chapter %s ` and on the `METplus website `__. + +Air Quality Modeling (AQM) Utilities +======================================= + +AQM Utilities (AQM-utils) includes the utility executables and python scripts to run SRW-AQM (Online-CMAQ). +For more information on AQM-utils, visit the GitHub repository at https://github.com/NOAA-EMC/AQM-utils. + +.. _nexus: + +NOAA Emission and eXchange Unified System (NEXUS) +=================================================== + +The NOAA Emission and eXchange Unified System (NEXUS) is an emissions processing system developed at the NOAA Air Resources Laboratory (ARL) for use with regional and global UFS atmospheric composition models. NEXUS provides a streamlined process to include new emissions inventories quickly and can flexibly blend different emissions datasets. NEXUS incorporates the :term:`ESMF`-compliant Harmonized Emissions Component (`HEMCO `__), which "comput[es] emissions from a user-selected ensemble of emission inventories and algorithms" and "allows users to re-grid, combine, overwrite, subset, and scale emissions from different inventories through a configuration file and with no change to the model source code" (:cite:t:`LinEtAl2021`). + +For more information on NEXUS, visit the GitHub repository at https://github.com/noaa-oar-arl/NEXUS. + +.. _uwtools: + +Unified Workflow Tools +======================== + +The Unified Workflow (UW) is a set of tools intended to unify the workflow for various UFS applications under one framework. The UW toolkit currently includes templater and config tools, which have been incorporated into the SRW App workflow and will soon be incorporated into other UFS repositories. Additional tools are under development. More details about the UW can be found in the `workflow-tools `__ GitHub repository and in the `UW Documentation `__. + +Build System and Workflow +========================= + +The SRW Application has a portable, CMake-based build system that packages together all the components required to build the SRW Application. This build system collects the components necessary for running the end-to-end SRW Application, including the UFS Weather Model and the pre- and post-processing software. Additional libraries necessary for the application (e.g., :term:`NCEPLIBS-external` and :term:`NCEPLIBS`) are not included in the SRW Application build system but are available pre-built on pre-configured platforms. On other systems, they can be installed via the HPC-Stack (see :doc:`HPC-Stack Documentation `). There is a small set of system libraries and utilities that are assumed to be present on the target computer: the CMake build software; a Fortran, C, and C++ compiler; and an :term:`MPI` library. + +Once built, users can generate a Rocoto-based workflow that will run each task in the proper sequence (see :numref:`Chapter %s ` or the `Rocoto documentation `__ for more information on Rocoto and workflow management). If Rocoto and/or a batch system is not present on the available platform, the individual components can be run in a stand-alone, command line fashion with provided run scripts. + +The SRW Application allows users to configure various elements of the workflow. For example, users can modify the parameters of the atmospheric model, such as start and end dates, duration, integration time step, and the physics suite used for the simulation. It also allows for configuration of other elements of the workflow; for example, users can choose whether to run some or all of the pre-processing, forecast model, and post-processing steps. More information on how to configure the workflow is available in :numref:`Section %s `. + +An optional Python plotting task is also included to create basic visualizations of the model output. The task outputs graphics in PNG format for several standard meteorological variables on the pre-defined :term:`CONUS` domain. A difference plotting option is also included to visually compare two runs for the same domain and resolution. These plots may be used to perform a visual check to verify that the application is producing reasonable results. Configuration instructions are provided in :numref:`Section %s `. + +The SRW Application has been tested on a variety of platforms widely used by researchers, including NOAA High-Performance Computing (HPC) systems (e.g., Hera, Orion, Jet); the National Center for Atmospheric Research (:term:`NCAR`) Cheyenne system; cloud environment, and generic Linux and MacOS systems using Intel and GNU compilers. Four `levels of support `__ have been defined for the SRW Application, including pre-configured (Level 1), configurable (Level 2), limited-test (Level 3), and build-only (Level 4) platforms. + +Preconfigured (Level 1) systems already have the required external libraries available in a central location (via :term:`HPC-Stack` or :term:`spack-stack`). The SRW Application is expected to build and run out-of-the-box on these systems, and users can :ref:`download the SRW App code ` without first installing prerequisites. + +Configurable platforms (Level 2) are platforms where all of the required libraries for building the SRW Application are expected to install successfully but are not available in a central location. Users will need to install the required libraries as part of the :ref:`SRW Application build ` process. Applications and models are expected to build and run once the required libraries are built. Release testing is conducted on these systems to ensure that the SRW App runs smoothly there. + +Limited-Test (Level 3) and Build-Only (Level 4) computational platforms are those in which the developers have built the code but little or no pre-release testing has been conducted, respectively. Users may need to perform additional troubleshooting on Level 3 or 4 systems since little or no pre-release testing has been conducted on these systems. + +On all platforms, the SRW App can be :ref:`run within a container ` that includes the prerequisite software. + +A complete description of the levels of support, along with a list of preconfigured and configurable platforms can be found in the `SRW Application Wiki `__. + diff --git a/docs/UsersGuide/source/BackgroundInfo/Introduction.rst b/docs/UsersGuide/source/BackgroundInfo/Introduction.rst new file mode 100644 index 0000000000..f4aa344fa7 --- /dev/null +++ b/docs/UsersGuide/source/BackgroundInfo/Introduction.rst @@ -0,0 +1,181 @@ +.. _Introduction: + +============== +Introduction +============== + +The Unified Forecast System (:term:`UFS`) is a community-based, coupled, comprehensive Earth modeling system. NOAA's operational model suite for numerical weather prediction (:term:`NWP`) is quickly transitioning to the UFS from a number of different modeling systems. The UFS enables research, development, and contribution opportunities within the broader :term:`Weather Enterprise` (including government, industry, and academia). For more information about the UFS, visit the `UFS Portal `__. + +The UFS includes `multiple applications `__ that support different forecast durations and spatial domains. This documentation describes the UFS Short-Range Weather (SRW) Application, which targets predictions of atmospheric behavior on a limited spatial domain and on time scales from minutes to several days. The most recent SRW Application includes a prognostic atmospheric model, pre- and post-processing, and a community workflow for running the system end-to-end. These components are documented within this User's Guide and supported through the `GitHub Discussions `__ forum. The SRW App also includes support for a verification package (METplus) for both deterministic and ensemble simulations and support for four stochastically perturbed physics schemes. + +Since the v2.1.0 release, developers have added a variety of features: + + * Bug fixes since the v2.1.0 release + * Pre-implementation Rapid Refresh Forecast System (RRFS) forecast configurations + * Air Quality Modeling (AQM) capabilities + * Updates to :term:`CCPP` that target the top of the ``main`` branch (which is ahead of CCPP v6.0.0). See :ref:`this page ` for a detailed summary of updates that came in ahead of the v2.1.0 release. + * Support for the :term:`UPP` inline post option (see :ref:`here `) + * Documentation updates to reflect the changes above + +The SRW App v2.1.0 citation is as follows and should be used when presenting results based on research conducted with the App: + +UFS Development Team. (2022, Nov. 17). Unified Forecast System (UFS) Short-Range Weather (SRW) Application (Version v2.1.0). Zenodo. https://doi.org/10.5281/zenodo.7277602 + +User's Guide Organization +============================ + +The SRW Application documentation is organized into four sections: *Background Information*; *Building, Running, and Testing the SRW App*; *Customizing the Workflow*; and *Reference*. + +Background Information +------------------------- + + * This **Introduction** section explains how the SRW App documentation is organized, how to use this guide, and where to find user support and component documentation. + * :numref:`Section %s: Technical Overview ` provides a technical overview, including SRW App prerequisites and an overview of the code directory structure. + * :numref:`Section %s: SRW Application Components ` provides a detailed description of the application components, including optional application components. + +Building, Running, and Testing the SRW App +-------------------------------------------- + + * :numref:`Section %s: Quick Start Guide ` is an overview of the workflow and gives instructions for its use on `Level 1 platforms `__. + * :numref:`Section %s: Container-Based Quick Start Guide ` explains how to run the SRW Application in a container. Containers may be run on a broad range of systems and come with SRW App prerequisites already installed. + * :numref:`Section %s: Building the SRW App ` provides a *detailed* explanation of how to build the SRW App. + * :numref:`Section %s: Running the SRW App ` provides a *detailed* explanation of how to run the SRW App after it has been built/compiled. It includes information on standard workflow tasks, additional optional tasks (e.g., METplus verification, plotting), and different techniques for running the workflow. + * :numref:`Section %s: Testing the SRW App ` explains how to run workflow end-to-end (WE2E) tests to ensure that new developments do not break the current workflow. + * :numref:`Section %s: Tutorials ` walks users through different SRW App experiment cases and analysis of results. + * :numref:`Section %s: METplus Verification Sample Cases ` explains how to run METplus verification as part of the workflow. + * :numref:`Section %s: Air Quality Modeling ` provides information specific to air quality modeling (AQM). This feature is currently unsupported, so documentation may be behind the current state of development, which is progressing rapidly. However, this section is a starting point for those interested in AQM. + +.. hint:: + * To get started with the SRW App, it is recommended that users try one of the following options: + + #. View :numref:`Section %s: Quick Start Guide ` for a quick overview of the workflow steps. Especially helpful for users with access to a `Level 1 platform `__. + #. To build the application in a container, which provides a more uniform work environment, users can refer to :numref:`Section %s: Container-Based Quick Start Guide `. + #. For detailed instructions on building and running the SRW App, users can refer to :numref:`Section %s: Building the SRW App ` and :numref:`Section %s: Running the SRW App `. + +Customizing the Workflow +--------------------------- + + * :numref:`Section %s: Workflow Parameters ` documents all of the user-configurable experiment parameters that can be set in the user configuration file (``config.yaml``). + * :numref:`Section %s: Input & Output Files ` describes application input and output files, as well as information on where to get publicly available data. + * :numref:`Section %s: Limited Area Model (LAM) Grids ` describes the SRW App predefined grids in detail and explains how to create a custom user-generated grid. + * :numref:`Section %s: Defining an SRW App Workflow ` explains how to build a customized SRW App workflow XML file. + * :numref:`Section %s: Template Variables ` explains how to use template variables. + +Reference Information +----------------------- + + * :numref:`Section %s: Rocoto Introductory Information ` provides an introduction to standard Rocoto commands with examples. + * :numref:`Section %s: FAQ ` answers users' frequently asked questions. + * :numref:`Section %s: Glossary ` defines important terms related to the SRW App. + + +SRW App Documentation Conventions +=================================== + +This guide uses particular conventions to indicate commands and code snippets, file and directory paths, variables, and options. + +.. code-block:: console + + Throughout the guide, this presentation style indicates shell commands, code snippets, etc. + +Text rendered as ``AaBbCc123`` typically refers to variables in scripts, names of files, or directories. + +Code that includes angle brackets (e.g., ``build__``) indicates that users should insert options appropriate to their SRW App configuration (e.g., ``build_hera_intel``). + +File or directory paths that begin with ``/path/to/`` should be replaced with the actual path on the user's system. For example, ``/path/to/modulefiles`` might be replaced by ``/Users/Jane.Smith/ufs-srweather-app/modulefiles``. + +Component Documentation +========================= + +A list of available component documentation is shown in :numref:`Table %s `. In general, technical documentation will explain how to use a particular component, whereas scientific documentation provides more in-depth information on the science involved in specific component files. + +.. _list_of_documentation: + +.. list-table:: Centralized list of documentation + :widths: 20 50 + :header-rows: 1 + + * - Documentation + - Location + * - HPC-Stack Documentation + - https://hpc-stack.readthedocs.io/en/latest/ + * - spack-stack Documentation + - https://spack-stack.readthedocs.io/en/latest/ + * - UFS_UTILS Technical Documentation + - https://noaa-emcufs-utils.readthedocs.io/en/latest + * - UFS_UTILS Scientific Documentation + - https://ufs-community.github.io/UFS_UTILS/index.html + * - UFS Weather Model User's Guide + - https://ufs-weather-model.readthedocs.io/en/latest + * - FV3 Technical Documentation + - https://noaa-emc.github.io/FV3_Dycore_ufs-v2.0.0/html/index.html + * - FV3 Scientific Documentation + - https://repository.library.noaa.gov/view/noaa/30725 + * - CCPP Technical Documentation + - https://ccpp-techdoc.readthedocs.io/en/latest/ + * - CCPP Scientific Documentation + - https://dtcenter.ucar.edu/GMTB/UFS_SRW_App_v2.2.0/sci_doc/index.html + * - Stochastic Physics Documentation + - https://stochastic-physics.readthedocs.io/en/latest/ + * - ESMF manual + - https://earthsystemmodeling.org/docs/release/latest/ESMF_usrdoc/ + * - Unified Post Processor User's Guide + - https://upp.readthedocs.io/en/latest/ + * - Unified Post Processor Scientific Documentation + - https://noaa-emc.github.io/UPP/ + * - Unified Workflow User's Guide + - https://unified-workflow.readthedocs.io/en/latest/ + * - METplus User's Guide + - https://metplus.readthedocs.io/en/latest/Users_Guide/index.html + * - HEMCO User's Guide (a component of the NEXUS AQM system) + - https://hemco.readthedocs.io/en/stable/ + +User Support and Contributions to Development +=============================================================== + +Questions +----------- + +The SRW App's `GitHub Discussions `__ forum provides online support for UFS users and developers to post questions and exchange information. When users encounter difficulties running the workflow, this is the place to post. Users can expect an initial response within two business days. + +When posting a question, it is recommended that users provide the following information: + +* The platform or system being used (e.g., Hera, Orion, MacOS, Linux) +* The version of the SRW Application being used (e.g., ``develop``, ``release/public-v2.1.0``). (To determine this, users can run ``git branch``, and the name of the branch with an asterisk ``*`` in front of it is the name of the branch they are working on.) Note that the version of the application being used and the version of the documentation being used should match, or users will run into difficulties. +* Stage of the application when the issue appeared (i.e., configuration, build/compilation, or name of a workflow task) +* Configuration file contents +* Full error message (preferably in text form rather than a screenshot) +* Current shell (e.g., bash, csh) and modules loaded +* Compiler + MPI combination being used + +Bug Reports +------------- + +If users (especially new users) believe they have identified a bug in the system, it is recommended that they first ask about the problem in `GitHub Discussions `__, since many "bugs" do not require a code change/fix --- instead, the user may be unfamiliar with the system and/or may have misunderstood some component of the system or the instructions, which is causing the problem. Asking for assistance in a `GitHub Discussion `__ post can help clarify whether there is a simple adjustment to fix the problem or whether there is a genuine bug in the code. Users are also encouraged to search `open issues `__ to see if their bug has already been identified. If there is a genuine bug, and there is no open issue to address it, users can report the bug by filing a `GitHub Issue `__. + +Feature Requests and Enhancements +----------------------------------- + +Users who want to request a feature enhancement or the addition of a new feature can file a `GitHub Issue `__ and add (or request that a code manager add) the ``EPIC Support Requested`` label. These feature requests will be forwarded to the Earth Prediction Innovation Center (`EPIC `__) management team for prioritization and eventual addition to the SRW App. + +Community Contributions +------------------------- + +The UFS community is encouraged to contribute to the development efforts of all related +utilities, model code, and infrastructure. As described above, users can post issues in the SRW App to report bugs or to announce upcoming contributions to the code base. Additionally, users can file issues in component repositories for contributions that directly concern those repositories. Contributions to the `ufs-srweather-app `__ repository should follow the guidelines contained in the `SRW App Contributor's Guide `__. For code to be accepted into a component repository, users must follow the code management rules of that component's authoritative repository. These rules are usually outlined in the User's Guide (see :numref:`Table %s `) or GitHub wiki for each respective repository (see :numref:`Table %s `). + +Future Direction +================= + +Users can expect to see incremental improvements and additional capabilities in upcoming releases of the SRW Application to enhance research opportunities and support operational forecast implementations. Planned enhancements include: + +* A more extensive set of supported developmental physics suites. +* A larger number of pre-defined domains/resolutions and a *fully supported* capability to create a user-defined domain. +* Add user-defined vertical levels (number and distribution). +* Inclusion of data assimilation and forecast restart/cycling capabilities. + + +.. bibliography:: ../references.bib + + + diff --git a/docs/UsersGuide/source/BackgroundInfo/TechnicalOverview.rst b/docs/UsersGuide/source/BackgroundInfo/TechnicalOverview.rst new file mode 100644 index 0000000000..a179d535e1 --- /dev/null +++ b/docs/UsersGuide/source/BackgroundInfo/TechnicalOverview.rst @@ -0,0 +1,332 @@ +.. _TechOverview: + +==================== +Technical Overview +==================== + +This chapter provides information on SRW App prerequistes, code repositories, and directory structure. + +.. _SRWPrerequisites: + +Prerequisites for Using the SRW Application +=============================================== + +Background Knowledge Prerequisites +-------------------------------------- + +The instructions in this documentation assume that users have certain background knowledge: + +* Familiarity with LINUX/UNIX systems +* Command line basics +* System configuration knowledge (e.g., compilers, environment variables, paths, etc.) +* Numerical Weather Prediction (e.g., concepts of parameterizations: physical, microphysical, convective) +* Meteorology (in particular, meteorology at the scales being predicted: 25km, 13km, and 3km resolutions) + +Additional background knowledge in the following areas could be helpful: + +* High-Performance Computing (HPC) Systems (for those running the SRW App on an HPC system) +* Programming (particularly Python and bash scripting) for those interested in contributing to the SRW App code +* Creating an SSH Tunnel to access HPC systems from the command line +* Containerization +* Workflow Managers/Rocoto + +.. _software-prereqs: + +Software/Operating System Requirements +----------------------------------------- +The UFS SRW Application has been designed so that any sufficiently up-to-date machine with a UNIX-based operating system should be capable of running the application. SRW App `Level 1 & 2 systems `__ already have these prerequisites installed. However, users working on other systems must ensure that the following requirements are installed on their system: + +**Minimum Platform Requirements:** + +* POSIX-compliant UNIX-style operating system + +* >82 GB disk space + + * 53 GB input data for a standard collection of global data, or "fix" file data (topography, climatology, observational data) for a short 12-hour test forecast on the :term:`CONUS` 25km domain. See data download instructions in :numref:`Section %s `. + * 8 GB for full :term:`HPC-Stack` installation + * 3 GB for ``ufs-srweather-app`` installation + * 1 GB for boundary conditions for a short 12-hour test forecast on the CONUS 25km domain. See data download instructions in :numref:`Section %s `. + * 17 GB for a 12-hour test forecast on the CONUS 25km domain, with model output saved hourly. + +* Fortran compiler released since 2018 + + * gfortran v9+ or ifort v18+ are the only ones tested, but others may work. + +* C and C++ compilers compatible with the Fortran compiler + + * gcc v9+, ifort v18+, and clang v9+ (macOS, native Apple clang, LLVM clang, GNU) have been tested + +* Python v3.6+, including prerequisite packages ``jinja2``, ``pyyaml``, and ``f90nml`` + + * Python packages ``scipy``, ``matplotlib``, ``pygrib``, ``cartopy``, and ``pillow`` are required for users who would like to use the provided graphics scripts. + +* Perl 5 + +* git v2.12+ + +* Lmod + +* wget + + * Only required for retrieving data using ``retrieve_data.py``. If data is prestaged, *wget* is not required. If data is retrieved using other means, *curl* may be used as an alternative. + +The following software is also required to run the SRW Application, but the :term:`HPC-Stack` (which contains the software libraries necessary for building and running the SRW App) can be configured to build these requirements: + +* CMake v3.20+ + +* :term:`MPI` (MPICH, OpenMPI, or other implementation) + + * Only **MPICH** or **OpenMPI** can be built with HPC-Stack. Other implementations must be installed separately by the user (if desired). + +For MacOS systems, some additional software packages are needed. When possible, it is recommended that users install and/or upgrade this software (along with software listed above) using the `Homebrew `__ package manager for MacOS. See :doc:`HPC-Stack Documentation: Chapter 3 ` and :numref:`Chapter %s ` for further guidance on installing these prerequisites on MacOS. + +* bash v4.x +* GNU compiler suite v11 or higher with gfortran +* cmake +* make +* coreutils +* gsed + +Optional but recommended prerequisites for all systems: + +* Conda for installing/managing Python packages +* Bash v4+ +* Rocoto Workflow Management System (1.3.1) +* Python packages ``scipy``, ``matplotlib``, ``pygrib``, ``cartopy``, and ``pillow`` for graphics + +.. _SRWStructure: + +Code Repositories and Directory Structure +========================================= + +.. _HierarchicalRepoStr: + +Hierarchical Repository Structure +----------------------------------- +The :term:`umbrella repository` for the SRW Application is named ``ufs-srweather-app`` and is available on GitHub at https://github.com/ufs-community/ufs-srweather-app. The SRW Application uses the ``manage_externals`` tool and a configuration file called ``Externals.cfg``, to pull in the appropriate versions of the external repositories associated with the SRW App (see :numref:`Table %s `). + +.. _top_level_repos: + +.. list-table:: List of top-level repositories that comprise the UFS SRW Application + :widths: 20 40 + :header-rows: 1 + + * - Repository Description + - Authoritative repository URL + * - Umbrella repository for the UFS Short-Range Weather (SRW) Application + - https://github.com/ufs-community/ufs-srweather-app + * - Repository for the UFS Weather Model + - https://github.com/ufs-community/ufs-weather-model + * - Repository for UFS Utilities, including pre-processing, chgres_cube, and more + - https://github.com/ufs-community/UFS_UTILS + * - Repository for the Unified Post Processor (UPP) + - https://github.com/NOAA-EMC/UPP + * - Repository for Air Quality Modeling (AQM) Utilities + - https://github.com/NOAA-EMC/AQM-utils + * - Repository for NEXUS + - https://github.com/noaa-oar-arl/NEXUS + * - Repository for the Unified Workflow (UW) Toolkit + - https://github.com/ufs-community/workflow-tools + +The UFS Weather Model contains a number of sub-repositories, which are documented `here `__. + +.. note:: + The prerequisite libraries (including NCEP Libraries and external libraries) are not included in the UFS SRW Application repository. The `HPC-Stack `__ repository assembles these prerequisite libraries. The HPC-Stack has already been built on `preconfigured (Level 1) platforms `__. However, it must be built on other systems. See the :doc:`HPC-Stack Documentation ` for details on installing the HPC-Stack. + + +.. _TopLevelDirStructure: + +Directory Structure +---------------------- +The ``ufs-srweather-app`` :term:`umbrella repository` is an NCO-compliant repository. Its structure follows the standards laid out in :term:`NCEP` Central Operations (NCO) WCOSS `Implementation Standards `__. This structure is implemented using the ``local_path`` settings contained within the ``Externals.cfg`` file. After ``manage_externals/checkout_externals`` is run (see :numref:`Section %s `), the specific GitHub repositories described in :numref:`Table %s ` are cloned into the target subdirectories shown below. Directories that will be created as part of the build process appear in parentheses and will not be visible until after the build is complete. Some directories have been removed for brevity. + +.. code-block:: console + + ufs-srweather-app + ├── (build) + ├── docs + │ └── UsersGuide + ├── etc + ├── (exec) + ├── (include) + ├── jobs + ├── (lib) + ├── manage_externals + ├── modulefiles + │ ├── build__.lua + │ └── wflow_.lua + ├── parm + │ ├── wflow + │ └── FV3LAM_wflow.xml + ├── (share) + ├── scripts + ├── sorc + │ ├── CMakeLists.txt + │ ├── (UPP) + │ │ ├── parm + │ │ └── sorc + │ │ └── ncep_post.fd + │ ├── (UFS_UTILS) + │ │ ├── sorc + │ │ │ ├── chgres_cube.fd + │ │ │ ├── fre-nctools.fd + │ │ │ ├── grid_tools.fd + │ │ │ ├── orog_mask_tools.fd + │ │ │ └── sfc_climo_gen.fd + │ │ └── ush + │ └── (ufs-weather-model) + │ └── FV3 + │ ├── atmos_cubed_sphere + │ └── ccpp + ├── tests/WE2E + ├── ush + │ ├── bash_utils + │ ├── machine + │ ├── Python + │ ├── python_utils + │ ├── test_data + │ └── wrappers + └── versions + +SRW App SubDirectories +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +:numref:`Table %s ` describes the contents of the most important SRW App subdirectories. :numref:`Table %s ` provides a more comprehensive explanation of the ``ufs-srweather-app`` files and subdirectories. Users can reference the `NCO Implementation Standards `__ (p. 19) for additional details on repository structure in NCO-compliant repositories. + +.. _Subdirectories: + +.. table:: *Subdirectories of the ufs-srweather-app repository* + + +-------------------------+----------------------------------------------------+ + | **Directory Name** | **Description** | + +=========================+====================================================+ + | docs | Repository documentation | + +-------------------------+----------------------------------------------------+ + | jobs | :term:`J-job ` scripts launched by Rocoto | + +-------------------------+----------------------------------------------------+ + | modulefiles | Files used to load modules needed for building and | + | | running the workflow | + +-------------------------+----------------------------------------------------+ + | parm | Parameter files used to configure the model, | + | | physics, workflow, and various SRW App components | + +-------------------------+----------------------------------------------------+ + | scripts | Scripts launched by the J-jobs | + +-------------------------+----------------------------------------------------+ + | sorc | External source code used to build the SRW App | + +-------------------------+----------------------------------------------------+ + | tests | Tests for baseline experiment configurations | + +-------------------------+----------------------------------------------------+ + | ush | Utility scripts used by the workflow | + +-------------------------+----------------------------------------------------+ + +.. _ExperimentDirSection: + +Experiment Directory Structure +-------------------------------- +When the user generates an experiment using the ``generate_FV3LAM_wflow.py`` script (:numref:`Step %s `), a user-defined experiment directory (``$EXPTDIR``) is created based on information specified in the ``config.yaml`` file. :numref:`Table %s ` shows the contents of the experiment directory before running the experiment workflow. + +.. _ExptDirStructure: + +.. table:: Files and subdirectory initially created in the experiment directory + :widths: 33 67 + + +---------------------------+--------------------------------------------------------------------------------------------------------------+ + | **File Name** | **Description** | + +===========================+==============================================================================================================+ + | config.yaml | User-specified configuration file, see :numref:`Section %s ` | + +---------------------------+--------------------------------------------------------------------------------------------------------------+ + | data_table | :term:`Cycle-independent` input file (empty) | + +---------------------------+--------------------------------------------------------------------------------------------------------------+ + | field_table | :term:`Tracers ` in the `forecast model | + | | `__ | + +---------------------------+--------------------------------------------------------------------------------------------------------------+ + | FV3LAM_wflow.xml | Rocoto XML file to run the workflow | + +---------------------------+--------------------------------------------------------------------------------------------------------------+ + | input.nml | :term:`Namelist` for the `UFS Weather Model | + | | `__ | + +---------------------------+--------------------------------------------------------------------------------------------------------------+ + | launch_FV3LAM_wflow.sh | Symlink to the ``ufs-srweather-app/ush/launch_FV3LAM_wflow.sh`` shell script, | + | | which can be used to (re)launch the Rocoto workflow. | + | | Each time this script is called, it appends information to a log | + | | file named ``log.launch_FV3LAM_wflow``. | + +---------------------------+--------------------------------------------------------------------------------------------------------------+ + | log.generate_FV3LAM_wflow | Log of the output from the experiment generation script | + | | (``generate_FV3LAM_wflow.py``) | + +---------------------------+--------------------------------------------------------------------------------------------------------------+ + | nems.configure | See `NEMS configuration file | + | | `__ | + +---------------------------+--------------------------------------------------------------------------------------------------------------+ + | suite_{CCPP}.xml | :term:`CCPP` suite definition file (:term:`SDF`) used by the forecast model | + +---------------------------+--------------------------------------------------------------------------------------------------------------+ + | var_defns.sh | Shell script defining the experiment parameters. It contains all | + | | of the primary parameters specified in the default and | + | | user-specified configuration files plus many secondary parameters | + | | that are derived from the primary ones by the experiment | + | | generation script. This file is sourced by various other scripts | + | | in order to make all the experiment variables available to these | + | | scripts. | + +---------------------------+--------------------------------------------------------------------------------------------------------------+ + | YYYYMMDDHH | Cycle directory (empty) | + +---------------------------+--------------------------------------------------------------------------------------------------------------+ + +In addition, running the SRW App in *community* mode creates the ``fix_am`` and ``fix_lam`` directories (see :numref:`Table %s `) in ``$EXPTDIR``. The ``fix_lam`` directory is initially empty but will contain some *fix* (time-independent) files after the grid, orography, and/or surface climatology generation tasks run. + +.. _FixDirectories: + +.. table:: Description of the fix directories + + +-------------------------+----------------------------------------------------------+ + | **Directory Name** | **Description** | + +=========================+==========================================================+ + | fix_am | Directory containing the global fix (time-independent) | + | | data files. The experiment generation script symlinks | + | | these files from a machine-dependent system directory. | + +-------------------------+----------------------------------------------------------+ + | fix_lam | Directory containing the regional fix (time-independent) | + | | data files that describe the regional grid, orography, | + | | and various surface climatology fields, as well as | + | | symlinks to pre-generated files. | + +-------------------------+----------------------------------------------------------+ + +Once the Rocoto workflow is launched, several files and directories are generated. A log file named ``log.launch_FV3LAM_wflow`` will be created (unless it already exists) in ``$EXPTDIR``. The first several workflow tasks (i.e., ``make_grid``, ``make_orog``, ``make_sfc_climo``, ``get_extrn_ics``, and ``get_extrn_lbcs``) are preprocessing tasks, and these tasks also result in the creation of new files and subdirectories, described in :numref:`Table %s `. + +.. _CreatedByWorkflow: + +.. table:: New directories and files created when the workflow is launched + :widths: 30 70 + + +---------------------------+--------------------------------------------------------------------+ + | **Directory/File Name** | **Description** | + +===========================+====================================================================+ + | YYYYMMDDHH | This is a “cycle directory” that is updated when the first | + | | cycle-specific workflow tasks (``get_extrn_ics`` and | + | | ``get_extrn_lbcs``) are run. These tasks are launched | + | | simultaneously for each cycle in the experiment. Cycle directories | + | | are created to contain cycle-specific files for each cycle that | + | | the experiment runs. If ``DATE_FIRST_CYCL`` and ``DATE_LAST_CYCL`` | + | | are different in the ``config.yaml`` file, more than one cycle | + | | directory will be created under the experiment directory. | + +---------------------------+--------------------------------------------------------------------+ + | grid | Directory generated by the ``make_grid`` task to store grid files | + | | for the experiment | + +---------------------------+--------------------------------------------------------------------+ + | log | Contains log files generated by the overall workflow and by its | + | | various tasks. View the files in this directory to determine why | + | | a task may have failed. | + +---------------------------+--------------------------------------------------------------------+ + | orog | Directory generated by the ``make_orog`` task containing the | + | | orography files for the experiment | + +---------------------------+--------------------------------------------------------------------+ + | sfc_climo | Directory generated by the ``make_sfc_climo`` task containing the | + | | surface climatology files for the experiment | + +---------------------------+--------------------------------------------------------------------+ + | FV3LAM_wflow.db | Database files that are generated when Rocoto is called (by the | + | FV3LAM_wflow_lock.db | launch script) to launch the workflow | + +---------------------------+--------------------------------------------------------------------+ + | log.launch_FV3LAM_wflow | The ``launch_FV3LAM_wflow.sh`` script appends its output to this | + | | log file each time it is called. View the last several | + | | lines of this file to check the status of the workflow. | + +---------------------------+--------------------------------------------------------------------+ + +The output files for an experiment are described in :numref:`Section %s `. +The workflow tasks are described in :numref:`Section %s `. + diff --git a/docs/UsersGuide/source/BackgroundInfo/index.rst b/docs/UsersGuide/source/BackgroundInfo/index.rst new file mode 100644 index 0000000000..07e787bc00 --- /dev/null +++ b/docs/UsersGuide/source/BackgroundInfo/index.rst @@ -0,0 +1,10 @@ +Background Information +========================= + +.. toctree:: + :maxdepth: 3 + + + Introduction + TechnicalOverview + Components diff --git a/docs/UsersGuide/source/BuildingRunningTesting/AQM.rst b/docs/UsersGuide/source/BuildingRunningTesting/AQM.rst new file mode 100644 index 0000000000..4c83bcee2b --- /dev/null +++ b/docs/UsersGuide/source/BuildingRunningTesting/AQM.rst @@ -0,0 +1,319 @@ +.. _AQM: + +===================================== +Air Quality Modeling (SRW-AQM) +===================================== + +The standard SRW App distribution uses the uncoupled version of the UFS Weather Model (atmosphere-only). However, users have the option to use a coupled version of the SRW App that includes the standard distribution (atmospheric model) plus the Air Quality Model (AQM). + +The AQM is a UFS Application that dynamically couples the Community Multiscale Air Quality (:term:`CMAQ`) model with the UFS Weather Model (WM) through the :term:`NUOPC` Layer to simulate temporal and spatial variations of atmospheric compositions (e.g., ozone and aerosol compositions). The CMAQ model, treated as a column chemistry model, updates concentrations of chemical species (e.g., ozone and aerosol compositions) at each integration time step. The transport terms (e.g., :term:`advection` and diffusion) of all chemical species are handled by the UFS WM as tracers. + +.. note:: + + Although this chapter is the primary documentation resource for running the AQM configuration, users may need to refer to :numref:`Chapter %s ` and :numref:`Chapter %s ` for additional information on building and running the SRW App, respectively. + +.. attention:: + + These instructions should work smoothly on Hera and WCOSS2, but users on other systems may need to make additional adjustments. + +Quick Start Guide (SRW-AQM) +===================================== + +Download the Code +------------------- + +Clone the ``develop`` branch of the authoritative SRW App repository: + +.. code-block:: console + + git clone -b develop https://github.com/ufs-community/ufs-srweather-app + cd ufs-srweather-app + +Checkout Externals +--------------------- + +Users must run the ``checkout_externals`` script to collect (or "check out") the individual components of the SRW App (AQM version) from their respective GitHub repositories. + +.. code-block:: console + + ./manage_externals/checkout_externals + +Build the SRW App with AQM +----------------------------- + +On Hera and WCOSS2, users can build the SRW App AQM binaries with the following command: + +.. code-block:: console + + ./devbuild.sh -p= -a=ATMAQ + +where ```` is ``hera``, or ``wcoss2``. The ``-a`` argument indicates the configuration/version of the application to build. + +Building the SRW App with AQM on other machines, including other `Level 1 `__ platforms, is not currently guaranteed to work, and users may have to make adjustments to the modulefiles for their system. + +If the SRW-AQM builds correctly, users should see the standard executables listed in :numref:`Table %s `. Additionally, users should see the AQM-specific executables described in :numref:`Table %s ` in the ``ufs-srweather-app/exec`` directory. + +.. _AQM-exec: + +.. list-table:: *Names and descriptions of additional executables produced when the ATMAQ option is enabled* + :widths: 20 50 + :header-rows: 1 + + * - Executable + - Description + * - decomp-ptemis-mpi + - Splits the point-source emission file into subdomain based on runtime configure setting + * - gefs2lbc_para + - Interpolates :term:`GOCART` concentration to be lateral boundary condition for regional air quality model and outputs a layer result for checking purpose + * - nexus + - Runs the NOAA Emission and eXchange Unified System (:ref:`NEXUS `) emissions processing system + +Load the ``workflow_tools`` Environment +-------------------------------------------- + +Load the python environment for the workflow: + +.. code-block:: console + + # On WCOSS2 (do not run on other systems): + source ../versions/run.ver.wcoss2 + # On all systems (including WCOSS2): + module use /path/to/ufs-srweather-app/modulefiles + module load wflow_ + +where ```` is ``hera`` or ``wcoss2``. The workflow should load on other platforms listed under the ``MACHINE`` variable in :numref:`Section %s `, but users may need to adjust other elements of the process when running on those platforms. + +If the console outputs a message, the user should run the commands specified in the message. For example, if the output says: + +.. code-block:: console + + Please do the following to activate conda: + > conda activate workflow_tools + +then the user should run ``conda activate workflow_tools``. Otherwise, the user can continue with configuring the workflow. + +.. _AQMConfig: + +Configure and Experiment +--------------------------- + +Users will need to configure their experiment by setting parameters in the ``config.yaml`` file. To start, users can copy a default experiment setting into ``config.yaml``: + +.. code-block:: console + + cd ush + cp config.aqm.community.yaml config.yaml + +Users may prefer to copy the ``config.aqm.nco.realtime.yaml`` for a default "nco" mode experiment instead. + +Users will need to change the ``MACHINE`` and ``ACCOUNT`` variables in ``config.yaml`` to match their system. They may also wish to adjust other experiment settings. For more information on each task and variable, see :numref:`Section %s `. + +The community AQM configuration assumes that users have :term:`HPSS` access and attempts to download the data from HPSS. However, if users have the data on their system already, they may prefer to add the following lines to ``task_get_extrn_*:`` in their ``config.yaml`` file, adjusting the file path to point to the correct data locations: + +.. code-block:: console + + task_get_extrn_ics: + USE_USER_STAGED_EXTRN_FILES: true + EXTRN_MDL_SOURCE_BASEDIR_ICS: /path/to/data + task_get_extrn_lbcs: + USE_USER_STAGED_EXTRN_FILES: true + EXTRN_MDL_SOURCE_BASEDIR_LBCS: /path/to/data + +On Level 1 systems, users can find :term:`ICs/LBCs` in the usual :ref:`input data locations ` under ``FV3GFS/netcdf/2023021700`` and ``FV3GFS/netcdf/2023021706``. Users can also download the data required for the community experiment from the `UFS SRW App Data Bucket `__. + +Users may also wish to change :term:`cron`-related parameters in ``config.yaml``. In the ``config.aqm.community.yaml`` file, which was copied into ``config.yaml``, cron is used for automatic submission and resubmission of the workflow: + +.. code-block:: console + + workflow: + USE_CRON_TO_RELAUNCH: true + CRON_RELAUNCH_INTVL_MNTS: 3 + +This means that cron will submit the launch script every 3 minutes. Users may choose not to submit using cron or to submit at a different frequency. Note that users should create a crontab by running ``crontab -e`` the first time they use cron. + +When using the basic ``config.aqm.community.yaml`` experiment, the AQM pre-processing tasks are automatically turned because ``"parm/wflow/aqm_prep.yaml"`` appears in the list of workflow files in the ``rocoto: tasks: taskgroups:`` section of ``config.yaml`` (see :numref:`Section %s ` for task descriptions). To turn on AQM post-processing tasks in the workflow, include ``"parm/wflow/aqm_post.yaml"`` in the ``rocoto: tasks: taskgroups:`` section, too (see :numref:`Section %s ` for task descriptions). + +.. attention:: + + The module required to run the post-processing tasks is available only on WCOSS2. Therefore, ``aqm_post.yaml`` should not be added to the ``rocoto: tasks: taskgroups:`` section of ``config.yaml`` on any other platforms. + +Generate the Workflow +------------------------ + +Generate the workflow: + +.. code-block:: console + + ./generate_FV3LAM_wflow.py + +Run the Workflow +------------------ + +If ``USE_CRON_TO_RELAUNCH`` is set to true in ``config.yaml`` (see :numref:`Section %s `), the workflow will run automatically. If it was set to false, users must submit the workflow manually from the experiment directory: + +.. code-block:: console + + cd / + ./launch_FV3LAM_wflow.sh + +Repeat the launch command regularly until a SUCCESS or FAILURE message appears on the terminal window. See :numref:`Section %s ` for more on the ```` and ```` variables. + +Users may check experiment status from the experiment directory with either of the following commands: + +.. code-block:: console + + # Check the experiment status (for cron jobs) + rocotostat -w FV3LAM_wflow.xml -d FV3LAM_wflow.db -v 10 + + # Check the experiment status and relaunch the workflow (for manual jobs) + ./launch_FV3LAM_wflow.sh; tail -n 40 log.launch_FV3LAM_wflow + +To see a description of each of the AQM workflow tasks, see :numref:`Section %s `. + +.. _AQMSuccess: + +Experiment Output +-------------------- + +The workflow run is complete when all tasks display a "SUCCEEDED" message. If everything goes smoothly, users will eventually see a workflow status table similar to the following: + +.. code-block:: console + + CYCLE TASK JOBID STATE EXIT STATUS TRIES DURATION + ============================================================================================ + 202302170000 make_grid 47411619 SUCCEEDED 0 1 36.0 + 202302170000 make_orog 47411728 SUCCEEDED 0 1 151.0 + 202302170000 make_sfc_climo 47411801 SUCCEEDED 0 1 58.0 + 202302170000 nexus_gfs_sfc 47411620 SUCCEEDED 0 1 37.0 + 202302170000 nexus_emission_00 47411729 SUCCEEDED 0 1 251.0 + 202302170000 nexus_emission_01 47411730 SUCCEEDED 0 1 250.0 + 202302170000 nexus_emission_02 47411731 SUCCEEDED 0 1 250.0 + 202302170000 nexus_post_split 47412034 SUCCEEDED 0 1 44.0 + 202302170000 fire_emission 47411621 SUCCEEDED 0 1 19.0 + 202302170000 point_source 47411732 SUCCEEDED 0 1 82.0 + 202302170000 aqm_lbcs 47412961 SUCCEEDED 0 1 159.0 + 202302170000 get_extrn_ics 47411622 SUCCEEDED 0 1 314.0 + 202302170000 get_extrn_lbcs 47411623 SUCCEEDED 0 1 0.0 + 202302170000 make_ics_mem000 47659593 SUCCEEDED 0 1 126.0 + 202302170000 make_lbcs_mem000 47659594 SUCCEEDED 0 1 113.0 + 202302170000 run_fcst_mem000 47659742 SUCCEEDED 0 1 763.0 + 202302170000 run_post_mem000_f000 47659910 SUCCEEDED 0 1 30.0 + 202302170000 run_post_mem000_f001 47660029 SUCCEEDED 0 1 30.0 + 202302170000 run_post_mem000_f002 47660030 SUCCEEDED 0 1 31.0 + ... + 202302170000 run_post_mem000_f006 47660110 SUCCEEDED 0 1 29.0 + ============================================================================================ + 202302170600 nexus_gfs_sfc 47659421 SUCCEEDED 0 1 44.0 + 202302170600 nexus_emission_00 47659475 SUCCEEDED 0 1 323.0 + 202302170600 nexus_emission_01 47659476 SUCCEEDED 0 1 323.0 + 202302170600 nexus_emission_02 47659477 SUCCEEDED 0 1 329.0 + 202302170600 nexus_post_split 47659595 SUCCEEDED 0 1 60.0 + 202302170600 fire_emission 47659422 SUCCEEDED 0 1 18.0 + 202302170600 point_source 47659478 SUCCEEDED 0 1 128.0 + 202302170600 aqm_ics 47659597 SUCCEEDED 0 1 159.0 + 202302170600 aqm_lbcs 47659598 SUCCEEDED 0 1 158.0 + 202302170600 get_extrn_ics 47659423 SUCCEEDED 0 1 493.0 + 202302170600 get_extrn_lbcs 47659424 SUCCEEDED 0 1 536.0 + 202302170600 make_ics_mem000 47659594 SUCCEEDED 0 1 134.0 + 202302170600 make_lbcs_mem000 47659596 SUCCEEDED 0 1 112.0 + 202302170600 run_fcst_mem000 47659812 SUCCEEDED 0 1 1429.0 + 202302170600 run_post_mem000_f000 47659998 SUCCEEDED 0 1 30.0 + 202302170600 run_post_mem000_f001 47660042 SUCCEEDED 0 1 31.0 + 202302170600 run_post_mem000_f002 47660043 SUCCEEDED 0 1 29.0 + ... + 202302170600 run_post_mem000_f012 47660134 SUCCEEDED 0 1 30.0 + +.. _AQM-more-tasks: + +Additional Tasks for AQM +=============================== + +Structure of SRW-AQM Workflow +-------------------------------- + +:numref:`Figure %s ` illustrates the full non-:term:`DA ` SRW-AQM workflow using a flowchart. Compared to the uncoupled (atmosphere-only) workflow (see :numref:`Table %s `), SRW-AQM has additional tasks for pre- and post-processing. For pre-processing, multiple emissions data such as NEXUS, fire, and point-source emissions are retrieved or created for air quality modeling. Moreover, the chemical initial conditions (ICs) are extracted from the restart files of the previous cycle and added to the existing IC files. The chemical lateral boundary conditions (LBCs) and the GEFS aerosol data are also added to the existing LBC files. For post-processing, air quality forecast products for ozone (O3) and 2.5-micron particulate matter (PM2.5) are generated, and the bias-correction technique is applied to improve the accuracy of the results. + +.. _FlowProcAQM: + +.. figure:: https://github.com/ufs-community/ufs-srweather-app/wiki/SRW-AQM_workflow.png + :alt: Flowchart of the SRW-AQM tasks. + + *Workflow Structure of SRW-AQM (non-DA)* + + +Pre-processing Tasks of SRW-AQM +------------------------------------ + +The pre-processing tasks for air quality modeling (AQM) are shown in :numref:`Table %s `. They are defined in the ``parm/wflow/aqm_prep.yaml`` file. + +.. _TasksPrepAQM: + +.. list-table:: *Tasks for Pre-Processing of AQM* + :widths: 20 50 + :header-rows: 1 + + * - Task Name + - Description + * - nexus_gfs_sfc + - Retrieves the GFS surface files from the previous cycle in near real-time (NRT) or from the current cycle in retrospective cases. The surface radiation, soil moisture, and temperature fields are needed to predict the :term:`MEGAN` biogenics emissions within the ``nexus_emission_##`` task. + * - nexus_emission_## + - Prepares the run directory with gridded emissions inputs, runs the :ref:`NEXUS` to create model-ready emissions for the given simulation day, and post processes NEXUS output to make it more readable. The task will also split the task into ``##`` jobs set by the user in ``config.yaml`` using the ``NUM_SPLIT_NEXUS`` variable. + * - nexus_post_split + - Concatenates the NEXUS emissions information into a single netCDF file (needed for the forecast) if NEXUS was split into multiple jobs using the ``NUM_SPLIT_NEXUS`` variable. + * - fire_emission + - Converts both satellite-retrieved gas and aerosol species emissions (RAVE) from mass (kg) to emissions rates (kg/m2/s) and creates 3-day hourly model-ready fire emissions input files. + * - point_source + - Aggregates the anthropogenic point source sectors of the National Emission Inventory (NEI) into a ready-to-input point-source emission file based on the weekday/weekend/holiday patterns of each sector and the date/time of the simulation. + * - aqm_ics + - Creates a chemical initial conditions file by using the previous cycle restart files. + * - aqm_lbcs + - Adds the chemical lateral boundary conditions (LBCs) to the meteorological LBCs to form the full set of ready-to-input LBCs for the simulation. This task includes two sub-tasks: (1) addition of the gaseous species LBCs and (2) addition of dynamic aerosol LBCs. The former adds static gaseous LBCs using monthly mean global data. The latter is the parallel job, which extracts the GEFS-Aerosol Model's output along the regional domain and performs the species conversion from :term:`GOCART` aerosols to CMAQ aerosols. + +Post-processing Tasks of SRW-AQM +------------------------------------ + +The post-processing tasks for air quality modeling (AQM) are shown in :numref:`Table %s `. They are defined in the ``parm/wflow/aqm_post.yaml`` file. Since the module required to run these tasks is available only on WCOSS2, ``aqm_post.yaml`` should not be added to the ``rocoto: tasks: taskgroups:`` section of the configuration file ``config.yaml`` on other platforms. + +.. _TasksPostAQM: + +.. list-table:: Tasks for Post-processing of AQM + :widths: 20 50 + :header-rows: 1 + + * - Task name + - Description + * - pre_post_stat + - Creates surface (i.e., model first level) meteorological and chemical files to support air quality product generation and generate training data to support bias correction tasks. + * - post_stat_o3 + - Generates air quality forecast products, including hourly average and statistical products, for O3 (e.g., daily 8-hour average maximum O3). + * - post_stat_pm25 + - This task generates air quality forecast products, including hourly average and statistical products, for PM2.5 (e.g., 24-hour average PM2.5). + * - bias_correction_o3 + - Applies a bias-correction technique (e.g., analog ensemble) to improve the raw model forecast for O3 and generates the bias-corrected O3 products. + * - bias_correction_pm25 + - Applies a bias-correction technique (e.g., analog ensemble) to improve the raw model forecast for PM2.5 and generates the bias-corrected PM2.5 products. + +WE2E Test for AQM +======================= + +Build the app for AQM: + +.. code-block:: console + + ./devbuild.sh -p=hera -a=ATMAQ + + +Add the WE2E test for AQM to the list file: + +.. code-block:: console + + echo "custom_ESGgrid" > my_tests.txt + echo "aqm_grid_AQM_NA13km_suite_GFS_v16" >> my_tests.txt + + +Run the WE2E test: + +.. code-block:: console + + $ ./run_WE2E_tests.py -t my_tests.txt -m hera -a gsd-fv3 -q + diff --git a/docs/UsersGuide/source/BuildSRW.rst b/docs/UsersGuide/source/BuildingRunningTesting/BuildSRW.rst similarity index 63% rename from docs/UsersGuide/source/BuildSRW.rst rename to docs/UsersGuide/source/BuildingRunningTesting/BuildSRW.rst index e768372233..4ba8073df5 100644 --- a/docs/UsersGuide/source/BuildSRW.rst +++ b/docs/UsersGuide/source/BuildingRunningTesting/BuildSRW.rst @@ -4,11 +4,11 @@ Building the SRW App ========================== -The Unified Forecast System (:term:`UFS`) Short-Range Weather (SRW) Application is an :term:`umbrella repository` consisting of a number of different :ref:`components ` housed in external repositories. Once the SRW App is built, users can configure experiments and generate predictions of atmospheric behavior over a limited spatial area and on time scales ranging from minutes out to several days. +The Unified Forecast System (:term:`UFS`) Short-Range Weather (SRW) Application is an :term:`umbrella repository` consisting of a number of different :ref:`components ` housed in external repositories. Once the SRW App is built (i.e., components are assembled/compiled), users can configure experiments and generate predictions of atmospheric behavior over a limited spatial area and on time scales ranging from minutes out to several days. .. attention:: - The SRW Application has `four levels of support `__. The steps described in this chapter will work most smoothly on preconfigured (Level 1) systems. This chapter can also serve as a starting point for running the SRW App on other systems (including generic Linux/Mac systems), but the user may need to perform additional troubleshooting. + The SRW Application has `four levels of support `__. The steps described in this chapter will work most smoothly on preconfigured (Level 1) systems. This chapter also provides guidance for running the SRW App on other systems (including generic Linux/Mac systems), but the user may need to perform additional steps and/or troubleshooting. .. note:: The :ref:`container approach ` is recommended for a smoother first-time build and run experience. Building without a container may allow for more customization. However, the non-container approach requires more in-depth system-based knowledge, especially on Level 3 and 4 systems, so it is less appropriate for beginners. @@ -22,7 +22,7 @@ To build the SRW App, users will complete the following steps: .. _AppBuildProc: -.. figure:: _static/SRW_build_process.png +.. figure:: https://github.com/ufs-community/ufs-srweather-app/wiki/SRW_build_process.png :alt: Flowchart describing the SRW App build process. *Overview of the SRW App Build Process* @@ -33,7 +33,9 @@ To build the SRW App, users will complete the following steps: Install the Prerequisite Software Stack ========================================== -Currently, installation of the prerequisite software stack is supported via HPC-Stack. :term:`HPC-Stack` is a repository that provides a unified, shell script-based system to build the software stack required for `UFS `__ applications such as the SRW App. +Users on any sufficiently up-to-date machine with a UNIX-based operating system should be able to install the prerequisite software stack and run the SRW Application. However, a list of prerequisites is available in :numref:`Section %s ` for reference. Users should install or update their system as required before attempting to install the software stack. + +Currently, installation of the prerequisite software stack is supported via HPC-Stack. :term:`HPC-Stack` is a :term:`repository` that provides a unified, shell script-based system to build the software stack required for `UFS `__ applications such as the SRW App. .. Attention:: Skip the HPC-Stack installation if working on a `Level 1 system `__ (e.g., Cheyenne, Hera, Orion, NOAA Cloud), and :ref:`continue to the next section `. @@ -41,7 +43,7 @@ Currently, installation of the prerequisite software stack is supported via HPC- Background ---------------- -The UFS Weather Model draws on over 50 code libraries to run its applications. These libraries range from libraries developed in-house at NOAA (e.g., NCEPLIBS, FMS) to libraries developed by NOAA's partners (e.g., PIO, ESMF) to truly third party libraries (e.g., netCDF). Individual installation of these libraries is not practical, so the `HPC-Stack `__ was developed as a central installation system to ensure that the infrastructure environment across multiple platforms is as similar as possible. Installation of the HPC-Stack is required to run the SRW App. +SRW App components, including the UFS Weather Model (:term:`WM `), draw on over 50 code libraries to run. These libraries range from libraries developed in-house at NOAA (e.g., NCEPLIBS, FMS) to libraries developed by NOAA's partners (e.g., PIO, ESMF) to truly third party libraries (e.g., netCDF). Individual installation of these libraries is not practical, so the `HPC-Stack `__ was developed as a central installation system to ensure that the infrastructure environment across multiple platforms is as similar as possible. Installation of the HPC-Stack (or :term:`spack-stack`) is required to run the SRW App. Instructions ------------------------- @@ -58,9 +60,9 @@ Users working on systems that fall under `Support Levels 2-4 `. .. attention:: - Although HPC-Stack is the fully-supported option as of the v2.1.0 release, UFS applications are gradually shifting to :term:`spack-stack`, which is a :term:`Spack`-based method for installing UFS prerequisite software libraries. The spack-stack is currently used on NOAA Cloud platforms and in containers, while HPC-Stack is still used on other Level 1 systems and is the software stack validated by the UFS Weather Model as of the v2.1.0 release. Users are encouraged to check out `spack-stack `__ to prepare for the upcoming shift in support from HPC-Stack to spack-stack. + Although HPC-Stack is currently the fully-supported software stack option, UFS applications are gradually shifting to :term:`spack-stack`, which is a :term:`Spack`-based method for installing UFS prerequisite software libraries. The spack-stack is currently used on NOAA Cloud platforms and in containers, while HPC-Stack is still used on other Level 1 systems and is the software stack validated by the UFS Weather Model. Users are encouraged to check out `spack-stack `__ to prepare for the upcoming shift in support from HPC-Stack to spack-stack. -After completing installation, continue to the next section (:numref:`Section %s: Download the UFS SRW Application Code `). +After completing installation, continue to the :ref:`next section ` to download the UFS SRW Application Code. .. _DownloadSRWApp: @@ -77,7 +79,9 @@ The cloned repository contains the configuration files and sub-directories shown .. code-block:: console + # In a bash shell, run: export SRW=$HOME/ufs-srweather-app + # In a csh shell, run: setenv SRW $HOME/ufs-srweather-app .. _FilesAndSubDirs: @@ -91,6 +95,10 @@ The cloned repository contains the configuration files and sub-directories shown +--------------------------------+-----------------------------------------------------------+ | devbuild.sh | SRW App build script | +--------------------------------+-----------------------------------------------------------+ + | devclean.sh | Convenience script that can be used to clean up code if | + | | something goes wrong when checking out externals or | + | | building the application. | + +--------------------------------+-----------------------------------------------------------+ | docs | Contains release notes, documentation, and User's Guide | +--------------------------------+-----------------------------------------------------------+ | environment.yml | Contains information on the package versions required for | @@ -128,11 +136,11 @@ The cloned repository contains the configuration files and sub-directories shown | | using this script. | +--------------------------------+-----------------------------------------------------------+ | scripts | Contains the *ex-script* for each workflow task. | - | | These scripts are where the script logic and executables | + | | These scripts are where the task logic and executables | | | are contained. | +--------------------------------+-----------------------------------------------------------+ - | sorc | Contains CMakeLists.txt; external repositories | - | | will be cloned into this directory. | + | sorc | Contains CMakeLists.txt; source code from external | + | | repositories is cloned into this directory. | +--------------------------------+-----------------------------------------------------------+ | tests | Contains SRW App tests, including workflow end-to-end | | | (WE2E) tests. | @@ -150,8 +158,6 @@ The cloned repository contains the configuration files and sub-directories shown | | respectively. | +--------------------------------+-----------------------------------------------------------+ -.. COMMENT: Is environment.yml deprecated? Remove? - .. _CheckoutExternals: Check Out External Components @@ -163,18 +169,18 @@ Run the executable that pulls in SRW App components from external repositories: .. code-block:: console - cd + cd /path/to/ufs-srweather-app/ ./manage_externals/checkout_externals The script should output dialogue indicating that it is retrieving different code repositories. It may take several minutes to download these repositories. To see more options for the ``checkout_externals`` script, users can run ``./manage_externals/checkout_externals -h``. For example: - * ``-S``: Outputs the status of the repositories managed by ``checkout_externals``. By default only summary information is provided. Use with the ``-v`` (verbose) option to see details. + * ``-S``: Outputs the status of the repositories managed by ``checkout_externals``. By default, only summary information is provided. Use with the ``-v`` (verbose) option to see details. * ``-x [EXCLUDE [EXCLUDE ...]]``: allows users to exclude components when checking out externals. - * ``-o``: By default only the required externals are checked out. This flag will also check out the optional externals. + * ``-o``: This flag will check out the optional external repositories in addition to the default repositories (by default, only the required external repositories are checked out). -Generally, users will not need to use the options and can simply run the script, but the options are available for those who are curious. +Generally, users will not need to use these options and can simply run the script, but the options are available for those who are curious. .. _BuildExecutables: @@ -207,25 +213,7 @@ where valid values are ``intel`` or ``gnu``. The last line of the console output should be ``[100%] Built target ufs-weather-model``, indicating that the UFS Weather Model executable has been built successfully. -If users want to build the optional ``GSI`` and ``rrfs_utl`` components for :term:`RRFS`, they can pass the ``gsi`` and ``rrfs_utils`` arguments to ``devbuild.sh``. For example: - -.. code-block:: console - - ./devbuild.sh -p=hera gsi rrfs_utils - -.. note:: - RRFS capabilities are currently build-only features. They are not yet available for use at runtime. - -The last few lines of the RRFS console output should be: - -.. code-block:: console - - [100%] Built target RRFS_UTILS - Install the project... - -- Install configuration: "RELEASE" - -- Installing: /path/to/ufs-srweather-app/exec/ufs_srweather_app.settings - -After running ``devbuild.sh``, the executables listed in :numref:`Table %s ` should appear in the ``ufs-srweather-app/exec`` directory. If users choose to build the ``GSI`` and ``rrfs_utils`` components, the executables listed in :numref:`Table %s ` will also appear there. If the ``devbuild.sh`` build method does not work, or if users are not on a supported machine, they will have to manually set up the environment and build the SRW App binaries with CMake as described in :numref:`Section %s `. +After running ``devbuild.sh``, the executables listed in :numref:`Table %s ` should appear in the ``ufs-srweather-app/exec`` directory. If the ``devbuild.sh`` build method does not work, or if users are not on a supported machine, they will have to manually set up the environment and build the SRW App binaries with CMake as described in :numref:`Section %s `. .. _ExecDescription: @@ -237,6 +225,8 @@ After running ``devbuild.sh``, the executables listed in :numref:`Table %s ` files required for the coupled model. | + +------------------------+---------------------------------------------------------------------------------+ | emcsfc_ice_blend | Blends National Ice Center sea ice cover and EMC sea ice concentration data to | | | create a global sea ice analysis used to update the GFS once per day | +------------------------+---------------------------------------------------------------------------------+ @@ -277,96 +267,16 @@ After running ``devbuild.sh``, the executables listed in :numref:`Table %s `, they should skip to step :numref:`Chapter %s `. + * If users successfully built the executables in :numref:`Table %s `, they should skip to step :numref:`Chapter %s: Running the SRW App `. * Users who want to build the SRW App on MacOS or generic Linux systems should skip to :numref:`Section %s ` and follow the approach there. If the ``devbuild.sh`` approach failed, users need to set up their environment to run a workflow on their specific platform. First, users should make sure ``Lmod`` is the app used for loading modulefiles. This is the case on most Level 1 systems; however, on systems such as Gaea/Odin, the default modulefile loader is from Cray and must be switched to Lmod. For example, on Gaea, users can run one of the following two commands depending on whether they have a bash or csh shell, respectively: @@ -395,19 +305,23 @@ From here, ``Lmod`` is ready to load the modulefiles needed by the SRW App. Thes .. code-block:: console - module use + module use /path/to/modulefiles module load build__ -where ```` is the full path to the ``modulefiles`` directory. +where ``/path/to/modulefiles/`` is the full path to the ``modulefiles`` directory. + +This will work on Level 1 systems, where a modulefile is available in the ``modulefiles`` directory. Users on Level 2-4 systems (including generic Linux/MacOS systems) will need to modify an appropriate ``build__`` modulefile. One of the current ``build__`` modulefiles can be copied and used as a template. However, users will need to adjust certain environment variables in their modulefile, such as the path to HPC-Stack, so that the SRW App can find and load the appropriate modules. + +.. note:: -This will work on Level 1 systems, where a modulefile is available in the ``modulefiles`` directory. On Level 2-4 systems (including generic Linux/MacOS systems), users will need to modify certain environment variables, such as the path to HPC-Stack, so that the SRW App can find and load the appropriate modules. For systems with Lmod installed, one of the current ``build__`` modulefiles can be copied and used as a template. To check whether Lmod is installed, run ``echo $LMOD_PKG``, and see if it outputs a path to the Lmod package. On systems without Lmod, users can modify or set the required environment variables with the ``export`` or ``setenv`` commands, depending on whether they are using a bash or csh/tcsh shell, respectively: + These instructions assume that Lmod (an SRW App prerequisite) is installed. To check whether Lmod is installed, run ``echo $LMOD_PKG``, and see if it outputs a path to the Lmod package. On systems without Lmod, users can modify or set the required environment variables with the ``export`` or ``setenv`` commands, depending on whether they are using a bash or csh/tcsh shell, respectively: -.. code-block:: + .. code-block:: - export = - setenv + export = + setenv -Note that building the SRW App without Lmod is not supported at this time. It should be possible to do so, but it has not been tested. Users are encouraged to install Lmod on their system. + However, building the SRW App without Lmod is not supported at this time. It should be possible to do so, but it has not been tested. Users are encouraged to install Lmod on their system. .. _BuildCMake: @@ -428,7 +342,7 @@ From the build directory, run the following commands to build the pre-processing cmake .. -DCMAKE_INSTALL_PREFIX=.. -DCMAKE_INSTALL_BINDIR=exec .. make -j 4 >& build.out & -``-DCMAKE_INSTALL_PREFIX`` specifies the location where the ``exec``, ``include``, ``lib``, and ``share`` directories will be created. These directories will contain various components of the SRW App. Its recommended value ``..`` denotes one directory up from the build directory. In the next line, the ``make`` argument ``-j 4`` indicates that the build will run in parallel with 4 threads. Although users can specify a larger or smaller number of threads (e.g., ``-j 8``, ``-j 2``), it is highly recommended to use at least 4 parallel threads to prevent overly long installation times. +``-DCMAKE_INSTALL_PREFIX`` specifies the location where the ``exec``, ``include``, ``lib``, and ``share`` directories will be created. These directories will contain various components of the SRW App. Its recommended value ``..`` denotes one directory up from the build directory. In the next line, the ``make`` argument ``-j 4`` indicates that the build will run in parallel with four threads. Although users can specify a larger or smaller number of threads (e.g., ``-j 8``, ``-j 2``), it is highly recommended to use at least four parallel threads to prevent overly long installation times. The build will take a few minutes to complete. When it starts, a random number is printed to the console, and when it is done, a ``[1]+ Done`` message is printed to the console. ``[1]+ Exit`` indicates an error. Output from the build will be in the ``ufs-srweather-app/build/build.out`` file. When the build completes, users should see the forecast model executable ``ufs_model`` and several pre- and post-processing executables in the ``ufs-srweather-app/exec`` directory. These executables are described in :numref:`Table %s `. @@ -442,7 +356,7 @@ Additional Details for Building on MacOS or Generic Linux ------------------------------------------------------------ .. note:: - Users who are **not** building the SRW App on MacOS or generic Linux platforms may skip to :numref:`Section %s ` to finish building the SRW App or continue to :numref:`Chapter %s ` to configure and run an experiment. + Users who are **not** building the SRW App on MacOS or generic Linux platforms may skip to :numref:`Section %s ` to finish building the SRW App or continue to :numref:`Section %s ` to configure and run an experiment. The SRW App can be built on MacOS and generic Linux machines after the prerequisite software has been installed on these systems (via :term:`HPC-Stack` or :term:`spack-stack`). The installation for MacOS is architecture-independent and has been tested using both x86_64 and M1 chips (running natively). The following configurations for MacOS have been tested: @@ -452,7 +366,7 @@ The SRW App can be built on MacOS and generic Linux machines after the prerequis Several Linux builds have been tested on systems with x86_64 architectures. -The ``./modulefiles/build__gnu.lua`` modulefile (where ```` is ``macos`` or ``linux``) is written as a Lmod module in the Lua language, and it can be loaded once the Lmod module environment has been initialized (which should have happened even prior to :ref:`installing HPC-Stack `). This module lists the location of the HPC-Stack modules, loads the meta-modules and modules, sets serial and parallel compilers, additional flags, and any environment variables needed for building the SRW App. The modulefile must be modified to include the absolute path to the user's HPC-Stack installation: +The ``$SRW/modulefiles/build__gnu.lua`` modulefile (where ```` is ``macos`` or ``linux``) is written as a Lmod module in the Lua language. It can be loaded once the Lmod module environment has been initialized (which should have happened even prior to :ref:`installing HPC-Stack `). The ``build__gnu`` modulefile lists the location of the HPC-Stack modules, loads the meta-modules and modules, sets serial and parallel compilers, additional flags, and any environment variables needed for building the SRW App. The modulefile must be modified to include the absolute path to the user's HPC-Stack installation: .. code-block:: console @@ -465,8 +379,8 @@ Next, users must source the Lmod setup file, just as they would on other systems .. code-block:: console - source etc/lmod-setup.sh - module use + source /path/to/ufs-srweather-app/etc/lmod-setup.sh + module use /path/to/ufs-srweather-app/modulefiles module load build__gnu export LDFLAGS+=" -L${MPI_ROOT}/lib " diff --git a/docs/UsersGuide/source/ContainerQuickstart.rst b/docs/UsersGuide/source/BuildingRunningTesting/ContainerQuickstart.rst similarity index 66% rename from docs/UsersGuide/source/ContainerQuickstart.rst rename to docs/UsersGuide/source/BuildingRunningTesting/ContainerQuickstart.rst index 8515e097a3..d2ccb82a8c 100644 --- a/docs/UsersGuide/source/ContainerQuickstart.rst +++ b/docs/UsersGuide/source/BuildingRunningTesting/ContainerQuickstart.rst @@ -4,34 +4,38 @@ Container-Based Quick Start Guide ==================================== -This Container-Based Quick Start Guide will help users build and run the "out-of-the-box" case for the Unified Forecast System (:term:`UFS`) Short-Range Weather (SRW) Application using a `Singularity `__ container. The :term:`container` approach provides a uniform enviroment in which to build and run the SRW App. Normally, the details of building and running the SRW App vary from system to system due to the many possible combinations of operating systems, compilers, :term:`MPIs `, and package versions available. Installation via Singularity container reduces this variability and allows for a smoother SRW App build experience. Normally, containers can only run on a single compute node and are not compatible with the `Rocoto workflow manager `__, so users must run each task in the workflow manually. However, the Singularity container described in this chapter has been adapted such that it is able to run across multiple nodes using Rocoto. This makes it an excellent starting point for beginners. The :ref:`non-container build approach ` may still be more appropriate for users who desire additional customizability, particularly if they already have experience running the SRW App. +This Container-Based Quick Start Guide will help users build and run the "out-of-the-box" case for the Unified Forecast System (:term:`UFS`) Short-Range Weather (SRW) Application using a `Singularity/Apptainer `__ container. The :term:`container` approach provides a uniform enviroment in which to build and run the SRW App. Normally, the details of building and running the SRW App vary from system to system due to the many possible combinations of operating systems, compilers, :term:`MPIs `, and package versions available. Installation via container reduces this variability and allows for a smoother SRW App build experience. -The "out-of-the-box" SRW App case described in this User's Guide builds a weather forecast for June 15-16, 2019. Multiple convective weather events during these two days produced over 200 filtered storm reports. Severe weather was clustered in two areas: the Upper Midwest through the Ohio Valley and the Southern Great Plains. This forecast uses a predefined 25-km Continental United States (:term:`CONUS`) grid (RRFS_CONUS_25km), the Global Forecast System (:term:`GFS`) version 16 physics suite (FV3_GFS_v16 :term:`CCPP`), and :term:`FV3`-based GFS raw external model data for initialization. +The basic "out-of-the-box" case described in this User's Guide builds a weather forecast for June 15-16, 2019. Multiple convective weather events during these two days produced over 200 filtered storm reports. Severe weather was clustered in two areas: the Upper Midwest through the Ohio Valley and the Southern Great Plains. This forecast uses a predefined 25-km Continental United States (:term:`CONUS`) grid (RRFS_CONUS_25km), the Global Forecast System (:term:`GFS`) version 16 physics suite (FV3_GFS_v16 :term:`CCPP`), and :term:`FV3`-based GFS raw external model data for initialization. .. attention:: * The SRW Application has `four levels of support `__. The steps described in this chapter will work most smoothly on preconfigured (Level 1) systems. However, this guide can serve as a starting point for running the SRW App on other systems, too. - * This chapter of the User's Guide should **only** be used for container builds. For non-container builds, see :numref:`Chapter %s ` for a Quick Start Guide or :numref:`Chapter %s ` for a detailed guide to building the SRW App **without** a container. + * This chapter of the User's Guide should **only** be used for container builds. For non-container builds, see :numref:`Section %s ` for a Quick Start Guide or :numref:`Section %s ` for a detailed guide to building the SRW App **without** a container. .. _DownloadCodeC: Download the Container -=========================================== +========================== -Prerequisites: +Prerequisites ------------------- -Users must have an **Intel** compiler and :term:`MPI` (available for free `here `__) in order to run the SRW App in the container provided using the method described in this chapter. Additionally, it is recommended that users install the `Rocoto workflow manager `__ on their system in order to take advantage of automated workflow options. Although it is possible to run an experiment without Rocoto, and some tips are provided, the only fully-supported and tested container option for the ``develop`` branch assumes that Rocoto is pre-installed. +Users must have an **Intel** compiler and :term:`MPI` (available for free `here `__) in order to run the SRW App in the container provided using the method described in this chapter. Additionally, it is recommended that users install the `Rocoto workflow manager `__ on their system in order to take advantage of automated workflow options. Although it is possible to run an experiment without Rocoto, and some tips are provided, the only fully-supported and tested container option assumes that Rocoto is pre-installed. -.. COMMENT: Remove "for the develop branch"? +Install Singularity/Apptainer +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -Install Singularity -^^^^^^^^^^^^^^^^^^^^^^^ +.. note:: + + As of November 2021, the Linux-supported version of Singularity has been `renamed `__ to *Apptainer*. Apptainer has maintained compatibility with Singularity, so ``singularity`` commands should work with either Singularity or Apptainer (see compatibility details `here `__.) -To build and run the SRW App using a Singularity container, first install the Singularity package according to the `Singularity Installation Guide `__. This will include the installation of dependencies and the installation of the Go programming language. SingularityCE Version 3.7 or above is recommended. +To build and run the SRW App using a Singularity/Apptainer container, first install the software according to the `Apptainer Installation Guide `__. This will include the installation of all dependencies. .. warning:: - Docker containers can only be run with root privileges, and users cannot have root privileges on :term:`HPCs `. Therefore, it is not possible to build the SRW App, which uses the HPC-Stack, inside a Docker container on an HPC system. However, a Singularity image may be built directly from a Docker image for use on the system. + Docker containers can only be run with root privileges, and users cannot have root privileges on :term:`HPCs `. Therefore, it is not possible to build the SRW App, which uses the spack-stack, inside a Docker container on an HPC system. However, a Singularity/Apptainer image may be built directly from a Docker image for use on the system. + +.. COMMENT: Update reference to HPC-Stack --> spack-stack? Working in the Cloud or on HPC Systems ----------------------------------------- @@ -40,8 +44,8 @@ For users working on systems with limited disk space in their ``/home`` director .. code-block:: - export SINGULARITY_CACHEDIR= - export SINGULARITY_TMPDIR= + export SINGULARITY_CACHEDIR=/absolute/path/to/writable/directory/cache + export SINGULARITY_TMPDIR=/absolute/path/to/writable/directory/tmp where ``/absolute/path/to/writable/directory/`` refers to a writable directory (usually a project or user directory within ``/lustre``, ``/work``, ``/scratch``, or ``/glade`` on NOAA Level 1 systems). If the ``cache`` and ``tmp`` directories do not exist already, they must be created with a ``mkdir`` command. @@ -65,7 +69,7 @@ Build the Container ------------------------ .. hint:: - If a ``singularity: command not found`` error message appears in any of the following steps, try running: ``module load singularity``. + If a ``singularity: command not found`` error message appears when working on Level 1 platforms, try running: ``module load singularity``. Level 1 Systems ^^^^^^^^^^^^^^^^^^ @@ -79,6 +83,8 @@ On most Level 1 systems, a container named ``ubuntu20.04-intel-srwapp-develop.im +==============+========================================================+ | Cheyenne | /glade/scratch/epicufsrt/containers | +--------------+--------------------------------------------------------+ + | Gaea | /lustre/f2/dev/role.epic/containers | + +--------------+--------------------------------------------------------+ | Hera | /scratch1/NCEPDEV/nems/role.epic/containers | +--------------+--------------------------------------------------------+ | Jet | /mnt/lfs4/HFIP/hfv3gfs/role.epic/containers | @@ -89,19 +95,22 @@ On most Level 1 systems, a container named ``ubuntu20.04-intel-srwapp-develop.im +--------------+--------------------------------------------------------+ .. note:: - Singularity is not available on Gaea, and therefore container use is not supported on Gaea. + * Singularity is only available on the Gaea C5 partition, and therefore container use is only supported on Gaea C5. + * The NOAA Cloud containers are accessible only to those with EPIC resources. -Users can simply copy the container to their local working directory. For example, on Hera: +Users can simply set an environment variable to point to the container: .. code-block:: console - cp /scratch1/NCEPDEV/nems/role.epic/containers/ubuntu20.04-intel-srwapp-develop.img . + export img=/path/to/ubuntu20.04-intel-srwapp-develop.img Users may convert the container ``.img`` file to a writable sandbox. This step is required when running on Cheyenne but is optional on other systems: .. code-block:: console - singularity build --sandbox ubuntu20.04-intel-srwapp ubuntu20.04-intel-srwapp-develop.img + singularity build --sandbox ubuntu20.04-intel-srwapp $img + +.. COMMENT: What about on Derecho? When making a writable sandbox on Level 1 systems, the following warnings commonly appear and can be ignored: @@ -130,6 +139,12 @@ Some users may prefer to issue the command without the ``sudo`` prefix. Whether sudo singularity build --sandbox ubuntu20.04-intel-srwapp docker://noaaepic/ubuntu20.04-intel-srwapp:release-public-v2.1.0 +For easier reference, users can set an environment variable to point to the container: + +.. code-block:: console + + export img=/path/to/ubuntu20.04-intel-srwapp + .. _WorkOnHPC: @@ -170,19 +185,14 @@ Copy ``stage-srw.sh`` from the container to the local working directory: .. code-block:: console - singularity exec -B /:/ ./ cp /opt/ufs-srweather-app/container-scripts/stage-srw.sh . - -where ```` is the name of the sandbox directory (i.e., ``ubuntu20.04-intel-srwapp``) or the name of the ``.img`` container file. - -.. hint:: - On Jet, users may need to bind to an ``lfs`` directory (e.g., ``/lfs4``) rather than ``/mnt``. + singularity exec -B /:/ $img cp /opt/ufs-srweather-app/container-scripts/stage-srw.sh . -If the command worked properly, ``stage-srw.sh`` should appear in the local directory. The command above also binds the local directory to the container so that data can be shared between them. On `Level 1 `__ systems, ```` is usually the topmost directory (e.g., ``/lustre``, ``/contrib``, ``/work``, or ``/home``). Additional directories can be bound by adding another ``-B /:/`` argument before the name of the container. In general, it is recommended that the local base directory and container directory have the same name. For example, if the host system's top-level directory is ``/user1234``, the user can create a ``user1234`` directory in the container sandbox and then bind it: +If the command worked properly, ``stage-srw.sh`` should appear in the local directory. The command above also binds the local directory to the container so that data can be shared between them. On `Level 1 `__ systems, ```` is usually the topmost directory (e.g., ``/lustre``, ``/contrib``, ``/work``, or ``/home``). Additional directories can be bound by adding another ``-B /:/`` argument before the name of the container. In general, it is recommended that the local base directory and container directory have the same name. For example, if the host system's top-level directory is ``/user1234``, the user can create a ``user1234`` directory in the writable container sandbox and then bind it: .. code-block:: console - mkdir /user1234 - singularity exec -B /user1234:/user1234 ./ubuntu20.04-intel-srwapp cp /opt/ufs-srweather-app/container-scripts/stage-srw.sh . + mkdir /path/to/container/user1234 + singularity exec -B /user1234:/user1234 $img cp /opt/ufs-srweather-app/container-scripts/stage-srw.sh . .. attention:: Be sure to bind the directory that contains the experiment data! @@ -191,7 +201,7 @@ To explore the container and view available directories, users can either ``cd`` .. code-block:: console - singularity shell ./ubuntu20.04-intel-srwapp-develop.img + singularity shell $img cd / ls @@ -210,7 +220,16 @@ Users can run ``exit`` to exit the shell. Download and Stage the Data ============================ -The SRW App requires input files to run. These include static datasets, initial and boundary condition files, and model configuration files. On Level 1 systems, the data required to run SRW App tests are already available as long as the bind argument (starting with ``-B``) in :numref:`Step %s ` included the directory with the input model data. See :numref:`Table %s ` for Level 1 data locations. For Level 2-4 systems, the data must be added manually by the user. Detailed instructions on how to add the data can be found in :numref:`Section %s `. Sections :numref:`%s ` and :numref:`%s ` contain useful background information on the input and output files used in the SRW App. +The SRW App requires input files to run. These include static datasets, initial and boundary condition files, and model configuration files. On Level 1 systems, the data required to run SRW App tests are already available as long as the bind argument (starting with ``-B``) in :numref:`Step %s ` included the directory with the input model data. See :numref:`Table %s ` for Level 1 data locations. For Level 2-4 systems, the data must be added manually by the user. In general, users can download fix file data and experiment data (:term:`ICs/LBCs`) from the `SRW App Data Bucket `__ and then untar it: + +.. code-block:: console + + wget https://noaa-ufs-srw-pds.s3.amazonaws.com/current_srw_release_data/fix_data.tgz + wget https://noaa-ufs-srw-pds.s3.amazonaws.com/current_srw_release_data/gst_data.tgz + tar -xzf fix_data.tgz + tar -xzf gst_data.tgz + +More detailed information can be found in :numref:`Section %s `. Sections :numref:`%s ` and :numref:`%s ` contain useful background information on the input and output files used in the SRW App. .. _GenerateForecastC: @@ -218,7 +237,7 @@ Generate the Forecast Experiment ================================= To generate the forecast experiment, users must: -#. :ref:`Activate the regional workflow ` +#. :ref:`Activate the workflow ` #. :ref:`Set experiment parameters ` #. :ref:`Run a script to generate the experiment workflow ` @@ -226,37 +245,39 @@ The first two steps depend on the platform being used and are described here for .. _SetUpPythonEnvC: -Activate the Regional Workflow -------------------------------------- +Activate the Workflow +------------------------ Copy the container's modulefiles to the local working directory so that the files can be accessed outside of the container: .. code-block:: console - singularity exec -B /:/ ./ cp -r /opt/ufs-srweather-app/modulefiles . + singularity exec -B /:/ $img cp -r /opt/ufs-srweather-app/modulefiles . After this command runs, the local working directory should contain the ``modulefiles`` directory. -To activate the regional workflow, run the following commands: +To activate the workflow, run the following commands: .. code-block:: console - module use + module use /path/to/modulefiles module load wflow_ where: - * ```` is replaced with the actual path to the modulefiles on the user's local system (often ``$PWD/modulefiles``), and + * ``/path/to/modulefiles`` is replaced with the actual path to the modulefiles on the user's local system (often ``$PWD/modulefiles``), and * ```` is a valid, lowercased machine/platform name (see the ``MACHINE`` variable in :numref:`Section %s `). -The ``wflow_`` modulefile will then output instructions to activate the regional workflow. The user should run the commands specified in the modulefile output. For example, if the output says: +The ``wflow_`` modulefile will then output instructions to activate the workflow. The user should run the commands specified in the modulefile output. For example, if the output says: .. code-block:: console Please do the following to activate conda: - > conda activate regional_workflow + > conda activate workflow_tools -then the user should run ``conda activate regional_workflow``. This will activate the ``regional_workflow`` conda environment. The command(s) will vary from system to system, but the user should see ``(regional_workflow)`` in front of the Terminal prompt at this point. +then the user should run ``conda activate workflow_tools``. This will activate the ``workflow_tools`` conda environment. The command(s) will vary from system to system, but the user should see ``(workflow_tools)`` in front of the Terminal prompt at this point. + +.. COMMENT: Containers are old and still say regional_workflow... .. _SetUpConfigFileC: @@ -267,14 +288,14 @@ Run ``stage-srw.sh``: .. code-block:: console - ./stage-srw.sh -c= -m= -p= -i= + ./stage-srw.sh -c= -m= -p= -i=$img where: * ``-c`` indicates the compiler on the user's local machine (e.g., ``intel/2022.1.2``) * ``-m`` indicates the :term:`MPI` on the user's local machine (e.g., ``impi/2022.1.2``) * ```` refers to the local machine (e.g., ``hera``, ``jet``, ``noaacloud``, ``mac``). See ``MACHINE`` in :numref:`Section %s ` for a full list of options. - * ``-i`` indicates the name of the container image that was built in :numref:`Step %s ` (``ubuntu20.04-intel-srwapp`` or ``ubuntu20.04-intel-srwapp-develop.img`` by default). + * ``-i`` indicates the container image that was built in :numref:`Step %s ` (``ubuntu20.04-intel-srwapp`` or ``ubuntu20.04-intel-srwapp-develop.img`` by default). For example, on Hera, the command would be: @@ -323,8 +344,6 @@ From here, users can follow the steps below to configure the out-of-the-box SRW USE_USER_STAGED_EXTRN_FILES: true EXTRN_MDL_SOURCE_BASEDIR_ICS: /scratch1/NCEPDEV/nems/role.epic/UFS_SRW_data/develop/input_model_data/FV3GFS/grib2/${yyyymmddhh} - EXTRN_MDL_FILES_ICS: [] - EXTRN_MDL_DATA_STORES: disk On other systems, users will need to change the path for ``EXTRN_MDL_SOURCE_BASEDIR_ICS`` and ``EXTRN_MDL_FILES_LBCS`` (below) to reflect the location of the system's data. The location of the machine's global data can be viewed :ref:`here ` for Level 1 systems. Alternatively, the user can add the path to their local data if they downloaded it as described in :numref:`Section %s `. @@ -334,8 +353,6 @@ From here, users can follow the steps below to configure the out-of-the-box SRW USE_USER_STAGED_EXTRN_FILES: true EXTRN_MDL_SOURCE_BASEDIR_LBCS: /scratch1/NCEPDEV/nems/role.epic/UFS_SRW_data/develop/input_model_data/FV3GFS/grib2/${yyyymmddhh} - EXTRN_MDL_FILES_LBCS: [] - EXTRN_MDL_DATA_STORES: disk .. _GenerateWorkflowC: @@ -362,29 +379,50 @@ The generated workflow will be in the experiment directory specified in the ``co cd ../../expt_dirs/test_community rocotostat -w FV3LAM_wflow.xml -d FV3LAM_wflow.db -v 10 -Users can track the experiment's progress by reissuing the ``rocotostat`` command above every so often until the experiment runs to completion. For users who do not have Rocoto installed, see :numref:`Section %s ` for information on how to run the workflow without Rocoto. +Users can track the experiment's progress by reissuing the ``rocotostat`` command above every so often until the experiment runs to completion. The following message usually means that the experiment is still getting set up: + +.. code-block:: console + + 08/04/23 17:34:32 UTC :: FV3LAM_wflow.xml :: ERROR: Can not open FV3LAM_wflow.db read-only because it does not exist + +After a few (3-5) minutes, ``rocotostat`` should show a status-monitoring table: + +.. code-block:: console + + CYCLE TASK JOBID STATE EXIT STATUS TRIES DURATION + ================================================================================== + 201906151800 make_grid 53583094 QUEUED - 0 0.0 + 201906151800 make_orog - - - - - + 201906151800 make_sfc_climo - - - - - + 201906151800 get_extrn_ics 53583095 QUEUED - 0 0.0 + 201906151800 get_extrn_lbcs 53583096 QUEUED - 0 0.0 + 201906151800 make_ics - - - - - + 201906151800 make_lbcs - - - - - + 201906151800 run_fcst - - - - - + 201906151800 run_post_f000 - - - - - + ... + 201906151800 run_post_f012 - - - - - + +When all tasks show ``SUCCEEDED``, the experiment has completed successfully. + +For users who do not have Rocoto installed, see :numref:`Section %s ` for guidance on how to run the workflow without Rocoto. Troubleshooting ------------------ -If a task goes DEAD, it will be necessary to restart it according to the instructions in :numref:`Section %s `. To determine what caused the task to go DEAD, users should view the log file for the task in ``$EXPTDIR/log/``, where ```` refers to the name of the task's log file. After fixing the problem and clearing the DEAD task, it is sometimes necessary to reinitialize the crontab. Users can copy-paste the crontab command from the bottom of the ``$EXPTDIR/log.generate_FV3LAM_wflow`` file into the crontab: + +If a task goes DEAD, it will be necessary to restart it according to the instructions in :numref:`Section %s `. To determine what caused the task to go DEAD, users should view the log file for the task in ``$EXPTDIR/log/``, where ```` refers to the name of the task's log file. After fixing the problem and clearing the DEAD task, it is sometimes necessary to reinitialize the crontab. Run ``crontab -e`` to open your configured editor. Inside the editor, copy-paste the crontab command from the bottom of the ``$EXPTDIR/log.generate_FV3LAM_wflow`` file into the crontab: .. code-block:: console crontab -e - i - */3 * * * * cd //expt_dirs/test_community && ./launch_FV3LAM_wflow.sh called_from_cron="TRUE" - esc - :wq - enter - -.. COMMENT: Check the crontab command to reflect python workflow.s + */3 * * * * cd /path/to/expt_dirs/test_community && ./launch_FV3LAM_wflow.sh called_from_cron="TRUE" where: - * ```` is replaced by the actual path to the user's experiment directory, and + * ``/path/to`` is replaced by the actual path to the user's experiment directory, and * ``esc`` and ``enter`` refer to the escape and enter **keys** (not a typed command). New Experiment =============== -To run a new experiment in the container at a later time, users will need to rerun the commands in :numref:`Section %s ` to reactivate the regional workflow. Then, users can configure a new experiment by updating the environment variables in ``config.yaml`` to reflect the desired experiment configuration. Basic instructions appear in :numref:`Section %s ` above, and detailed instructions can be viewed in :numref:`Section %s `. After adjusting the configuration file, regenerate the experiment by running ``./generate_FV3LAM_wflow.py``. +To run a new experiment in the container at a later time, users will need to rerun the commands in :numref:`Section %s ` to reactivate the workflow. Then, users can configure a new experiment by updating the environment variables in ``config.yaml`` to reflect the desired experiment configuration. Basic instructions appear in :numref:`Section %s ` above, and detailed instructions can be viewed in :numref:`Section %s `. After adjusting the configuration file, regenerate the experiment by running ``./generate_FV3LAM_wflow.py``. diff --git a/docs/UsersGuide/source/BuildingRunningTesting/DefaultVarsTable.rst b/docs/UsersGuide/source/BuildingRunningTesting/DefaultVarsTable.rst new file mode 100644 index 0000000000..1ddb7f18d1 --- /dev/null +++ b/docs/UsersGuide/source/BuildingRunningTesting/DefaultVarsTable.rst @@ -0,0 +1,106 @@ +:orphan: + +================================================ +Table of Variables in ``config_defaults.yaml`` +================================================ + +.. list-table:: Configuration variables specified in the config_defaults.yaml script + :widths: 20 50 + :header-rows: 1 + + * - Group Name + - Configuration variables + * - User + - RUN_ENVIR, MACHINE, ACCOUNT, HOMEdir, USHdir, SCRIPTSdir, JOBSdir, SORCdir, PARMdir, MODULESdir, EXECdir, VX_CONFIG_DIR, METPLUS_CONF, MET_CONFIG, UFS_WTHR_MDL_DIR, ARL_NEXUS_DIR + * - Platform + - WORKFLOW_MANAGER, NCORES_PER_NODE, TASKTHROTTLE, BUILD_MOD_FN, WFLOW_MOD_FN, BUILD_VER_FN, RUN_VER_FN, SCHED,PARTITION_DEFAULT, QUEUE_DEFAULT, PARTITION_HPSS, + QUEUE_HPSS, PARTITION_FCST, QUEUE_FCST, REMOVE_MEMORY, RUN_CMD_SERIAL, RUN_CMD_UTILS, RUN_CMD_FCST, RUN_CMD_POST, RUN_CMD_PRDGEN, RUN_CMD_AQM, + RUN_CMD_AQMLBC, SCHED_NATIVE_CMD, CCPA_OBS_DIR, MRMS_OBS_DIR, NDAS_OBS_DIR, NOHRSC_OBS_DIR, DOMAIN_PREGEN_BASEDIR, PRE_TASK_CMDS, + TEST_EXTRN_MDL_SOURCE_BASEDIR, TEST_AQM_INPUT_BASEDIR, TEST_PREGEN_BASEDIR, TEST_ALT_EXTRN_MDL_SYSBASEDIR_ICS, TEST_ALT_EXTRN_MDL_SYSBASEDIR_LBCS, + TEST_VX_FCST_INPUT_BASEDIR, FIXgsm, FIXaer, FIXlut, FIXorg, FIXsfc, FIXshp, FIXgsi, FIXcrtm, FIXcrtmupp, EXTRN_MDL_DATA_STORES + * - Workflow + - WORKFLOW_ID, RELATIVE_LINK_FLAG, USE_CRON_TO_RELAUNCH, CRON_RELAUNCH_INTVL_MNTS, CRONTAB_LINE, LOAD_MODULES_RUN_TASK_FP, EXPT_BASEDIR, EXPT_SUBDIR, EXEC_SUBDIR, + EXPTDIR, DOT_OR_USCORE, EXPT_CONFIG_FN, CONSTANTS_FN, RGNL_GRID_NML_FN, FV3_NML_FN, FV3_NML_BASE_SUITE_FN, FV3_NML_YAML_CONFIG_FN, FV3_NML_BASE_ENS_FN, + FV3_EXEC_FN, DATA_TABLE_FN, DIAG_TABLE_FN, FIELD_TABLE_FN, DIAG_TABLE_TMPL_FN, FIELD_TABLE_TMPL_FN, MODEL_CONFIG_FN, NEMS_CONFIG_FN, AQM_RC_FN, AQM_RC_TMPL_FN, + FV3_NML_BASE_SUITE_FP, FV3_NML_YAML_CONFIG_FP, FV3_NML_BASE_ENS_FP, DATA_TABLE_TMPL_FP, DIAG_TABLE_TMPL_FP, FIELD_TABLE_TMPL_FP, + MODEL_CONFIG_TMPL_FP, NEMS_CONFIG_TMPL_FP, AQM_RC_TMPL_FP, DATA_TABLE_FP, FIELD_TABLE_FP, NEMS_CONFIG_FP, FV3_NML_FP, FV3_NML_CYCSFC_FP, + FV3_NML_RESTART_FP, FV3_NML_STOCH_FP, FV3_NML_RESTART_STOCH_FP, FCST_MODEL, WFLOW_XML_FN, GLOBAL_VAR_DEFNS_FN, ROCOTO_YAML_FN, EXTRN_MDL_VAR_DEFNS_FN, + WFLOW_LAUNCH_SCRIPT_FN, WFLOW_LAUNCH_LOG_FN, GLOBAL_VAR_DEFNS_FP, ROCOTO_YAML_FP, WFLOW_LAUNCH_SCRIPT_FP, WFLOW_LAUNCH_LOG_FP, FIXdir, FIXam, + FIXclim, FIXlam, THOMPSON_MP_CLIMO_FN, THOMPSON_MP_CLIMO_FP, CCPP_PHYS_SUITE, CCPP_PHYS_SUITE_FN, CCPP_PHYS_SUITE_IN_CCPP_FP, CCPP_PHYS_SUITE_FP, + FIELD_DICT_FN, FIELD_DICT_IN_UWM_FP, FIELD_DICT_FP, GRID_GEN_METHOD, PREDEF_GRID_NAME, DATE_FIRST_CYCL, DATE_LAST_CYCL, INCR_CYCL_FREQ, FCST_LEN_HRS, + FCST_LEN_CYCL, LONG_FCST_LEN, CYCL_HRS_SPINSTART, CYCL_HRS_PRODSTART, BOUNDARY_LEN_HRS, BOUNDARY_LONG_LEN_HRS, BOUNDARY_PROC_GROUP_NUM, + PREEXISTING_DIR_METHOD, VERBOSE, DEBUG, COMPILER, SYMLINK_FIX_FILES, DO_REAL_TIME, COLDSTART, WARMSTART_CYCLE_DIR, + * - NCO + - envir_default, NET_default, RUN_default, model_ver_default, OPSROOT_default, COMROOT_default, DATAROOT_default, DCOMROOT_default, LOGBASEDIR_default, + COMIN_BASEDIR, COMOUT_BASEDIR, NWGES, NWGES_BASEDIR, DBNROOT_default, SENDECF_default, SENDDBN_default, SENDDBN_NTC_default, SENDCOM_default, + SENDWEB_default, KEEPDATA_default, MAILTO_default, MAILCC_default + * - gsi + - niter1, niter2, l_obsprvdiag, diag_radardbz, write_diag_2, bkgerr_vs, bkgerr_hzscl, usenewgfsberror, netcdf_diag, binary_diag, readin_localization, + beta1_inv, ens_h, ens_v, regional_ensemble_option, grid_ratio_fv3, grid_ratio_ens, i_en_perts_io, q_hyb_ens, ens_fast_read, l_PBL_pseudo_SurfobsT, + l_PBL_pseudo_SurfobsQ, i_use_2mQ4B, i_use_2mT4B, i_T_Q_adjust, l_rtma3d, i_precip_vertical_check, HYBENSMEM_NMIN, ANAVINFO_FN, ANAVINFO_DBZ_FN, + ENKF_ANAVINFO_FN, ENKF_ANAVINFO_DBZ_FN, CONVINFO_FN, BERROR_FN, OBERROR_FN, HYBENSINFO_FN, cld_bld_hgt, l_precip_clear_only, l_qnr_from_qr, beta_recenter + * - rrfs + - DO_RRFS_DEV, DO_NLDN_LGHT, DO_ENKFUPDATE, DO_DACYCLE, DO_SURFACE_CYCLE + * - task_make_grid + - GRID_DIR, ESGgrid_LON_CTR, ESGgrid_LAT_CTR, ESGgrid_DELX, ESGgrid_DELY, ESGgrid_NX, ESGgrid_NY, ESGgrid_WIDE_HALO_WIDTH, ESGgrid_PAZI, + GFDLgrid_LON_T6_CTR, GFDLgrid_LAT_T6_CTR, GFDLgrid_NUM_CELLS, GFDLgrid_STRETCH_FAC, GFDLgrid_REFINE_RATIO, GFDLgrid_ISTART_OF_RGNL_DOM_ON_T6G, + GFDLgrid_IEND_OF_RGNL_DOM_ON_T6G, GFDLgrid_JSTART_OF_RGNL_DOM_ON_T6G, GFDLgrid_JEND_OF_RGNL_DOM_ON_T6G, GFDLgrid_USE_NUM_CELLS_IN_FILENAMES + * - task_make_orog + - KMP_AFFINITY_MAKE_OROG, OMP_NUM_THREADS_MAKE_OROG, OMP_STACKSIZE_MAKE_OROG, OROG_DIR + * - task_make_sfc_climo + - KMP_AFFINITY_MAKE_SFC_CLIMO, OMP_NUM_THREADS_MAKE_SFC_CLIMO, OMP_STACKSIZE_MAKE_SFC_CLIMO, SFC_CLIMO_DIR + * - task_get_extrn_ics + - EXTRN_MDL_NAME_ICS, EXTRN_MDL_ICS_OFFSET_HRS, FV3GFS_FILE_FMT_ICS, EXTRN_MDL_SYSBASEDIR_ICS, USE_USER_STAGED_EXTRN_FILES, + EXTRN_MDL_SOURCE_BASEDIR_ICS, EXTRN_MDL_FILES_ICS + * - task_get_extrn_lbcs + - EXTRN_MDL_NAME_LBCS, LBC_SPEC_INTVL_HRS, EXTRN_MDL_LBCS_OFFSET_HRS, FV3GFS_FILE_FMT_LBCS, LBCS_SEARCH_HRS, EXTRN_MDL_LBCS_SEARCH_OFFSET_HRS, EXTRN_MDL_SYSBASEDIR_LBCS, + USE_USER_STAGED_EXTRN_FILES,EXTRN_MDL_SOURCE_BASEDIR_LBCS, EXTRN_MDL_FILES_LBCS + * - task_make_ics + - KMP_AFFINITY_MAKE_ICS, OMP_NUM_THREADS_MAKE_ICS, OMP_STACKSIZE_MAKE_ICS, USE_FVCOM, FVCOM_WCSTART, FVCOM_DIR, FVCOM_FILE, VCOORD_FILE + * - task_make_lbcs + - KMP_AFFINITY_MAKE_LBCS, OMP_NUM_THREADS_MAKE_LBCS, OMP_STACKSIZE_MAKE_LBCS, VCOORD_FILE + * - task_run_fcst + - NNODES_RUN_FCST, PPN_RUN_FCST, FV3_EXEC_FP, IO_LAYOUT_Y, KMP_AFFINITY_RUN_FCST, OMP_NUM_THREADS_RUN_FCST, OMP_STACKSIZE_RUN_FCST, DT_ATMOS, FHROT, RESTART_INTERVAL, WRITE_DOPOST, + LAYOUT_X, LAYOUT_Y, BLOCKSIZE, QUILTING, PRINT_ESMF, PE_MEMBER01, WRTCMP_write_groups, WRTCMP_write_tasks_per_group, WRTCMP_output_grid, WRTCMP_cen_lon, + WRTCMP_cen_lat, WRTCMP_lon_lwr_left, WRTCMP_lat_lwr_left, WRTCMP_lon_upr_rght, WRTCMP_lat_upr_rght, WRTCMP_dlon, + WRTCMP_dlat, WRTCMP_stdlat1, WRTCMP_stdlat2, WRTCMP_nx, WRTCMP_ny, WRTCMP_dx, WRTCMP_dy, USE_MERRA_CLIMO, DO_FCST_RESTART + * - task_run_post + - KMP_AFFINITY_RUN_POST, OMP_NUM_THREADS_RUN_POST, OMP_STACKSIZE_RUN_POST, SUB_HOURLY_POST, DT_SUB_HOURLY_POST_MNTS, + USE_CUSTOM_POST_CONFIG_FILE, CUSTOM_POST_CONFIG_FP, POST_OUTPUT_DOMAIN_NAME, TESTBED_FIELDS_FN + * - task_run_prdgen: + - KMP_AFFINITY_RUN_PRDGEN, OMP_NUM_THREADS_RUN_PRDGEN, OMP_STACKSIZE_RUN_PRDGEN, DO_PARALLEL_PRDGEN, ADDNL_OUTPUT_GRIDS: [] + * - task_plot_allvars: + - COMOUT_REF, PLOT_FCST_START, PLOT_FCST_INC, PLOT_FCST_END, PLOT_DOMAINS + * - task_analysis_gsi + - TN_ANALYSIS_GSI, TN_OBSERVER_GSI, TN_OBSERVER_GSI_ENSMEAN, KMP_AFFINITY_ANALYSIS, OMP_NUM_THREADS_ANALYSIS, OMP_STACKSIZE_ANALYSIS, OBSPATH_TEMPLATE + * - task_process_radarref + - RADAR_REF_THINNING, RADARREFL_MINS, RADARREFL_TIMELEVEL, OBS_SUFFIX + * - task_get_da_obs + - NLDN_NEEDED, NLDN_LIGHTNING, NSSLMOSAIC, RAP_OBS_BUFR + * - task_process_bufrobs + - OBSPATH_TEMPLATE + * - task_nexus_emission + - PPN_NEXUS_EMISSION, KMP_AFFINITY_NEXUS_EMISSION, OMP_NUM_THREADS_NEXUS_EMISSION, OMP_STACKSIZE_NEXUS_EMISSION + * - task_bias_correction_o3 + - KMP_AFFINITY_BIAS_CORRECTION_O3, OMP_NUM_THREADS_BIAS_CORRECTION_O3, OMP_STACKSIZE_BIAS_CORRECTION_O3 + * - task_bias_correction_pm25 + - KMP_AFFINITY_BIAS_CORRECTION_PM25, OMP_NUM_THREADS_BIAS_CORRECTION_PM25, OMP_STACKSIZE_BIAS_CORRECTION_PM25 + * - Global + - USE_CRTM, CRTM_DIR, DO_ENSEMBLE, NUM_ENS_MEMBERS, ENSMEM_NAMES, FV3_NML_ENSMEM_FPS, ENS_TIME_LAG_HRS, DO_SHUM, DO_SPPT, DO_SKEB, ISEED_SHUM, ISEED_SPPT, ISEED_SKEB, NEW_LSCALE, SHUM_MAG, SHUM_LSCALE, SHUM_TSCALE, SHUM_INT, + SPPT_MAG, SPPT_LOGIT, SPPT_LSCALE, SPPT_TSCALE, SPPT_INT, SPPT_SFCLIMIT, + SKEB_MAG, SKEB_LSCALE, SKEP_TSCALE, SKEB_INT, SKEBNORM, SKEB_VDOF, USE_ZMTNBLCK, DO_SPP, ISEED_SPP, SPP_VAR_LIST, SPP_MAG_LIST, SPP_LSCALE, + SPP_TSCALE, SPP_SIGTOP1, SPP_SIGTOP2, SPP_STDDEV_CUTOFF, DO_LSM_SPP, LSM_SPP_TSCALE, LSM_SPP_LSCALE, ISEED_LSM_SPP, LSM_SPP_VAR_LIST, + LSM_SPP_MAG_LIST, HALO_BLEND, PRINT_DIFF_PGR + * - Verification + - OBS_CCPA_APCP01h_FN_TEMPLATE, OBS_CCPA_APCPgt01h_FN_TEMPLATE, OBS_MRMS_REFC_FN_TEMPLATE, OBS_MRMS_RETOP_FN_TEMPLATE, + OBS_NDAS_SFCorUPA_FN_TEMPLATE, OBS_NDAS_SFCorUPA_FN_METPROC_TEMPLATE, VX_FCST_MODEL_NAME, VX_FIELDS, VX_APCP_ACCUMS_HRS, VX_FCST_INPUT_BASEDIR, + VX_OUTPUT_BASEDIR, VX_NDIGITS_ENSMEM_NAMES, FCST_SUBDIR_TEMPLATE, FCST_FN_TEMPLATE, FCST_FN_METPROC_TEMPLATE, NUM_MISSING_OBS_FILES_MAX, NUM_MISSING_FCST_FILES_MAX + * - cpl_aqm_parm + - CPL_AQM, DO_AQM_DUST, DO_AQM_CANOPY, DO_AQM_PRODUCT, DO_AQM_CHEM_LBCS, DO_AQM_GEFS_LBCS, DO_AQM_SAVE_AIRNOW_HIST, DO_AQM_SAVE_FIRE, DCOMINbio_default, + DCOMINdust_default, DCOMINcanopy_default, DCOMINfire_default, DCOMINchem_lbcs_default, DCOMINgefs_default, DCOMINpt_src_default, + DCOMINairnow_default, COMINbicor, COMOUTbicor, AQM_CONFIG_DIR, AQM_BIO_FILE, AQM_DUST_FILE_PREFIX, AQM_DUST_FILE_SUFFIX, AQM_CANOPY_FILE_PREFIX, + AQM_CANOPY_FILE_SUFFIX, AQM_FIRE_FILE_PREFIX, AQM_FIRE_FILE_SUFFIX, AQM_FIRE_FILE_OFFSET_HRS, AQM_FIRE_ARCHV_DIR, AQM_RC_FIRE_FREQUENCY, + AQM_RC_PRODUCT_FN, AQM_RC_PRODUCT_FREQUENCY, AQM_LBCS_FILES, AQM_GEFS_FILE_PREFIX, AQM_GEFS_FILE_CYC, NEXUS_INPUT_DIR, NEXUS_FIX_DIR, + NEXUS_GRID_FN, NUM_SPLIT_NEXUS: 3NEXUS_GFS_SFC_OFFSET_HRS, NEXUS_GFS_SFC_DIR, NEXUS_GFS_SFC_ARCHV_DIR + * - Rocoto + - attrs, cycledefs, entities, log, tasks: taskgroups diff --git a/docs/UsersGuide/source/Quickstart.rst b/docs/UsersGuide/source/BuildingRunningTesting/Quickstart.rst similarity index 71% rename from docs/UsersGuide/source/Quickstart.rst rename to docs/UsersGuide/source/BuildingRunningTesting/Quickstart.rst index 06a5ee8112..df5a61b1ef 100644 --- a/docs/UsersGuide/source/Quickstart.rst +++ b/docs/UsersGuide/source/BuildingRunningTesting/Quickstart.rst @@ -4,7 +4,7 @@ Quick Start Guide ==================== -This chapter provides a brief summary of how to build and run the SRW Application. The steps will run most smoothly on `Level 1 `__ systems. Users should expect to reference other chapters of this User's Guide, particularly :numref:`Chapter %s: Building the SRW App ` and :numref:`Chapter %s: Running the SRW App `, for additional explanations regarding each step. +This chapter provides a brief summary of how to build and run the SRW Application. The steps will run most smoothly on `Level 1 `__ systems. Users should expect to reference other chapters of this User's Guide, particularly :numref:`Section %s: Building the SRW App ` and :numref:`Section %s: Running the SRW App `, for additional explanations regarding each step. Install the HPC-Stack @@ -13,12 +13,15 @@ SRW App users who are not working on a `Level 1 `__ to prepare for the upcoming shift in support from HPC-Stack to spack-stack. + .. _QuickBuildRun: Building and Running the UFS SRW Application =============================================== -For a detailed explanation of how to build and run the SRW App on any supported system, see :numref:`Chapter %s: Building the SRW App ` and :numref:`Chapter %s: Running the SRW App `. :numref:`Figure %s ` outlines the steps of the build process. The overall procedure for generating an experiment is shown in :numref:`Figure %s `, with the scripts to generate and run the workflow shown in red. An overview of the required steps appears below. However, users can expect to access other referenced sections of this User's Guide for more detail. +For a detailed explanation of how to build and run the SRW App on any supported system, see :numref:`Section %s: Building the SRW App ` and :numref:`Section %s: Running the SRW App `. :numref:`Figure %s ` outlines the steps of the build process. The overall procedure for generating an experiment is shown in :numref:`Figure %s `, with the scripts to generate and run the workflow shown in red. An overview of the required steps appears below. However, users can expect to access other referenced sections of this User's Guide for more detail. #. Clone the SRW App from GitHub: @@ -45,12 +48,11 @@ For a detailed explanation of how to build and run the SRW App on any supported #. Users on a `Level 2-4 `__ system must download and stage data (both the fix files and the :term:`IC/LBC ` files) according to the instructions in :numref:`Section %s `. Standard data locations for Level 1 systems appear in :numref:`Table %s `. - #. Load the python environment for the regional workflow. Users on Level 2-4 systems will need to use one of the existing ``wflow_`` modulefiles (e.g., ``wflow_macos``) and adapt it to their system. Then, run: + #. Load the python environment for the workflow. Users on Level 2-4 systems will need to use one of the existing ``wflow_`` modulefiles (e.g., ``wflow_macos``) and adapt it to their system. Then, run: .. code-block:: console - source - module use + module use /path/to/ufs-srweather-app/modulefiles module load wflow_ where ```` refers to a valid machine name (see :numref:`Section %s `). After loading the workflow, users should follow the instructions printed to the console. For example, if the output says: @@ -58,15 +60,14 @@ For a detailed explanation of how to build and run the SRW App on any supported .. code-block:: console Please do the following to activate conda: - > conda activate regional_workflow + > conda activate workflow_tools - then the user should run ``conda activate regional_workflow`` to activate the regional workflow environment. - - .. note:: - If users source the *lmod-setup* file on a system that doesn't need it, it will not cause any problems (it will simply do a ``module purge``). + then the user should run ``conda activate workflow_tools`` to activate the workflow environment. #. Configure the experiment: + Copy the contents of the sample experiment from ``config.community.yaml`` to ``config.yaml``: + .. code-block:: console cd ush @@ -80,7 +81,7 @@ For a detailed explanation of how to build and run the SRW App on any supported ./generate_FV3LAM_wflow.py - #. Run the regional workflow. There are several methods available for this step, which are discussed in :numref:`Section %s `. One possible method is summarized below. It requires the :ref:`Rocoto Workflow Manager `. + #. Run the workflow from the experiment directory (``$EXPTDIR``). By default, the path to this directory is ``${EXPT_BASEDIR}/${EXPT_SUBDIR}`` (see :numref:`Section %s ` for more detail). There are several methods for running the workflow, which are discussed in :numref:`Section %s `. One possible method is summarized below. It requires the :ref:`Rocoto Workflow Manager `. .. code-block:: console @@ -93,6 +94,6 @@ For a detailed explanation of how to build and run the SRW App on any supported ./launch_FV3LAM_wflow.sh; tail -n 40 log.launch_FV3LAM_wflow - The workflow must be relaunched regularly and repeatedly until the log output includes a ``Workflow status: SUCCESS`` message indicating that the experiment has finished. + The workflow must be relaunched regularly and repeatedly until the log output includes a ``Workflow status: SUCCESS`` message indicating that the experiment has finished. The :term:`cron` utility may be used to automate repeated runs. The last section of the log messages from running ``./generate_FV3LAM_wflow.py`` instruct users how to use that functionality. Users may also refer to :numref:`Section %s ` for instructions. -Optionally, users may :ref:`configure their own grid `, instead of using a predefined grid, and/or :ref:`plot the output ` of their experiment(s). +Optionally, users may :ref:`configure their own grid `, instead of using a predefined grid, and/or :ref:`plot the output ` of their experiment(s). diff --git a/docs/UsersGuide/source/BuildingRunningTesting/RunSRW.rst b/docs/UsersGuide/source/BuildingRunningTesting/RunSRW.rst new file mode 100644 index 0000000000..0d45322897 --- /dev/null +++ b/docs/UsersGuide/source/BuildingRunningTesting/RunSRW.rst @@ -0,0 +1,1206 @@ +.. role:: bolditalic + :class: bolditalic + +.. role:: raw-html(raw) + :format: html + +.. _RunSRW: + +=========================== +Running the SRW App +=========================== + +This section explains how to set up and run the basic "out-of-the-box" case for the SRW Application. However, the steps are relevant to any SRW App experiment and can be modified to suit user goals. This chapter assumes that users have already built the SRW App by following the steps in :numref:`Section %s ` (or :numref:`Section %s ` if running the containerized version of the SRW App). + +The out-of-the-box SRW App case builds a weather forecast for June 15-16, 2019. Multiple convective weather events during these two days produced over 200 filtered storm reports. Severe weather was clustered in two areas: the Upper Midwest through the Ohio Valley and the Southern Great Plains. This forecast uses a predefined 25-km Continental United States (:term:`CONUS`) domain (RRFS_CONUS_25km), the Global Forecast System (:term:`GFS`) version 16 physics suite (FV3_GFS_v16 :term:`CCPP`), and :term:`FV3`-based GFS raw external model data for initialization. + +.. attention:: + + The SRW Application has `four levels of support `__. The steps described in this section will work most smoothly on preconfigured (Level 1) systems. They should also work on other systems (including generic Linux/Mac systems), but the user may need to perform additional troubleshooting. + + +The overall procedure for generating an experiment is shown in :numref:`Figure %s `, with the scripts to generate and run the workflow shown in red. Once the SRW App has been built, as described in :numref:`Chapter %s `, the steps to run a forecast are as follows: + + #. :ref:`Download and stage data ` + #. :ref:`Optional: Configure a new grid ` + #. :ref:`Generate an SRW App experiment ` + + * :ref:`Load the workflow environment ` + * :ref:`Set the experiment configuration parameters ` + * :ref:`Optional: Plot the output ` + * :ref:`Optional: Configure METplus Verification Suite ` + + #. :ref:`Run the SRW App workflow ` + +.. _AppOverallProc: + +.. figure:: https://github.com/ufs-community/ufs-srweather-app/wiki/SRW_run_process.png + :alt: Flowchart describing the SRW App workflow steps. + + *Overall Layout of the SRW App Workflow* + +.. _Data: + +Download and Stage the Data +============================ + +The SRW App requires input files to run. These include static datasets, initial and boundary conditions files, and model configuration files. On Level 1 systems, the data required to run SRW App tests are already available in the following locations: + +.. _DataLocations: +.. table:: Data Locations for Level 1 Systems + + +--------------+------------------------------------------------------------------------------+ + | Machine | File location | + +==============+==============================================================================+ + | Cheyenne | /glade/work/epicufsrt/contrib/UFS_SRW_data/develop/input_model_data/ | + +--------------+------------------------------------------------------------------------------+ + | Gaea | /lustre/f2/dev/role.epic/contrib/UFS_SRW_data/develop/input_model_data/ | + +--------------+------------------------------------------------------------------------------+ + | Hera | /scratch1/NCEPDEV/nems/role.epic/UFS_SRW_data/develop/input_model_data/ | + +--------------+------------------------------------------------------------------------------+ + | Jet | /mnt/lfs4/HFIP/hfv3gfs/role.epic/UFS_SRW_data/develop/input_model_data/ | + +--------------+------------------------------------------------------------------------------+ + | NOAA Cloud | /contrib/EPIC/UFS_SRW_data/develop/input_model_data/ | + +--------------+------------------------------------------------------------------------------+ + | Orion | /work/noaa/epic-ps/role-epic-ps/UFS_SRW_data/develop/input_model_data/ | + +--------------+------------------------------------------------------------------------------+ + | WCOSS2 | /lfs/h2/emc/lam/noscrub/UFS_SRW_App/develop/input_model_data/ | + +--------------+------------------------------------------------------------------------------+ + +For Level 2-4 systems, the data must be added to the user's system. Detailed instructions on how to add the data can be found in :numref:`Section %s: Downloading and Staging Input Data `. Sections :numref:`%s ` and :numref:`%s ` contain useful background information on the input and output files used in the SRW App. + +.. _GridSpecificConfig: + +Grid Configuration +======================= + +The SRW App officially supports the four predefined grids shown in :numref:`Table %s `. The out-of-the-box SRW App case uses the ``RRFS_CONUS_25km`` predefined grid option. More information on the predefined and user-generated grid options can be found in :numref:`Section %s: Limited Area Model (LAM) Grids `. Users who plan to utilize one of the four predefined domain (grid) options may continue to the next step (:numref:`Step %s: Generate the Forecast Experiment `). Users who plan to create a new custom predefined grid should refer to the instructions in :numref:`Section %s: Creating User-Generated Grids `. At a minimum, these users will need to add the new grid name to the ``valid_param_vals.yaml`` file and add the corresponding grid-specific parameters in the ``predef_grid_params.yaml`` file. + +.. _PredefinedGrids: + +.. table:: Predefined Grids Supported in the SRW App + + +----------------------+-------------------+--------------------------------+ + | **Grid Name** | **Grid Type** | **Quilting (write component)** | + +======================+===================+================================+ + | RRFS_CONUS_25km | ESG grid | lambert_conformal | + +----------------------+-------------------+--------------------------------+ + | RRFS_CONUS_13km | ESG grid | lambert_conformal | + +----------------------+-------------------+--------------------------------+ + | RRFS_CONUS_3km | ESG grid | lambert_conformal | + +----------------------+-------------------+--------------------------------+ + | SUBCONUS_Ind_3km | ESG grid | lambert_conformal | + +----------------------+-------------------+--------------------------------+ + +.. COMMENT: Revisit before SRW w/RRFS release + +.. _GenerateForecast: + +Generate the Forecast Experiment +================================= +Generating the forecast experiment requires three steps: + +#. :ref:`Load the workflow environment ` +#. :ref:`Set experiment configuration parameters ` +#. :ref:`Run a script to generate the experiment workflow ` + +The first two steps depend on the platform being used and are described here for each Level 1 platform. Users will need to adjust the instructions to reflect their machine's configuration if they are working on a Level 2-4 platform. Information in :numref:`Section %s: Configuring the Workflow ` can help with this. + +.. _SetUpPythonEnv: + +Load the Conda/Python Environment +------------------------------------ + +The SRW App workflow is often referred to as the *regional workflow* because it runs experiments on a regional scale (unlike the *global workflow* used in other applications). The SRW App workflow requires installation of Python3 using conda; it also requires additional packages built in a separate conda evironment named ``workflow_tools``. On Level 1 systems, a ``workflow_tools`` environment already exists, and users merely need to load the environment. On Level 2-4 systems, users must create and then load the environment. The process for each is described in detail below. + +.. _Load-WF-L1: + +Loading the Workflow Environment on Level 1 Systems +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +.. attention:: + + Users on a Level 2-4 system should skip to the :ref:`next section ` for instructions. + +The ``workflow_tools`` conda/Python environment has already been set up on Level 1 platforms and can be activated in the following way: + +.. code-block:: console + + source /path/to/etc/lmod-setup.sh + module use /path/to/modulefiles + module load wflow_ + +where ```` refers to a valid machine name (see :numref:`Section %s ` for ``MACHINE`` options). In a csh shell environment, users should replace ``lmod-setup.sh`` with ``lmod-setup.csh``. + +.. note:: + If users source the lmod-setup file on a system that doesn't need it, it will not cause any problems (it will simply do a ``module purge``). + +The ``wflow_`` modulefile will then output instructions to activate the SRW App workflow. The user should run the commands specified in the modulefile output. The command may vary from system to system. For example, if the output says: + +.. code-block:: console + + Please do the following to activate conda: + > conda activate workflow_tools + +then the user should run ``conda activate workflow_tools``. This activates the ``workflow_tools`` conda environment, and the user typically sees ``(workflow_tools)`` in front of the Terminal prompt at this point. + +After loading the workflow environment, users may continue to :numref:`Section %s ` for instructions on setting the experiment configuration parameters. + +.. _Load-WF-L234: + +Loading the Workflow Environment on Level 2-4 Systems +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Users on non-Level 1 systems will need to create a conda workflow environment, modify a ``wflow_*`` file to reflect the location of required modules, and load the workflow modules using the modified ``wflow_*`` file. + +Create a *conda* Workflow Environment +``````````````````````````````````````` + +.. note:: + Examples in this subsection presume that the user is running in the Terminal with a bash shell environment. If this is not the case, users will need to adjust the commands to fit their command line application and shell environment. + +.. _MacMorePackages: + +MacOS ONLY: Install/Upgrade Mac-Specific Packages +""""""""""""""""""""""""""""""""""""""""""""""""""" + +.. attention:: + + This subsection is for Mac OS users only. Users on Linux systems can skip to :ref:`Creating the workflow_tools Environment on Linux and Mac OS ` for instructions. + + +MacOS requires the installation of a few additional packages and, possibly, an upgrade to bash. Users running on MacOS should execute the following commands: + +.. code-block:: console + + bash --version + brew install bash # or: brew upgrade bash + brew install coreutils + brew gsed # follow directions to update the PATH env variable + + +.. _LinuxMacVEnv: + +Creating the ``workflow_tools`` Environment on Linux and Mac OS +""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" + +On generic Mac and Linux systems, users need to create a conda ``workflow_tools`` environment. The environment can be stored in a local path, which could be a default location or a user-specified location (e.g., ``$HOME/condaenv/venvs/`` directory). (To determine the default location, use the ``conda info`` command, and look for the ``envs directories`` list.) The following is a brief recipe for creating a virtual conda environment on non-Level 1 platforms. It uses the aarch64 (64-bit ARM) Miniforge for Linux and installs into $HOME/conda. Adjust as necessary for your target system. + +.. code-block:: console + + wget https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-Linux-aarch64.sh + bash Miniforge3-Linux-aarch64.sh -bfp ~/conda + rm Miniforge3-Linux-aarch64.sh + source ~/conda/etc/profile.d/conda.sh + conda activate + conda install -y conda-build conda-verify + cd path/to/your/workflow-tools/clone + conda build recipe + conda create -y -n workflow_tools -c local workflow_tools + conda activate workflow_tools + +In future shells, you can activate and use this environment with: + +.. code-block:: console + + source ~/conda/etc/profile.d/conda.sh + conda activate uwtools + +See the `workflow-tools respository `__ for additional documentation. + +Modify a ``wflow_`` File +`````````````````````````````````````` + +Users can copy one of the provided ``wflow_`` files from the ``modulefiles`` directory and use it as a template to create a ``wflow_`` file that functions on their system. The ``wflow_macos`` and ``wflow_linux`` template modulefiles are provided as a starting point, but any ``wflow_`` file could be used. Users must modify the files to provide paths for python, miniconda modules, module loads, conda initialization, and the user's ``workflow_tools`` conda environment. + +Load the Workflow Environment +``````````````````````````````` + +After creating a ``workflow_tools`` environment and making modifications to a ``wflow_`` file, users can run the commands below to activate the workflow environment: + +.. code-block:: console + + source /path/to/etc/lmod-setup.sh + module use /path/to/modulefiles + module load wflow_ + +where ```` refers to a valid machine name (i.e., ``linux`` or ``macos``). + +.. note:: + If users source the lmod-setup file on a system that doesn't need it, it will not cause any problems (it will simply do a ``module purge``). + +The ``wflow_`` modulefile will then output the following instructions: + +.. code-block:: console + + Please do the following to activate conda: + > conda activate workflow_tools + +After running ``conda activate workflow_tools``, the user will typically see ``(workflow_tools)`` in front of the Terminal prompt. This indicates that the workflow environment has been loaded successfully. + +.. note:: + ``conda`` needs to be initialized before running ``conda activate workflow_tools`` command. Depending on the user's system and login setup, this may be accomplished in a variety of ways. Conda initialization usually involves the following command: ``source /etc/profile.d/conda.sh``, where ```` is the base conda installation directory. + +.. _ExptConfig: + +Set Experiment Configuration Parameters +------------------------------------------ + +Each experiment requires certain basic information to run (e.g., date, grid, physics suite). Default values are assigned in ``config_defaults.yaml``, and users adjust the desired variables in the experiment configuration file named ``config.yaml``. When generating a new experiment, the SRW App first reads and assigns default values from ``config_defaults.yaml``. Then, it reads and (re)assigns variables from the user's custom ``config.yaml`` file. + +.. _DefaultConfigSection: + +Default configuration: ``config_defaults.yaml`` +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +In general, ``config_defaults.yaml`` is split into sections by category (e.g., ``user:``, ``platform:``, ``workflow:``, ``task_make_grid:``). Users can view a full list of categories and configuration parameters in the :doc:`Table of Variables in config_defaults.yaml `. Definitions and default values of each of the variables can be found in the ``config_defaults.yaml`` file comments and in :numref:`Section %s: Workflow Parameters `. Some of these default values are intentionally invalid in order to ensure that the user assigns valid values in their ``config.yaml`` file. There is usually no need for a user to modify ``config_defaults.yaml`` because any settings provided in ``config.yaml`` will override the settings in ``config_defaults.yaml``. + +.. _UserSpecificConfig: + +User-specific configuration: ``config.yaml`` +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +The user must set the specifics of their experiment configuration in a ``config.yaml`` file located in the ``ufs-srweather-app/ush`` directory. Two example templates are provided in that directory: ``config.community.yaml`` and ``config.nco.yaml``. The first file is a basic example for creating and running an experiment in *community* mode (with ``RUN_ENVIR`` set to ``community``). The second is an example for creating and running an experiment in the *NCO* (operational) mode (with ``RUN_ENVIR`` set to ``nco``). The *community* mode is recommended in most cases, and user support is available for running in community mode. The operational/NCO mode is typically used by developers at the Environmental Modeling Center (:term:`EMC`) and the Global Systems Laboratory (:term:`GSL`) who are working on pre-implementation testing for the Rapid Refresh Forecast System (:term:`RRFS`). :numref:`Table %s ` compares the configuration variables that appear in the ``config.community.yaml`` with their default values in ``config_default.yaml``. + +.. _ConfigCommunity: + +.. table:: Configuration variables specified in the config.community.yaml script + + +--------------------------------+-------------------+------------------------------------+ + | **Parameter** | **Default Value** | **config.community.yaml Value** | + +================================+===================+====================================+ + | RUN_ENVIR | "nco" | "community" | + +--------------------------------+-------------------+------------------------------------+ + | MACHINE | "BIG_COMPUTER" | "hera" | + +--------------------------------+-------------------+------------------------------------+ + | ACCOUNT | "" | "an_account" | + +--------------------------------+-------------------+------------------------------------+ + | CCPA_OBS_DIR | "" | "" | + +--------------------------------+-------------------+------------------------------------+ + | NOHRSC_OBS_DIR | "" | "" | + +--------------------------------+-------------------+------------------------------------+ + | MRMS_OBS_DIR | "" | "" | + +--------------------------------+-------------------+------------------------------------+ + | NDAS_OBS_DIR | "" | "" | + +--------------------------------+-------------------+------------------------------------+ + | USE_CRON_TO_RELAUNCH | false | false | + +--------------------------------+-------------------+------------------------------------+ + | EXPT_SUBDIR | "" | "test_community" | + +--------------------------------+-------------------+------------------------------------+ + | CCPP_PHYS_SUITE | "FV3_GFS_v16" | "FV3_GFS_v16" | + +--------------------------------+-------------------+------------------------------------+ + | PREDEF_GRID_NAME | "" | "RRFS_CONUS_25km" | + +--------------------------------+-------------------+------------------------------------+ + | DATE_FIRST_CYCL | "YYYYMMDDHH" | '2019061518' | + +--------------------------------+-------------------+------------------------------------+ + | DATE_LAST_CYCL | "YYYYMMDDHH" | '2019061518' | + +--------------------------------+-------------------+------------------------------------+ + | FCST_LEN_HRS | 24 | 12 | + +--------------------------------+-------------------+------------------------------------+ + | PREEXISTING_DIR_METHOD | "delete" | "rename" | + +--------------------------------+-------------------+------------------------------------+ + | VERBOSE | true | true | + +--------------------------------+-------------------+------------------------------------+ + | COMPILER | "intel" | "intel" | + +--------------------------------+-------------------+------------------------------------+ + | EXTRN_MDL_NAME_ICS | "FV3GFS" | "FV3GFS" | + +--------------------------------+-------------------+------------------------------------+ + | FV3GFS_FILE_FMT_ICS | "nemsio" | "grib2" | + +--------------------------------+-------------------+------------------------------------+ + | EXTRN_MDL_NAME_LBCS | "FV3GFS" | "FV3GFS" | + +--------------------------------+-------------------+------------------------------------+ + | LBC_SPEC_INTVL_HRS | 6 | 6 | + +--------------------------------+-------------------+------------------------------------+ + | FV3GFS_FILE_FMT_LBCS | "nemsio" | "grib2" | + +--------------------------------+-------------------+------------------------------------+ + | QUILTING | true | true | + +--------------------------------+-------------------+------------------------------------+ + | COMOUT_REF | "" | "" | + +--------------------------------+-------------------+------------------------------------+ + | DO_ENSEMBLE | false | false | + +--------------------------------+-------------------+------------------------------------+ + | NUM_ENS_MEMBERS | 1 | 2 | + +--------------------------------+-------------------+------------------------------------+ + +.. _GeneralConfig: + +General Instructions for All Systems +``````````````````````````````````````` + +To get started with a basic forecast in *community* mode, make a copy of ``config.community.yaml``. From the ``ufs-srweather-app`` directory, run: + +.. code-block:: console + + cd ush + cp config.community.yaml config.yaml + +The default settings in this file include a predefined 25-km :term:`CONUS` grid (RRFS_CONUS_25km), the :term:`GFS` v16 physics suite (FV3_GFS_v16 :term:`CCPP`), and :term:`FV3`-based GFS raw external model data for initialization. + +Next, users should edit the new ``config.yaml`` file to customize it for their machine. On most systems, the following fields need to be updated or added to the appropriate section of the ``config.yaml`` file in order to run the out-of-the-box SRW App case: + +.. code-block:: console + + user: + MACHINE: hera + ACCOUNT: an_account + workflow: + EXPT_SUBDIR: test_community + task_get_extrn_ics: + USE_USER_STAGED_EXTRN_FILES: true + EXTRN_MDL_SOURCE_BASEDIR_ICS: "/path/to/UFS_SRW_App/develop/input_model_data///" + task_get_extrn_lbcs: + USE_USER_STAGED_EXTRN_FILES: true + EXTRN_MDL_SOURCE_BASEDIR_LBCS: "/path/to/UFS_SRW_App/develop/input_model_data///" + +where: + * ``MACHINE`` refers to a valid machine name (see :numref:`Section %s ` for options). + * ``ACCOUNT`` refers to a valid account name. Not all systems require a valid account name, but most Level 1 & 2 systems do. + + .. hint:: + + To determine an appropriate ACCOUNT field for Level 1 systems, run ``groups``, and it will return a list of projects you have permissions for. Not all of the listed projects/groups have an HPC allocation, but those that do are potentially valid account names. + + * ``EXPT_SUBDIR`` is changed to an experiment name of the user's choice. + * ``/path/to/`` is the path to the SRW App data on the user's machine (see :numref:`Section %s ` for data locations on Level 1 systems). + * ```` refers to a subdirectory containing the experiment data from a particular model. Valid values on Level 1 systems correspond to the valid values for ``EXTRN_MDL_NAME_ICS`` and ``EXTRN_MDL_NAME_LBCS`` (see :numref:`Section %s ` or :numref:`%s ` for options). + * ```` refers to one of 3 possible data formats: ``grib2``, ``nemsio``, or ``netcdf``. + * ```` refers to a subdirectory containing data for the :term:`cycle` date (in YYYYMMDDHH format). + +On platforms where Rocoto and :term:`cron` are available, users can automate resubmission of their experiment workflow by adding the following lines to the ``workflow:`` section of the ``config.yaml`` file: + +.. code-block:: console + + USE_CRON_TO_RELAUNCH: true + CRON_RELAUNCH_INTVL_MNTS: 3 + +.. note:: + + On Orion, *cron* is only available on the orion-login-1 node, so users will need to work on that node when running *cron* jobs on Orion. + +When running with GNU compilers (i.e., if the modulefile used to set up the build environment in :numref:`Section %s ` uses a GNU compiler), users must also set ``COMPILER: "gnu"`` in the ``workflow:`` section of the ``config.yaml`` file. + +.. note:: + + On ``JET``, users should add ``PARTITION_DEFAULT: xjet`` and ``PARTITION_FCST: xjet`` to the ``platform:`` section of the ``config.yaml`` file. + +For example, to run the out-of-the-box experiment on Gaea using cron to automate job submission, users can add or modify variables in the ``user``, ``workflow``, ``task_get_extrn_ics``, and ``task_get_extrn_lbcs`` sections of ``config.yaml`` according to the following example (unmodified variables are not shown here): + + .. code-block:: + + user: + MACHINE: gaea + ACCOUNT: hfv3gfs + workflow: + EXPT_SUBDIR: run_basic_srw + USE_CRON_TO_RELAUNCH: true + CRON_RELAUNCH_INTVL_MNTS: 3 + task_get_extrn_ics: + USE_USER_STAGED_EXTRN_FILES: true + EXTRN_MDL_SOURCE_BASEDIR_ICS: /lustre/f2/dev/role.epic/contrib/UFS_SRW_data/develop/input_model_data/FV3GFS/grib2/2019061518 + task_get_extrn_lbcs: + USE_USER_STAGED_EXTRN_FILES: true + EXTRN_MDL_SOURCE_BASEDIR_LBCS: /lustre/f2/dev/role.epic/contrib/UFS_SRW_data/develop/input_model_data/FV3GFS/grib2/2019061518 + +To determine whether the ``config.yaml`` file adjustments are valid, users can run the following script from the ``ush`` directory: + +.. code-block:: console + + ./config_utils.py -c config.yaml -v config_defaults.yaml -k "(?\!rocoto\b)" + +A correct ``config.yaml`` file will output a ``SUCCESS`` message. A ``config.yaml`` file with problems will output a ``FAILURE`` message describing the problem. For example: + +.. code-block:: console + + INVALID ENTRY: EXTRN_MDL_FILES_ICS=[] + FAILURE + +.. hint:: + + * The ``workflow_tools`` environment must be loaded for the ``config_utils.py`` script to validate the ``config.yaml`` file. + + * Valid values for configuration variables should be consistent with those in the ``ush/valid_param_vals.yaml`` script. + + * Various sample configuration files can be found within the subdirectories of ``tests/WE2E/test_configs``. + + * Users can find detailed information on configuration parameter options in :numref:`Section %s: Configuring the Workflow `. + +**Next Steps:** + + * To configure an experiment for a general Linux or Mac system, see the :ref:`next section ` for additional required steps. + * To add the graphics plotting tasks to the experiment workflow, go to section :numref:`Section %s: Plotting Configuration `. + * To configure an experiment to run METplus verification tasks, see :numref:`Section %s `. + * Otherwise, skip to :numref:`Section %s ` to generate the workflow. + +.. _LinuxMacExptConfig: + +Configuring an Experiment on General Linux and MacOS Systems +`````````````````````````````````````````````````````````````` + +.. note:: + Examples in this subsection presume that the user is running in the Terminal with a bash shell environment. If this is not the case, users will need to adjust the commands to fit their command line application and shell environment. + +**Optional: Install Rocoto** + +.. note:: + Users may `install Rocoto `__ if they want to make use of a workflow manager to run their experiments. However, this option has not yet been tested on MacOS and has had limited testing on general Linux plaforms. + + +**Configure the SRW App:** + +After following the steps in :numref:`Section %s: General Configuration ` above, users should have a ``config.yaml`` file with settings from ``community.config.yaml`` and updates similar to this: + +.. code-block:: console + + user: + MACHINE: macos + ACCOUNT: user + workflow: + EXPT_SUBDIR: my_test_expt + COMPILER: gnu + task_get_extrn_ics: + USE_USER_STAGED_EXTRN_FILES: true + EXTRN_MDL_SOURCE_BASEDIR_ICS: /path/to/input_model_data/FV3GFS/grib2/2019061518 + task_get_extrn_lbcs: + USE_USER_STAGED_EXTRN_FILES: true + EXTRN_MDL_SOURCE_BASEDIR_LBCS: /path/to/input_model_data/FV3GFS/grib2/2019061518 + +Due to the limited number of processors on MacOS systems, users must also configure the domain decomposition parameters directly in the section of the ``predef_grid_params.yaml`` file pertaining to the grid they want to use. Domain decomposition needs to take into account the number of available CPUs and configure the variables ``LAYOUT_X``, ``LAYOUT_Y``, and ``WRTCMP_write_tasks_per_group`` accordingly. + +The example below is for systems with 8 CPUs: + +.. code-block:: console + + task_run_fcst: + LAYOUT_X: 3 + LAYOUT_Y: 2 + WRTCMP_write_tasks_per_group: 2 + +.. note:: + The number of MPI processes required by the forecast will be equal to ``LAYOUT_X`` * ``LAYOUT_Y`` + ``WRTCMP_write_tasks_per_group``. + +For a machine with 4 CPUs, the following domain decomposition could be used: + +.. code-block:: console + + task_run_fcst: + LAYOUT_X: 3 + LAYOUT_Y: 1 + WRTCMP_write_tasks_per_group: 1 + +**Configure the Machine File** + +Configure a ``macos.yaml`` or ``linux.yaml`` machine file in ``ufs-srweather-app/ush/machine`` based on the number of CPUs (``NCORES_PER_NODE``) in the system (usually 8 or 4 in MacOS; varies on Linux systems). Job scheduler (``SCHED``) options can be viewed :ref:`here `. Users must also set the path to the fix file directories. + +.. code-block:: console + + platform: + # Architecture information + WORKFLOW_MANAGER: none + NCORES_PER_NODE: 8 + SCHED: none + # Run commands for executables + RUN_CMD_FCST: 'mpirun -np ${PE_MEMBER01}' + RUN_CMD_POST: 'mpirun -np 4' + RUN_CMD_SERIAL: time + RUN_CMD_UTILS: 'mpirun -np 4' + # Commands to run at the start of each workflow task. + PRE_TASK_CMDS: '{ ulimit -a; }' + + task_make_orog: + # Path to location of static input files used by the make_orog task + FIXorg: path/to/FIXorg/files + + task_make_sfc_climo: + # Path to location of static surface climatology input fields used by sfc_climo_gen + FIXsfc: path/to/FIXsfc/files + + task_run_fcst: + FIXaer: /path/to/FIXaer/files + FIXgsm: /path/to/FIXgsm/files + FIXlut: /path/to/FIXlut/files + + data: + # Used by setup.py to set the values of EXTRN_MDL_SOURCE_BASEDIR_ICS and EXTRN_MDL_SOURCE_BASEDIR_LBCS + FV3GFS: /Users/username/DATA/UFS/FV3GFS + +The ``data:`` section of the machine file can point to various data sources that the user has pre-staged on disk. For example: + +.. code-block:: console + + data: + FV3GFS: + nemsio: /Users/username/DATA/UFS/FV3GFS/nemsio + grib2: /Users/username/DATA/UFS/FV3GFS/grib2 + netcdf: /Users/username/DATA/UFS/FV3GFS/netcdf + RAP: /Users/username/DATA/UFS/RAP/grib2 + HRRR: /Users/username/DATA/UFS/HRRR/grib2 + +This can be helpful when conducting multiple experiments with different types of data. + +**Next Steps:** + + * To add the graphics plotting tasks to the experiment workflow, go to the next section :ref:`Plotting Configuration `. + * To configure an experiment to run METplus verification tasks, see :numref:`Section %s `. + * Otherwise, skip to :numref:`Section %s ` to generate the workflow. + +.. _PlotOutput: + +Plotting Configuration (optional) +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +An optional Python plotting task (PLOT_ALLVARS) can be activated in the workflow to generate plots for the :term:`FV3`-:term:`LAM` post-processed :term:`GRIB2` +output over the :term:`CONUS`. It generates graphics plots for a number of variables, including: + + * 2-m temperature + * 2-m dew point temperature + * 10-m winds + * 250 hPa winds + * Accumulated precipitation + * Composite reflectivity + * Surface-based :term:`CAPE`/:term:`CIN` + * Max/Min 2-5 km updraft helicity + * Sea level pressure (SLP) + +.. COMMENT: * 500 hPa heights, winds, and vorticity --> seems to be omitted? Why? + +This workflow task can produce both plots from a single experiment and difference plots that compare the same cycle from two experiments. When plotting the difference, the two experiments must be on the same domain and available for +the same cycle starting date/time and forecast hours. Other parameters may differ (e.g., the experiments may use different physics suites). + +.. _Cartopy: + +Cartopy Shapefiles +````````````````````` + +The Python plotting tasks require a path to the directory where the Cartopy Natural Earth shapefiles are located. The medium scale (1:50m) cultural and physical shapefiles are used to create coastlines and other geopolitical borders on the map. On `Level 1 `__ systems, this path is already set in the system's machine file using the variable ``FIXshp``. Users on other systems will need to download the shapefiles and update the path of ``$FIXshp`` in the machine file they are using (e.g., ``$SRW/ush/machine/macos.yaml`` for a generic MacOS system, where ``$SRW`` is the path to the ``ufs-srweather-app`` directory). The subset of shapefiles required for the plotting task can be obtained from the `SRW Data Bucket `__. The full set of medium-scale (1:50m) Cartopy shapefiles can be downloaded `here `__. + +Task Configuration +````````````````````` + +Users will need to add or modify certain variables in ``config.yaml`` to run the plotting task(s). At a minimum, to activate the ``plot_allvars`` tasks, users must add the task yaml file to the default list of ``taskgroups`` under the ``rocoto: tasks:`` section. + +.. code-block:: console + + rocoto: + tasks: + taskgroups: '{{ ["parm/wflow/prep.yaml", "parm/wflow/coldstart.yaml", "parm/wflow/post.yaml", "parm/wflow/plot.yaml"]|include }}' + +Users may also wish to adjust the start, end, and increment value for the plotting task in the ``config.yaml`` file. For example: + +.. code-block:: console + + task_plot_allvars: + PLOT_FCST_START: 0 + PLOT_FCST_INC: 6 + PLOT_FCST_END: 12 + +If the user chooses not to set these values, the default values will be used (see :numref:`Section %s ` for defaults). + +.. note:: + If a forecast starts at 18 UTC, this is considered the 0th forecast hour, so "starting forecast hour" should be 0, not 18. + +When plotting output from a single experiment, no further adjustments are necessary. The output files (in ``.png`` format) will be located in the experiment directory under the ``$CDATE/postprd`` subdirectory where ``$CDATE`` +corresponds to the cycle date and hour in YYYYMMDDHH format (e.g., ``2019061518``). + +Plotting the Difference Between Two Experiments +"""""""""""""""""""""""""""""""""""""""""""""""""" + +When plotting the difference between two experiments (``expt1`` and ``expt2``), users must set the ``COMOUT_REF`` template variable in ``expt2``'s ``config.yaml`` file to point at forecast output from the ``expt1`` directory. For example, in *community* mode, users can set ``COMOUT_REF`` as follows in the ``expt2`` configuration file: + +.. code-block:: console + + task_plot_allvars: + COMOUT_REF: '${EXPT_BASEDIR}/expt1/${PDY}${cyc}/postprd' + +This will ensure that ``expt2`` can produce a difference plot comparing ``expt1`` and ``expt2``. In *community* mode, using default directory names and settings, ``$COMOUT_REF`` will resemble ``/path/to/expt_dirs/test_community/2019061518/postprd``. Additional details on the plotting variables are provided in :numref:`Section %s `. + +The output files (in ``.png`` format) will be located in the ``postprd`` directory for the experiment. + +**Next Steps:** + + * To configure an experiment to run METplus verification tasks, see the :ref:`next section `. + * Otherwise, skip to :numref:`Section %s ` to generate the workflow. + + +.. _VXConfig: + +Configure METplus Verification Suite (Optional) +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Users who want to use the METplus verification suite to evaluate their forecasts need to add additional information to their machine file (``ush/machine/.yaml``) or their ``config.yaml`` file. Other users may skip to the next step (:numref:`Section %s: Generate the SRW App Workflow `). + +.. note:: + If METplus users update their METplus installation, they must update the module load statements in ``ufs-srweather-app/modulefiles/tasks//run_vx.local`` to correspond to their system's updated installation: + + .. code-block:: console + + module use -a /path/to/met/modulefiles + module load met/ + module load metplus/ + +To use METplus verification, MET and METplus modules need to be installed. To turn on verification tasks in the workflow, include the ``parm/wflow/verify_*.yaml`` file(s) in the ``rocoto: tasks: taskgroups:`` section of ``config.yaml``. For example: + +.. code-block:: console + + rocoto: + tasks: + taskgroups: '{{ ["parm/wflow/prep.yaml", "parm/wflow/coldstart.yaml", "parm/wflow/post.yaml", "parm/wflow/verify_pre.yaml", "parm/wflow/verify_det.yaml"]|include }}' + +:numref:`Table %s ` indicates which functions each ``verify_*.yaml`` file configures. Users must add ``verify_pre.yaml`` anytime they want to do VX; it runs preprocessing tasks that are necessary for both deterministic and ensemble VX. Then users can add ``verify_det.yaml`` for deterministic VX or ``verify_ens.yaml`` for ensemble VX (or both). Note that ensemble VX requires the user to be running an ensemble forecast or to stage ensemble forecast files in an appropriate location. + +.. _VX-yamls: + +.. list-table:: Verification YAML Task Groupings + :widths: 20 50 + :header-rows: 1 + + * - File + - Description + * - verify_pre.yaml + - Contains (meta)tasks that are prerequisites for both deterministic and ensemble verification (vx) + * - verify_det.yaml + - Perform deterministic vx + * - verify_ens.yaml + - Perform ensemble vx (must set ``DO_ENSEMBLE: true`` in ``config.yaml``) + +The ``verify_*.yaml`` files include the definitions of several common verification tasks by default. Individual verification tasks appear in :numref:`Table %s `. The tasks in the ``verify_*.yaml`` files are independent of each other, so users may want to turn some off depending on the needs of their experiment. To turn off a task, simply include its entry from ``verify_*.yaml`` as an empty YAML entry in ``config.yaml``. For example, to turn off PointStat tasks: + +.. code-block:: console + + rocoto: + tasks: + taskgroups: '{{ ["parm/wflow/prep.yaml", "parm/wflow/coldstart.yaml", "parm/wflow/post.yaml", "parm/wflow/verify_pre.yaml", "parm/wflow/verify_det.yaml"]|include }}' + metatask_vx_ens_member: + metatask_PointStat_mem#mem#: + + +More information about configuring the ``rocoto:`` section can be found in :numref:`Section %s `. + +If users have access to NOAA :term:`HPSS` but have not pre-staged the data, the default ``verify_pre.yaml`` taskgroup will activate the tasks, and the workflow will attempt to download the appropriate data from NOAA HPSS. In this case, the ``*_OBS_DIR`` paths must be set to the location where users want the downloaded data to reside. + +Users who do not have access to NOAA HPSS and do not have the data on their system will need to download :term:`CCPA`, :term:`MRMS`, and :term:`NDAS` data manually from collections of publicly available data, such as the ones listed `here `__. + +Users who have already staged the observation data needed for METplus (i.e., the :term:`CCPA`, :term:`MRMS`, and :term:`NDAS` data) on their system should set the path to this data in ``config.yaml``. + +.. code-block:: console + + platform: + CCPA_OBS_DIR: /path/to/UFS_SRW_App/develop/obs_data/ccpa/proc + NOHRSC_OBS_DIR: /path/to/UFS_SRW_App/develop/obs_data/nohrsc/proc + MRMS_OBS_DIR: /path/to/UFS_SRW_App/develop/obs_data/mrms/proc + NDAS_OBS_DIR: /path/to/UFS_SRW_App/develop/obs_data/ndas/proc + +.. _GenerateWorkflow: + +Generate the SRW App Workflow +-------------------------------- + +Run the following command from the ``ufs-srweather-app/ush`` directory to generate the workflow: + +.. code-block:: console + + ./generate_FV3LAM_wflow.py + +The last line of output from this script, starting with ``*/1 * * * *`` or ``*/3 * * * *``, can be saved and used later to automatically run portions of the workflow if users have the Rocoto workflow manager installed on their system. + +This workflow generation script creates an experiment directory and populates it with all the data needed to run through the workflow. The flowchart in :numref:`Figure %s ` describes the experiment generation process. The ``generate_FV3LAM_wflow.py``: + + #. Runs the ``setup.py`` script to set the configuration parameters. This script reads three other configuration scripts in order: + + a. ``config_defaults.yaml`` (:numref:`Section %s `) + b. ``config.yaml`` (:numref:`Section %s `), and + c. ``set_predef_grid_params.py``. + + #. Symlinks the time-independent (fix) files and other necessary data input files from their location to the experiment directory (``$EXPTDIR``). + #. Creates the input namelist file ``input.nml`` based on the ``input.nml.FV3`` file in the ``parm`` directory. + #. Creates the workflow XML file ``FV3LAM_wflow.xml`` that is executed when running the experiment with the Rocoto workflow manager. + +The generated workflow will appear in ``$EXPTDIR``, where ``EXPTDIR=${EXPT_BASEDIR}/${EXPT_SUBDIR}``. These variables were specified in ``config_defaults.yaml`` and ``config.yaml`` in :numref:`Step %s `. The settings for these paths can also be viewed in the console output from the ``./generate_FV3LAM_wflow.py`` script or in the ``log.generate_FV3LAM_wflow`` file, which can be found in ``$EXPTDIR``. + +.. _WorkflowGeneration: + +.. figure:: https://github.com/ufs-community/ufs-srweather-app/wiki/SRW_regional_workflow_gen.png + :alt: Flowchart of the workflow generation process. Scripts are called in the following order: source_util_funcs.sh (which calls bash_utils), then set_FV3nml_sfc_climo_filenames.py, set_FV3nml_ens_stoch_seeds.py, create_diag_table_file.py, and setup.py. setup.py calls several scripts: set_cycle_dates.py, set_grid_params_GFDLgrid.py, set_grid_params_ESGgrid.py, link_fix.py, set_ozone_param.py, set_thompson_mp_fix_files.py, config_defaults.yaml, config.yaml, and valid_param_vals.yaml. Then, it sets a number of variables, including FIXgsm, TOPO_DIR, and SFC_CLIMO_INPUT_DIR variables. Next, set_predef_grid_params.py is called, and the FIXam and FIXLAM directories are set, along with the forecast input files. The setup script also calls set_extrn_mdl_params.py, sets the GRID_GEN_METHOD with HALO, checks various parameters, and generates shell scripts. Then, the workflow generation script produces a YAML configuration file and generates the actual Rocoto workflow XML file from the template file (by calling uwtools set_template). The workflow generation script checks the crontab file and, if applicable, copies certain fix files to the experiment directory. Then, it copies templates of various input files to the experiment directory and sets parameters for the input.nml file. Finally, it generates the workflow. Additional information on each step appears in comments within each script. + + *Experiment Generation Description* + +.. _WorkflowTaskDescription: + +Description of Workflow Tasks +-------------------------------- + +.. note:: + This section gives a general overview of workflow tasks. To begin running the workflow, skip to :numref:`Step %s ` + +:numref:`Figure %s ` illustrates the overall workflow. Individual tasks that make up the workflow are detailed in the ``FV3LAM_wflow.xml`` file. :numref:`Table %s ` describes the function of each baseline task. The first three pre-processing tasks; ``MAKE_GRID``, ``MAKE_OROG``, and ``MAKE_SFC_CLIMO``; are optional. If the user stages pre-generated grid, orography, and surface climatology fix files, these three tasks can be skipped by removing the ``prep.yaml`` file from the default ``taskgroups`` entry in the ``config.yaml`` file before running the ``generate_FV3LAM_wflow.py`` script: + +.. code-block:: console + + rocoto: + tasks: + taskgroups: '{{ ["parm/wflow/coldstart.yaml", "parm/wflow/post.yaml"]|include }}' + +.. _WorkflowTasksFig: + +.. figure:: https://github.com/ufs-community/ufs-srweather-app/wiki/SRW_wflow_flowchart.png + :alt: Flowchart of the default workflow tasks. If the make_grid, make_orog, and make_sfc_climo tasks are toggled off, they will not be run. If toggled on, make_grid, make_orog, and make_sfc_climo will run consecutively by calling the corresponding exregional script in the scripts directory. The get_ics, get_lbcs, make_ics, make_lbcs, and run_fcst tasks call their respective exregional scripts. The run_post task will run, and if METplus verification tasks have been configured, those will run during post-processing by calling their exregional scripts. + + *Flowchart of the Default Workflow Tasks* + + +The ``FV3LAM_wflow.xml`` file runs the specific j-job scripts (``jobs/JREGIONAL_[task name]``) in the prescribed order when the experiment is launched via the ``launch_FV3LAM_wflow.sh`` script or the ``rocotorun`` command. Each j-job task has its own source script (or "ex-script") named ``exregional_[task name].sh`` in the ``scripts`` directory. Two database files named ``FV3LAM_wflow.db`` and ``FV3LAM_wflow_lock.db`` are generated and updated by the Rocoto calls. There is usually no need for users to modify these files. To relaunch the workflow from scratch, delete these two ``*.db`` files and then call the launch script repeatedly for each task. + + +.. _WorkflowTasksTable: + +.. table:: Baseline Workflow Tasks in the SRW App + + +----------------------+------------------------------------------------------------+ + | **Workflow Task** | **Task Description** | + +======================+============================================================+ + | make_grid | Pre-processing task to generate regional grid files. Only | + | | needs to be run once per experiment. | + +----------------------+------------------------------------------------------------+ + | make_orog | Pre-processing task to generate orography files. Only | + | | needs to be run once per experiment. | + +----------------------+------------------------------------------------------------+ + | make_sfc_climo | Pre-processing task to generate surface climatology files. | + | | Only needs to be run once per experiment. | + +----------------------+------------------------------------------------------------+ + | get_extrn_ics | Cycle-specific task to obtain external data for the | + | | initial conditions (ICs) | + +----------------------+------------------------------------------------------------+ + | get_extrn_lbcs | Cycle-specific task to obtain external data for the | + | | lateral boundary conditions (LBCs) | + +----------------------+------------------------------------------------------------+ + | make_ics_* | Generate ICs from the external data | + +----------------------+------------------------------------------------------------+ + | make_lbcs_* | Generate LBCs from the external data | + +----------------------+------------------------------------------------------------+ + | run_fcst_* | Run the forecast model (UFS Weather Model) | + +----------------------+------------------------------------------------------------+ + | run_post_* | Run the post-processing tool (UPP) | + +----------------------+------------------------------------------------------------+ + +In addition to the baseline tasks described in :numref:`Table %s ` above, users may choose to run a variety of optional tasks, including plotting and verification tasks. + +.. _PlottingTaskTable: + +.. table:: Plotting Task in the SRW App + + +----------------------+------------------------------------------------------------+ + | **Workflow Task** | **Task Description** | + +======================+============================================================+ + | plot_allvars | Run the plotting task and, optionally, the difference | + | | plotting task | + +----------------------+------------------------------------------------------------+ + +METplus verification tasks are described in :numref:`Table %s ` below. The column "taskgroup" indicates the taskgroup file that must be included in the user's ``config.yaml`` file under ``rocoto: tasks: taskgroups:`` (see :numref:`Section %s ` for more details). For each task, ``mem###`` refers to either ``mem000`` (if running a deterministic forecast) or a specific forecast member number (if running an ensemble forecast). "Metatasks" indicate task definitions that will become more than one workflow task based on different variables, number of hours, etc., as described in the Task Description column. See :numref:`Section %s ` for more details about Metatasks. + +.. _VXWorkflowTasksTable: + +.. list-table:: Verification (VX) Workflow Tasks in the SRW App + :widths: 20 20 50 + :header-rows: 1 + + * - Workflow Task + - ``taskgroup`` + - Task Description + * - :bolditalic:`task_get_obs_ccpa` + - ``verify_pre.yaml`` + - If user has staged :term:`CCPA` data for verification, checks to ensure that data exists in the specified location (``CCPA_OBS_DIR``). If data does not exist, attempts to retrieve that data from NOAA :term:`HPSS`. + * - :bolditalic:`task_get_obs_ndas` + - ``verify_pre.yaml`` + - If user has staged :term:`NDAS` data for verification, checks to ensure that data exists in the specified location (``NDAS_OBS_DIR``). If data does not exist, attempts to retrieve that data from NOAA HPSS. + * - :bolditalic:`task_get_obs_nohrsc` + - ``verify_pre.yaml`` + - Retrieves and organizes hourly :term:`NOHRSC` data from NOAA HPSS. Can only be run if ``verify_pre.yaml`` is included in a ``tasksgroups`` list *and* user has access to NOAA :term:`HPSS` data. ``ASNOW`` should also be added to the ``VX_FIELDS`` list. + * - :bolditalic:`task_get_obs_mrms` + - ``verify_pre.yaml`` + - If user has staged :term:`MRMS` data for verification, checks to ensure that data exists in the specified location (``MRMS_OBS_DIR``). If data does not exist, attempts to retrieve that data from NOAA HPSS. + * - :bolditalic:`task_run_MET_Pb2nc_obs` + - ``verify_pre.yaml`` + - Convert files from prepbufr to NetCDF format. + * - :bolditalic:`metatask_PcpCombine_obs` + - ``verify_pre.yaml`` + - Derive 3-hr, 6-hr, and 24-hr accumulated precipitation observations from the 1-hr observation files. In log files, tasks will be named like ``MET_PcpCombine_obs_APCP##h``, where ``##h`` is 03h, 06h, or 24h. + * - :bolditalic:`metatask_check_post_output_all_mems` + - ``verify_pre.yaml`` + - Ensure that required post-processing tasks have completed and that the output exists in the correct form and location for each forecast member. In log files, tasks will be named like ``check_post_output_mem###``. + * - :bolditalic:`metatask_PcpCombine_fcst_APCP_all_accums_all_mems` + - ``verify_pre.yaml`` + - Derive accumulated precipitation forecast for 3-hr, 6-hr, and 24-hr windows for all forecast members based on 1-hr precipitation forecast values. In log files, tasks will be named like ``MET_PcpCombine_fcst_APCP##h_mem###``, where ``##h`` is 03h, 06h, or 24h. + * - :bolditalic:`metatask_PcpCombine_fcst_ASNOW_all_accums_all_mems` + - ``verify_pre.yaml`` + - Derive accumulated snow forecast for 6-hr and 24-hr windows for all forecast members based on 1-hr precipitation forecast values. In log files, tasks will be named like ``MET_PcpCombine_fcst_ASNOW##h_mem###``, where ``##h`` is 06h or 24h. + * - :bolditalic:`metatask_GridStat_CCPA_all_accums_all_mems` + - ``verify_det.yaml`` + - Runs METplus grid-to-grid verification for 1-h, 3-h, 6-h, and 24-h (i.e., daily) accumulated precipitation. In log files, tasks will be named like ``run_MET_GridStat_vx_APCP##h_mem###``. + * - :bolditalic:`metatask_GridStat_NOHRSC_all_accums_all_mems` + - ``verify_det.yaml`` + - Runs METplus grid-to-grid verification for 6-h and 24-h (i.e., daily) accumulated snow. In log files, tasks will be named like ``run_MET_GridStat_vx_ASNOW##h_mem###``. + * - :bolditalic:`metatask_GridStat_MRMS_all_mems` + - ``verify_det.yaml`` + - Runs METplus grid-to-grid verification for composite reflectivity and :term:`echo top`. In log files, tasks will be named like ``run_MET_GridStat_vx_REFC_mem###`` or ``run_MET_GridStat_vx_RETOP_mem###``. + * - :bolditalic:`metatask_PointStat_NDAS_all_mems` + - ``verify_det.yaml`` + - Runs METplus grid-to-point verification for surface and upper-air variables. In log files, tasks will be named like ``run_MET_PointStat_vx_SFC_mem###`` or ``run_MET_PointStat_vx_UPA_mem###``. + * - :bolditalic:`metatask_GenEnsProd_EnsembleStat_CCPA` :raw-html:`

` + (formerly *VX_ENSGRID_##h*) + - ``verify_ens.yaml`` + - Runs METplus grid-to-grid ensemble verification for 1-h, 3-h, 6-h, and 24-h (i.e., daily) accumulated precipitation. In log files, tasks will be named like ``run_MET_EnsembleStat_vx_APCP##h`` or ``run_MET_GenEnsProd_vx_APCP##h``. Can only be run if ``DO_ENSEMBLE: true`` in ``config.yaml``. + * - :bolditalic:`metatask_GenEnsProd_EnsembleStat_NOHRSC` + - ``verify_ens.yaml`` + - Runs METplus grid-to-grid ensemble verification for 6-h and 24-h (i.e., daily) accumulated snow. In log files, tasks will be named like ``run_MET_EnsembleStat_vx_ASNOW##h`` or ``run_MET_GenEnsProd_vx_ASNOW##h``. Can only be run if ``DO_ENSEMBLE: true`` in ``config.yaml``. + * - :bolditalic:`metatask_GenEnsProd_EnsembleStat_MRMS` :raw-html:`

` + (formerly *VX_ENSGRID_[REFC|RETOP]*) + - ``verify_ens.yaml`` + - Runs METplus grid-to-grid ensemble verification for composite reflectivity and :term:`echo top`. In log files, tasks will be named like ``run_MET_GenEnsProd_vx_[REFC|RETOP]`` or ``run_MET_EnsembleStat_vx_[REFC|RETOP]``. Can only be run if ``DO_ENSEMBLE: true`` in ``config.yaml``. + * - :bolditalic:`metatask_GridStat_CCPA_ensmeanprob_all_accums` :raw-html:`

` + (formerly *VX_ENSGRID_MEAN_##h* and *VX_ENSGRID_PROB_##h*) + - ``verify_ens.yaml`` + - Runs METplus grid-to-grid verification for (1) ensemble mean 1-h, 3-h, 6-h, and 24h (i.e., daily) accumulated precipitation and (2) 1-h, 3-h, 6-h, and 24h (i.e., daily) accumulated precipitation probabilistic output. In log files, the ensemble mean subtask will be named like ``run_MET_GridStat_vx_ensmean_APCP##h`` and the ensemble probabilistic output subtask will be named like ``run_MET_GridStat_vx_ensprob_APCP##h``, where ``##h`` is 01h, 03h, 06h, or 24h. Can only be run if ``DO_ENSEMBLE: true`` in ``config.yaml``. + * - :bolditalic:`metatask_GridStat_NOHRSC_ensmeanprob_all_accums` + - ``verify_ens.yaml`` + - Runs METplus grid-to-grid verification for (1) ensemble mean 6-h and 24h (i.e., daily) accumulated snow and (2) 6-h and 24h (i.e., daily) accumulated snow probabilistic output. In log files, the ensemble mean subtask will be named like ``run_MET_GridStat_vx_ensmean_ASNOW##h`` and the ensemble probabilistic output subtask will be named like ``run_MET_GridStat_vx_ensprob_ASNOW##h``, where ``##h`` is 06h or 24h. Can only be run if ``DO_ENSEMBLE: true`` in ``config.yaml``. + * - :bolditalic:`metatask_GridStat_MRMS_ensprob` :raw-html:`

` + (formerly *VX_ENSGRID_PROB_[REFC|RETOP]*) + - ``verify_ens.yaml`` + - Runs METplus grid-to-grid verification for ensemble probabilities for composite reflectivity and :term:`echo top`. In log files, tasks will be named like ``run_MET_GridStat_vx_ensprob_[REFC|RETOP]``. Can only be run if ``DO_ENSEMBLE: true`` in ``config.yaml``. + * - :bolditalic:`metatask_GenEnsProd_EnsembleStat_NDAS` :raw-html:`

` + (formerly *VX_ENSPOINT*) + - ``verify_ens.yaml`` + - Runs METplus grid-to-point ensemble verification for surface and upper-air variables. In log files, tasks will be named like ``run_MET_GenEnsProd_vx_[SFC|UPA]`` or ``run_MET_EnsembleStat_vx_[SFC|UPA]``. Can only be run if ``DO_ENSEMBLE: true`` in ``config.yaml``. + * - :bolditalic:`metatask_PointStat_NDAS_ensmeanprob` :raw-html:`

` + (formerly *VX_ENSPOINT_[MEAN|PROB]*) + - ``verify_ens.yaml`` + - Runs METplus grid-to-point verification for (1) ensemble mean surface and upper-air variables and (2) ensemble probabilities for surface and upper-air variables. In log files, tasks will be named like ``run_MET_PointStat_vx_ensmean_[SFC|UPA]`` or ``run_MET_PointStat_vx_ensprob_[SFC|UPA]``. Can only be run if ``DO_ENSEMBLE: true`` in ``config.yaml``. + +.. _Run: + +Run the Workflow +======================= + +The workflow can be run using the Rocoto workflow manager (see :numref:`Section %s `) or using standalone wrapper scripts (see :numref:`Section %s `). + +.. attention:: + + If users are running the SRW App on a system that does not have Rocoto installed (e.g., `Level 3 & 4 `__ systems, such as many MacOS or generic Linux systems), they should follow the process outlined in :numref:`Section %s `. + + +.. _UseRocoto: + +Run the Workflow Using Rocoto +-------------------------------- + +The information in this section assumes that Rocoto is available on the desired platform. All official HPC platforms for the UFS SRW App make use of the Rocoto workflow management software for running experiments. However, if Rocoto is not available, it is still possible to run the workflow using stand-alone scripts according to the process outlined in :numref:`Section %s `. + +There are three ways to run the workflow with Rocoto: (1) automation via crontab (2) by calling the ``launch_FV3LAM_wflow.sh`` script, and (3) by manually issuing the ``rocotorun`` command. + +.. note:: + Users may find it helpful to review :numref:`Section %s: Rocoto Introductory Information ` to gain a better understanding of Rocoto commands and workflow management before continuing, but this is not required to run the experiment. + +Optionally, an environment variable can be set to navigate to the experiment directory (``$EXPTDIR``) more easily. If the login shell is bash, it can be set as follows: + +.. code-block:: console + + export EXPTDIR=/path/to/experiment/directory + +If the login shell is csh/tcsh, it can instead be set using: + +.. code-block:: console + + setenv EXPTDIR /path/to/experiment/directory + + +.. _Automate: + +Automated Option +^^^^^^^^^^^^^^^^^^^ + +The simplest way to run the Rocoto workflow is to automate the process using a job scheduler such as :term:`Cron`. For automatic resubmission of the workflow at regular intervals (e.g., every 3 minutes), the user can add the following commands to their ``config.yaml`` file *before* generating the experiment (as outlined in :numref:`Section %s `): + +.. code-block:: console + + USE_CRON_TO_RELAUNCH: true + CRON_RELAUNCH_INTVL_MNTS: 3 + +This will automatically add an appropriate entry to the user's :term:`cron table` and launch the workflow. Alternatively, the user can add a crontab entry manually using the ``crontab -e`` command. As mentioned in :numref:`Section %s `, the last line of output from ``./generate_FV3LAM_wflow.py`` (starting with ``*/3 * * * *``), can be pasted into the crontab file. It can also be found in the ``$EXPTDIR/log.generate_FV3LAM_wflow`` file. The crontab entry should resemble the following: + +.. code-block:: console + + */3 * * * * cd /path/to/experiment/directory && ./launch_FV3LAM_wflow.sh called_from_cron="TRUE" + +where ``/path/to/experiment/directory`` is changed to correspond to the user's ``$EXPTDIR``. The number ``3`` can be changed to a different positive integer; it simply means that the workflow will be resubmitted every three minutes. + +.. hint:: + + * On NOAA Cloud instances, ``*/1 * * * *`` (or ``CRON_RELAUNCH_INTVL_MNTS: 1``) is the preferred option for cron jobs because compute nodes will shut down if they remain idle too long. If the compute node shuts down, it can take 15-20 minutes to start up a new one. + * On other NOAA HPC systems, administrators discourage using ``*/1 * * * *`` due to load problems. ``*/3 * * * *`` (or ``CRON_RELAUNCH_INTVL_MNTS: 3``) is the preferred option for cron jobs on non-NOAA Cloud systems. + +To check the experiment progress: + +.. code-block:: console + + cd $EXPTDIR + rocotostat -w FV3LAM_wflow.xml -d FV3LAM_wflow.db -v 10 + +After finishing the experiment, open the crontab using ``crontab -e`` and delete the crontab entry. + +.. _Success: + +The workflow run is complete when all tasks have "SUCCEEDED". If everything goes smoothly, users will eventually see a workflow status table similar to the following: + +.. code-block:: console + + CYCLE TASK JOBID STATE EXIT STATUS TRIES DURATION + ========================================================================================================== + 201906151800 make_grid 4953154 SUCCEEDED 0 1 5.0 + 201906151800 make_orog 4953176 SUCCEEDED 0 1 26.0 + 201906151800 make_sfc_climo 4953179 SUCCEEDED 0 1 33.0 + 201906151800 get_extrn_ics 4953155 SUCCEEDED 0 1 2.0 + 201906151800 get_extrn_lbcs 4953156 SUCCEEDED 0 1 2.0 + 201906151800 make_ics_mem000 4953184 SUCCEEDED 0 1 16.0 + 201906151800 make_lbcs_mem000 4953185 SUCCEEDED 0 1 71.0 + 201906151800 run_fcst_mem000 4953196 SUCCEEDED 0 1 1035.0 + 201906151800 run_post_mem000_f000 4953244 SUCCEEDED 0 1 5.0 + 201906151800 run_post_mem000_f001 4953245 SUCCEEDED 0 1 4.0 + ... + 201906151800 run_post_mem000_f012 4953381 SUCCEEDED 0 1 7.0 + +If users choose to run METplus verification tasks as part of their experiment, the output above will include additional lines after ``run_post_mem000_f012``. The output will resemble the following but may be significantly longer when using ensemble verification: + +.. code-block:: console + + CYCLE TASK JOBID STATE EXIT STATUS TRIES DURATION + ================================================================================================================ + 201906151800 make_grid 30466134 SUCCEEDED 0 1 5.0 + ... + 201906151800 run_post_mem000_f012 30468271 SUCCEEDED 0 1 7.0 + 201906151800 get_obs_ccpa 46903539 SUCCEEDED 0 1 9.0 + 201906151800 get_obs_mrms 46903540 SUCCEEDED 0 1 12.0 + 201906151800 get_obs_ndas 46903541 SUCCEEDED 0 1 9.0 + ... + 201906151800 run_gridstatvx 30468420 SUCCEEDED 0 1 53.0 + 201906151800 run_gridstatvx_refc 30468421 SUCCEEDED 0 1 934.0 + 201906151800 run_gridstatvx_retop 30468422 SUCCEEDED 0 1 1002.0 + 201906151800 run_gridstatvx_03h 30468491 SUCCEEDED 0 1 43.0 + 201906151800 run_gridstatvx_06h 30468492 SUCCEEDED 0 1 29.0 + 201906151800 run_gridstatvx_24h 30468493 SUCCEEDED 0 1 20.0 + 201906151800 run_pointstatvx 30468423 SUCCEEDED 0 1 670.0 + ... + 201906151800 run_MET_GridStat_vx_APCP01h_mem000 - - - - - + 201906151800 run_MET_GridStat_vx_APCP03h_mem000 - - - - - + 201906151800 run_MET_GridStat_vx_APCP06h_mem000 - - - - - + 201906151800 run_MET_GridStat_vx_REFC_mem000 - - - - - + 201906151800 run_MET_GridStat_vx_RETOP_mem000 - - - - - + 201906151800 run_MET_PointStat_vx_SFC_mem000 - - - - - + 201906151800 run_MET_PointStat_vx_UPA_mem000 - - - - - + +Launch the Rocoto Workflow Using a Script +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Users who prefer not to automate their experiments can run the Rocoto workflow using the ``launch_FV3LAM_wflow.sh`` script provided. Simply call it without any arguments from the experiment directory: + +.. code-block:: console + + cd $EXPTDIR + ./launch_FV3LAM_wflow.sh + +This script creates a log file named ``log.launch_FV3LAM_wflow`` in ``$EXPTDIR`` or appends information to the file if it already exists. The launch script also creates the ``log/FV3LAM_wflow.log`` file, which shows Rocoto task information. Check the end of the log file periodically to see how the experiment is progressing: + +.. code-block:: console + + tail -n 40 log.launch_FV3LAM_wflow + +In order to launch additional tasks in the workflow, call the launch script again; this action will need to be repeated until all tasks in the workflow have been launched. To (re)launch the workflow and check its progress on a single line, run: + +.. code-block:: console + + ./launch_FV3LAM_wflow.sh; tail -n 40 log.launch_FV3LAM_wflow + +This will output the last 40 lines of the log file, which lists the status of the workflow tasks (e.g., SUCCEEDED, DEAD, RUNNING, SUBMITTING, QUEUED). The number 40 can be changed according to the user's preferences. The output will look similar to this: + +.. code-block:: console + + CYCLE TASK JOBID STATE EXIT STATUS TRIES DURATION + ====================================================================================================== + 201906151800 make_grid druby://hfe01:33728 SUBMITTING - 0 0.0 + 201906151800 make_orog - - - - - + 201906151800 make_sfc_climo - - - - - + 201906151800 get_extrn_ics druby://hfe01:33728 SUBMITTING - 0 0.0 + 201906151800 get_extrn_lbcs druby://hfe01:33728 SUBMITTING - 0 0.0 + 201906151800 make_ics_mem000 - - - - - + 201906151800 make_lbcs_mem000 - - - - - + 201906151800 run_fcst_mem000 - - - - - + 201906151800 run_post_mem000_f000 - - - - - + 201906151800 run_post_mem000_f001 - - - - - + 201906151800 run_post_mem000_f002 - - - - - + 201906151800 run_post_mem000_f003 - - - - - + 201906151800 run_post_mem000_f004 - - - - - + 201906151800 run_post_mem000_f005 - - - - - + 201906151800 run_post_mem000_f006 - - - - - + + Summary of workflow status: + ~~~~~~~~~~~~~~~~~~~~~~~~~~ + + 0 out of 1 cycles completed. + Workflow status: IN PROGRESS + +If all the tasks complete successfully, the "Workflow status" at the bottom of the log file will change from "IN PROGRESS" to "SUCCESS". If certain tasks could not complete, the "Workflow status" will instead change to "FAILURE". Error messages for each task can be found in the task log files located in ``$EXPTDIR/log``. Users can look at the log file for a failed task to determine what caused the failure. For example, if the ``make_grid`` task failed, users can open the ``make_grid.log`` file to see what caused the problem: + +.. code-block:: console + + cd $EXPTDIR/log + vi make_grid.log + +After making any required changes, users can restart a DEAD or failed task as described in :numref:`Section %s of the FAQ `. + +The workflow run is complete when all tasks have "SUCCEEDED", and the ``rocotostat`` command outputs a table similar to the one :ref:`above `. + + +.. _RocotoManualRun: + +Launch the Rocoto Workflow Manually +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +**Load Rocoto** + +Instead of running the ``./launch_FV3LAM_wflow.sh`` script, users can load Rocoto and any other required modules manually. This gives the user more control over the process and allows them to view experiment progress more easily. On Level 1 systems, the Rocoto modules are loaded automatically in :numref:`Step %s `. For most other systems, users can load a modified ``wflow_`` modulefile, or they can use a variant on the following commands to load the Rocoto module: + +.. code-block:: console + + module use + module load rocoto + +Some systems may require a version number (e.g., ``module load rocoto/1.3.3``) + +**Run the Rocoto Workflow** + +After loading Rocoto, ``cd`` to the experiment directory and call ``rocotorun`` to launch the workflow tasks. This will start any tasks that do not have a dependency. As the workflow progresses through its stages, ``rocotostat`` will show the state of each task and allow users to monitor progress: + +.. code-block:: console + + cd $EXPTDIR + rocotorun -w FV3LAM_wflow.xml -d FV3LAM_wflow.db -v 10 + rocotostat -w FV3LAM_wflow.xml -d FV3LAM_wflow.db -v 10 + +The ``rocotorun`` and ``rocotostat`` commands above will need to be resubmitted regularly and repeatedly until the experiment is finished. In part, this is to avoid having the system time out. This also ensures that when one task ends, tasks dependent on it will run as soon as possible, and ``rocotostat`` will capture the new progress. + +If the experiment fails, the ``rocotostat`` command will indicate which task failed. Users can look at the log file in the ``log`` subdirectory for the failed task to determine what caused the failure. For example, if the ``make_grid`` task failed, users can open the ``make_grid.log`` file to see what caused the problem: + +.. code-block:: console + + cd $EXPTDIR/log + vi make_grid.log + +.. note:: + + If users have the `Slurm workload manager `__ on their system, they can run the ``squeue`` command in lieu of ``rocotostat`` to check what jobs are currently running. + + +.. _RunUsingStandaloneScripts: + +Run the Workflow Using Stand-Alone Scripts +--------------------------------------------- + +The SRW App workflow can be run using standalone shell scripts in cases where the Rocoto software is not available on a given platform. If Rocoto *is* available, see :numref:`Section %s ` to run the workflow using Rocoto. + +.. attention:: + + When working on an HPC system, users should allocate a compute node prior to running their experiment. The proper command will depend on the system's resource manager, but some guidance is offered in :numref:`Section %s `. It may be necessary to reload the ``build__`` scripts (see :numref:`Section %s `) and the workflow environment (see :numref:`Section %s `). + +.. note:: + Examples in this subsection presume that the user is running in the Terminal with a bash shell environment. If this is not the case, users will need to adjust the commands to fit their command line application and shell environment. + +#. ``cd`` into the experiment directory. For example, from ``ush``, presuming default directory settings: + + .. code-block:: console + + cd ../../expt_dirs/test_community + +#. Set the environment variable ``$EXPTDIR``: + + .. code-block:: console + + export EXPTDIR=`pwd` + +#. Copy the wrapper scripts from the ``ush`` directory into the experiment directory. Each workflow task has a wrapper script that sets environment variables and runs the job script. + + .. code-block:: console + + cp /path/to/ufs-srweather-app/ush/wrappers/* . + +#. Set the ``OMP_NUM_THREADS`` variable. + + .. code-block:: console + + export OMP_NUM_THREADS=1 + +#. Run each of the listed scripts in order. Scripts with the same stage number (listed in :numref:`Table %s `) may be run simultaneously. + + .. code-block:: console + + ./run_make_grid.sh + ./run_get_ics.sh + ./run_get_lbcs.sh + ./run_make_orog.sh + ./run_make_sfc_climo.sh + ./run_make_ics.sh + ./run_make_lbcs.sh + ./run_fcst.sh + ./run_post.sh + +Each task should finish with error code 0. For example: + +.. code-block:: console + + End exregional_get_extrn_mdl_files.sh at Wed Nov 16 18:08:19 UTC 2022 with error code 0 (time elapsed: 00:00:01) + +Check the batch script output file in your experiment directory for a “SUCCESS” message near the end of the file. + +.. _RegionalWflowTasks: + +.. table:: List of tasks in the SRW App workflow in the order that they are executed. + Scripts with the same stage number may be run simultaneously. The number of + processors and wall clock time is a good starting point for Cheyenne or Hera + when running a 48-h forecast on the 25-km CONUS domain. For a brief description of tasks, see :numref:`Table %s `. + + +------------+------------------------+----------------+----------------------------+ + | **Stage/** | **Task Run Script** | **Number of** | **Wall Clock Time (H:mm)** | + | | | **Processors** | | + +============+========================+================+============================+ + | 1 | run_get_ics.sh | 1 | 0:20 (depends on HPSS vs | + | | | | FTP vs staged-on-disk) | + +------------+------------------------+----------------+----------------------------+ + | 1 | run_get_lbcs.sh | 1 | 0:20 (depends on HPSS vs | + | | | | FTP vs staged-on-disk) | + +------------+------------------------+----------------+----------------------------+ + | 1 | run_make_grid.sh | 24 | 0:20 | + +------------+------------------------+----------------+----------------------------+ + | 2 | run_make_orog.sh | 24 | 0:20 | + +------------+------------------------+----------------+----------------------------+ + | 3 | run_make_sfc_climo.sh | 48 | 0:20 | + +------------+------------------------+----------------+----------------------------+ + | 4 | run_make_ics.sh | 48 | 0:30 | + +------------+------------------------+----------------+----------------------------+ + | 4 | run_make_lbcs.sh | 48 | 0:30 | + +------------+------------------------+----------------+----------------------------+ + | 5 | run_fcst.sh | 48 | 0:30 | + +------------+------------------------+----------------+----------------------------+ + | 6 | run_post.sh | 48 | 0:25 (2 min per output | + | | | | forecast hour) | + +------------+------------------------+----------------+----------------------------+ + +Users can access log files for specific tasks in the ``$EXPTDIR/log`` directory. To see how the experiment is progressing, users can also check the end of the ``log.launch_FV3LAM_wflow`` file from the command line: + +.. code-block:: console + + tail -n 40 log.launch_FV3LAM_wflow + +.. hint:: + If any of the scripts return an error that "Primary job terminated normally, but one process returned a non-zero exit code," there may not be enough space on one node to run the process. On an HPC system, the user will need to allocate a(nother) compute node. The process for doing so is system-dependent, and users should check the documentation available for their HPC system. Instructions for allocating a compute node on NOAA HPC systems can be viewed in :numref:`Section %s ` as an example. + +.. note:: + On most HPC systems, users will need to submit a batch job to run multi-processor jobs. On some HPC systems, users may be able to run the first two jobs (serial) on a login node/command-line. Example scripts for Slurm (Hera) and PBS (Cheyenne) resource managers are provided (``sq_job.sh`` and ``qsub_job.sh``, respectively). These examples will need to be adapted to each user's system. Alternatively, some batch systems allow users to specify most of the settings on the command line (with the ``sbatch`` or ``qsub`` command, for example). + +.. COMMENT: Test manual run section. diff --git a/docs/UsersGuide/source/Tutorial.rst b/docs/UsersGuide/source/BuildingRunningTesting/Tutorial.rst similarity index 90% rename from docs/UsersGuide/source/Tutorial.rst rename to docs/UsersGuide/source/BuildingRunningTesting/Tutorial.rst index f96e39ef52..c27544f54e 100644 --- a/docs/UsersGuide/source/Tutorial.rst +++ b/docs/UsersGuide/source/BuildingRunningTesting/Tutorial.rst @@ -52,27 +52,27 @@ On `Level 1 # OR: source etc/lmod-setup.csh when running in a csh/tcsh shell - module use /path/to/ufs-srweather-app/modulefiles + module use modulefiles module load wflow_ where ```` is a valid, lowercased machine name (see ``MACHINE`` in :numref:`Section %s ` for valid values). -After loading the workflow, users should follow the instructions printed to the console. Usually, the instructions will tell the user to run ``conda activate regional_workflow``. For example, a user on Hera with permissions on the ``nems`` project may issue the following commands to load the regional workflow (replacing ``User.Name`` with their actual username): +After loading the workflow, users should follow the instructions printed to the console. Usually, the instructions will tell the user to run ``conda activate workflow_tools``. For example, a user on Hera with permissions on the ``nems`` project may issue the following commands to load the workflow (replacing ``User.Name`` with their actual username): .. code-block:: console source /scratch1/NCEPDEV/nems/User.Name/ufs-srweather-app/etc/lmod-setup.sh hera module use /scratch1/NCEPDEV/nems/User.Name/ufs-srweather-app/modulefiles module load wflow_hera - conda activate regional_workflow + conda activate workflow_tools Configuration ------------------------- @@ -118,7 +118,7 @@ Start in the ``user:`` section and change the ``MACHINE`` and ``ACCOUNT`` variab For a detailed description of these variables, see :numref:`Section %s `. -Users do not need to change the ``platform:`` section of the configuration file for this tutorial. The default parameters in the ``platform:`` section pertain to METplus verification, which is not addressed here. For more information on verification, see :numref:`Chapter %s `. +Users do not need to change the ``platform:`` section of the configuration file for this tutorial. The default parameters in the ``platform:`` section pertain to METplus verification, which is not addressed here. For more information on verification, see :numref:`Section %s `. In the ``workflow:`` section of ``config.yaml``, update ``EXPT_SUBDIR`` and ``PREDEF_GRID_NAME``. @@ -142,24 +142,27 @@ In the ``workflow:`` section of ``config.yaml``, update ``EXPT_SUBDIR`` and ``PR Users may also want to set ``USE_CRON_TO_RELAUNCH: true`` and add ``CRON_RELAUNCH_INTVL_MNTS: 3``. This will automate submission of workflow tasks when running the experiment. However, not all systems have :term:`cron`. -``EXPT_SUBDIR:`` This variable can be changed to any name the user wants. This tutorial uses ``control`` to establish a baseline, or "control", forecast. Users can choose any name they want, from "gfsv16_physics_fcst" to "forecast1" to "a;skdfj". However, the best names will indicate useful information about the experiment. For example, this tutorial helps users to compare the output from two different forecasts: one that uses the FV3_GFS_v16 physics suite and one that uses the FV3_RRFS_v1beta physics suite. Therefore, "gfsv16_physics_fcst" could be a good alternative directory name. +``EXPT_SUBDIR:`` This variable can be changed to any name the user wants from "gfsv16_physics_fcst" to "forecast1" to "a;skdfj". However, the best names will indicate useful information about the experiment. This tutorial uses ``control`` to establish a baseline, or "control", forecast. Since this tutorial helps users to compare the output from two different forecasts --- one that uses the FV3_GFS_v16 physics suite and one that uses the FV3_RRFS_v1beta physics suite --- "gfsv16_physics_fcst" could be a good alternative directory name. ``PREDEF_GRID_NAME:`` This experiment uses the SUBCONUS_Ind_3km grid, rather than the default RRFS_CONUS_25km grid. The SUBCONUS_Ind_3km grid is a high-resolution grid (with grid cell size of approximately 3km) that covers a small area of the U.S. centered over Indianapolis, IN. For more information on this grid, see :numref:`Section %s `. For a detailed description of other ``workflow:`` variables, see :numref:`Section %s `. To turn on the plotting for the experiment, the YAML configuration file -should be included in the ``rocoto:taskgroups:`` section, like this: +should be included in the ``rocoto:tasks:taskgroups:`` section, like this: .. code-block:: console rocoto: tasks: + metatask_run_ensemble: + task_run_fcst_mem#mem#: + walltime: 02:00:00 taskgroups: '{{ ["parm/wflow/prep.yaml", "parm/wflow/coldstart.yaml", "parm/wflow/post.yaml", "parm/wflow/plot.yaml"]|include }}' For more information on how to turn on/off tasks in the worklfow, please -see :numref:`Section %s ` +see :numref:`Section %s `. In the ``task_get_extrn_ics:`` section, add ``USE_USER_STAGED_EXTRN_FILES`` and ``EXTRN_MDL_SOURCE_BASEDIR_ICS``. Users will need to adjust the file path to reflect the location of data on their system (see :numref:`Section %s ` for locations on `Level 1 `__ systems). @@ -171,7 +174,7 @@ In the ``task_get_extrn_ics:`` section, add ``USE_USER_STAGED_EXTRN_FILES`` and USE_USER_STAGED_EXTRN_FILES: true EXTRN_MDL_SOURCE_BASEDIR_ICS: /path/to/UFS_SRW_App/develop/input_model_data/FV3GFS/grib2/${yyyymmddhh} -For a detailed description of the ``task_get_extrn_ics:`` variables, see :numref:`Section %s `. +For a detailed description of the ``task_get_extrn_ics:`` variables, see :numref:`Section %s `. Similarly, in the ``task_get_extrn_lbcs:`` section, add ``USE_USER_STAGED_EXTRN_FILES`` and ``EXTRN_MDL_SOURCE_BASEDIR_LBCS``. Users will need to adjust the file path to reflect the location of data on their system (see :numref:`Section %s ` for locations on Level 1 systems). @@ -248,6 +251,8 @@ Once the control case is running, users can return to the ``config.yaml`` file ( Later, users may want to conduct additional experiments using the FV3_HRRR and FV3_WoFS_v0 physics suites. Like FV3_RRFS_v1beta, these physics suites were designed for use with high-resolution grids for storm-scale predictions. +.. COMMENT: Maybe also FV3_RAP? + Next, users will need to modify the data parameters in ``task_get_extrn_ics:`` and ``task_get_extrn_lbcs:`` to use HRRR and RAP data rather than FV3GFS data. Users will need to change the following lines in each section: .. code-block:: console @@ -262,14 +267,21 @@ Next, users will need to modify the data parameters in ``task_get_extrn_ics:`` a HRRR and RAP data are better than FV3GFS data for use with the FV3_RRFS_v1beta physics scheme because these datasets use the same physics :term:`parameterizations` that are in the FV3_RRFS_v1beta suite. They focus on small-scale weather phenomena involved in storm development, so forecasts tend to be more accurate when HRRR/RAP data are paired with FV3_RRFS_v1beta and a high-resolution (e.g., 3-km) grid. Using HRRR/RAP data with FV3_RRFS_v1beta also limits the "spin-up adjustment" that takes place when initializing with model data coming from different physics. -``EXTRN_MDL_LBCS_OFFSET_HRS:`` This variable allows users to use lateral boundary conditions (LBCs) from a previous forecast run that was started earlier than the start time of the forecast being configured in this experiment. This variable is set to 0 by default except when using RAP data; with RAP data, the default value is 3, so the forecast will look for LBCs from a forecast started 3 hours earlier (i.e., at 2019061515 --- 15z --- instead of 2019061518). To avoid this, users must set ``EXTRN_MDL_LBCS_OFFSET_HRS`` explicitly. +``EXTRN_MDL_LBCS_OFFSET_HRS:`` This variable allows users to use lateral boundary conditions (:term:`LBCs`) from a previous forecast run that was started earlier than the start time of the forecast being configured in this experiment. This variable is set to 0 by default except when using RAP data; with RAP data, the default value is 3, so the forecast will look for LBCs from a forecast started 3 hours earlier (i.e., at 2019061515 --- 15z --- instead of 2019061518). To avoid this, users must set ``EXTRN_MDL_LBCS_OFFSET_HRS`` explicitly. -Add a section to ``config.yaml`` to increase the maximum wall time (``WTIME_RUN_POST``) for the postprocessing tasks. The wall time is the maximum length of time a task is allowed to run. On some systems, the default of 15 minutes may be enough, but on others (e.g., NOAA Cloud), the post-processing time exceeds 15 minutes, so the tasks fail. +Under ``rocoto:tasks:``, add a section to increase the maximum wall time for the postprocessing tasks. The walltime is the maximum length of time a task is allowed to run. On some systems, the default of 15 minutes may be enough, but on others (e.g., NOAA Cloud), the post-processing time exceeds 15 minutes, so the tasks fail. .. code-block:: console - task_run_post: - WTIME_RUN_POST: 00:20:00 + rocoto: + tasks: + metatask_run_ensemble: + task_run_fcst_mem#mem#: + walltime: 02:00:00 + taskgroups: '{{ ["parm/wflow/prep.yaml", "parm/wflow/coldstart.yaml", "parm/wflow/post.yaml", "parm/wflow/plot.yaml"]|include }}' + metatask_run_ens_post: + run_fcst_mem#mem#: + walltime: 00:20:00 Lastly, users must set the ``COMOUT_REF`` variable in the ``task_plot_allvars:`` section to create difference plots that compare output from the two experiments. ``COMOUT_REF`` is a template variable, so it references other workflow variables within it (see :numref:`Section %s ` for details on template variables). ``COMOUT_REF`` should provide the path to the ``control`` experiment forecast output using single quotes as shown below: @@ -302,6 +314,13 @@ To see experiment progress, users should navigate to their experiment directory. If users have not automated their workflow using cron, they will need to ensure that they continue issuing ``rocotorun`` commands to launch all of the tasks in each experiment. While switching between experiment directories to run ``rocotorun`` and ``rocotostat`` commands in both directories is possible, it may be easier to finish the ``control`` experiment's tasks before starting on ``test_expt``. +As with the ``control`` experiment, users can save the location of the ``test_expt`` directory in an environment variable (e.g., ``$TEST``). This makes it easier to navigate between directories later. For example: + +.. code-block:: console + + export TEST=/path/to/expt_dirs/test_expt + +Users should substitute ``/path/to/expt_dirs/test_expt`` with the actual path on their system. Compare and Analyze Results ----------------------------- @@ -319,7 +338,7 @@ In summary, users can run the ``scp`` command in a new terminal/command prompt w scp username@your-IP-address:/path/to/source_file_or_directory /path/to/destination_file_or_directory # OR - scp -P 12345 username@localhost:/path/to/source_file_or_directory path/to/destination_file_or_directory + scp -P 12345 username@localhost:/path/to/source_file_or_directory /path/to/destination_file_or_directory Users would need to modify ``username``, ``your-IP-address``, ``-P 12345``, and the file paths to reflect their systems' information. See the :ref:`Introduction to SSH & Data Transfer ` for example commands. @@ -356,7 +375,7 @@ The plots generated by the experiment cover a variety of variables. After downlo | Max/Min 2 - 5 km updraft helicity | uh25_diff_regional_fhhh.png | +-----------------------------------------+-----------------------------------+ -Each difference plotting ``.png`` file contains three subplots. The plot for the second experiment (``test_expt``) appears in the top left corner, and the plot for the first experiment (``control``) appears in the top right corner. The difference plot that compares both experiments appear at the bottom. Areas of white signify no difference between the plots. Therefore, if the forecast output from both experiments is exactly the same, the difference plot will show a white square (see :ref:`Sea Level Pressure ` for an example). If the forecast output from both experiments is extremely different, the plot will show lots of color. +Each difference plotting ``.png`` file contains three subplots. The plot for the second experiment (``test_expt``) appears in the top left corner, and the plot for the first experiment (``control``) appears in the top right corner. The difference plot that compares both experiments appears at the bottom. Areas of white signify no difference between the plots. Therefore, if the forecast output from both experiments is exactly the same, the difference plot will show a white square (see :ref:`Sea Level Pressure ` as an example). If the forecast output from both experiments is extremely different, the plot will show lots of color. In general, it is expected that the results for ``test_expt`` (using FV3_RRFS_v1beta physics and HRRR/RAP data) will be more accurate than the results for ``control`` (using FV3_GFS_v16 physics and FV3GFS data) because the physics in ``test_expt`` is designed for high-resolution, storm-scale prediction over a short period of time. The ``control`` experiment physics is better for predicting the evolution of larger scale weather phenomena, like jet stream movement and cyclone development, since the cumulus physics in the FV3_GFS_v16 suite is not configured to run at 3-km resolution. @@ -393,7 +412,7 @@ The predictions diverge further by f012, where a solid section of light blue in Composite Reflectivity `````````````````````````` -Reflectivity images visually represent the weather based on the energy (measured in decibels [dBZ]) reflected back from radar. Composite reflectivity generates an image based on reflectivity scans at multiple elevation angles, or "tilts", of the antenna. See https://www.weather.gov/jetstream/refl for a more detailed explanation of composite reflectivity. +Reflectivity images visually represent the weather based on the energy (measured in decibels [dBZ]) reflected back from radar. Composite reflectivity generates an image based on reflectivity scans at multiple elevation angles, or "tilts", of the antenna. See https://www.noaa.gov/jetstream/reflectivity for a more detailed explanation of composite reflectivity. At f000, the ``test_expt`` plot (top left) is showing more severe weather than the ``control`` plot (top right). The ``test_expt`` plot shows a vast swathe of the Indianapolis region covered in yellow with spots of orange, corresponding to composite reflectivity values of 35+ dBZ. The ``control`` plot radar image covers a smaller area of the grid, and with the exception of a few yellow spots, composite reflectivity values are <35 dBZ. The difference plot (bottom) shows areas where the ``test_expt`` plot (red) and the ``control`` plot (blue) have reflectivity values greater than 20 dBZ. The ``test_expt`` plot has significantly more areas with high composite reflectivity values. @@ -520,7 +539,7 @@ Sample Forecast #3: Southern Plains Winter Weather Event Weather Summary -------------------- -A polar vortex brought arctic air to much of the U.S. and Mexico. A series of cold fronts and vorticity disturbances helped keep this cold air in place for an extended period of time resulting in record-breaking cold temperatures for many southern states and Mexico. This particular case captures two winter weather disturbances between February 14, 2021 at 06z and February 17, 2021 at 06z that brought several inches of snow to Oklahoma City. A lull on February 16, 2021 resulted in record daily low temperatures. +A polar vortex brought arctic air to much of the U.S. and Mexico. A series of cold fronts and vorticity disturbances helped keep this cold air in place for an extended period of time, resulting in record-breaking cold temperatures for many southern states and Mexico. This particular case captures two winter weather disturbances between February 14, 2021 at 06z and February 17, 2021 at 06z that brought several inches of snow to Oklahoma City. A lull on February 16, 2021 resulted in record daily low temperatures. **Weather Phenomena:** Snow and record-breaking cold temperatures diff --git a/docs/UsersGuide/source/VXCases.rst b/docs/UsersGuide/source/BuildingRunningTesting/VXCases.rst similarity index 82% rename from docs/UsersGuide/source/VXCases.rst rename to docs/UsersGuide/source/BuildingRunningTesting/VXCases.rst index 4b01a283dc..533f6e7fb2 100644 --- a/docs/UsersGuide/source/VXCases.rst +++ b/docs/UsersGuide/source/BuildingRunningTesting/VXCases.rst @@ -1,22 +1,22 @@ .. _VXCases: -============================ -Verification Sample Cases -============================ +=================================== +METplus Verification Sample Cases +=================================== Introduction =============== The goal of these sample cases is to provide the UFS community with datasets that they can modify and run to see if their changes can improve the forecast and/or reduce the model biases. Each case covers an interesting weather event. The case that was added with the v2.1.0 release was a severe weather event over Indianapolis on June 15-16, 2019. In the future, additional sample cases will be provided. -Each sample case contains model output from a control run; this output includes ``postprd`` (post-processed) and ``metprd`` (MET verification-processed) directories. Under the ``postprd`` directory, users will find the :term:`UPP` output of the model run along with plots for several forecast variables. These can be used for a visual/qualitative comparison of forecasts. The ``metprd`` directory contains METplus verification statistics files, which can be used for a quantitative comparison of forecast outputs. +Each sample case contains model output from a control run; this output includes ``postprd`` (post-processed) and ``metprd`` (MET verification-processed) directories. Under the ``postprd`` directory, users will find the :term:`UPP` output of the model run along with plots for several forecast variables (when plotting tasks are run). These can be used for a visual/qualitative comparison of forecasts. The ``metprd`` directory contains METplus verification statistics files, which can be used for a quantitative comparison of forecast outputs. Prerequisites ================ This chapter assumes that users have already (1) built the SRW App v2.1.0 successfully and (2) installed MET and METplus on their system. -For instructions on how to build the v2.1.0 release of the SRW App, see the v2.1.0 release documentation on :ref:`Building the SRW App `. The release code is used to provide a consistent point of comparison; the ``develop`` branch code is constantly receiving updates, which makes it unsuited to this purpose. Users will have an easier time if they run through the out-of-the-box case described in :numref:`Chapter %s ` before attempting to run any verification sample cases, but doing so is optional. +For instructions on how to build the v2.1.0 release of the SRW App, see the v2.1.0 release documentation on :ref:`Building the SRW App `. The release code is used to provide a consistent point of comparison; the ``develop`` branch code is constantly receiving updates, which makes it unsuited to this purpose. Users will have an easier time if they run through the out-of-the-box case described in the v2.1.0 release documentation on :ref:`Running the SRW App ` before attempting to run any verification sample cases, but doing so is optional. For information on MET and METplus, see :numref:`Section %s `, which contains information on METplus, links to a list of existing MET/METplus builds on `Level 1 & 2 `__ systems, and links to installation instructions and documentation for users on other systems. @@ -49,19 +49,19 @@ On other systems, users need to download the ``Indy-Severe-Weather.tgz`` file us #. Download directly from the S3 bucket using a browser. The data is available at https://noaa-ufs-srw-pds.s3.amazonaws.com/index.html#sample_cases/release-public-v2.1.0/. - #. Download from a terminal using the AWS command line interface (CLI) if installed: + #. Download from a terminal using the AWS command line interface (CLI), if installed: .. code-block:: console aws s3 cp https://noaa-ufs-srw-pds.s3.amazonaws.com/index.html#sample_cases/release-public-v2.1.0/Indy-Severe-Weather.tgz Indy-Severe-Weather.tgz - #. Download from a terminal using wget: + #. Download from a terminal using ``wget``: .. code-block:: console wget https://noaa-ufs-srw-pds.s3.amazonaws.com/sample_cases/release-public-v2.1.0/Indy-Severe-Weather.tgz -This tar file contains :term:`IC/LBC ` files, observation data, model/forecast output and MET verification output for the sample forecast. Users who have never run the SRW App on their system before will also need to download the fix files required for SRW App forecasts and the NaturalEarth shapefiles required for plotting. Users can download the fix file data from a browser at https://noaa-ufs-srw-pds.s3.amazonaws.com/current_srw_release_data/fix_data.tgz or visit :numref:`Section %s ` for instructions on how to download the data with wget. NaturalEarth files are available at https://noaa-ufs-srw-pds.s3.amazonaws.com/NaturalEarth/NaturalEarth.tgz. See the :ref:`Graphics ` chapter of the release documentation for more information. +This tar file contains :term:`IC/LBC ` files, observation data, model/forecast output, and MET verification output for the sample forecast. Users who have never run the SRW App on their system before will also need to download (1) the fix files required for SRW App forecasts and (2) the NaturalEarth shapefiles required for plotting. Users can download the fix file data from a browser at https://noaa-ufs-srw-pds.s3.amazonaws.com/current_srw_release_data/fix_data.tgz or visit :numref:`Section %s ` for instructions on how to download the data with ``wget``. NaturalEarth files are available at https://noaa-ufs-srw-pds.s3.amazonaws.com/NaturalEarth/NaturalEarth.tgz. See the :ref:`Graphics ` chapter of the release documentation for more information. After downloading ``Indy-Severe-Weather.tgz`` using one of the three methods above, untar the downloaded compressed archive file: @@ -80,37 +80,37 @@ Record the path to this file output using the ``pwd`` command: Users can untar the fix files and Natural Earth files by substituting those file names in the commands above. -Load the Regional Workflow -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +Load the Workflow +^^^^^^^^^^^^^^^^^^^^ -First, navigate to the ``ufs-srweather-app/ush`` directory. Then, load the regional workflow environment: +First, navigate to the ``ufs-srweather-app/ush`` directory. Then, load the workflow environment: .. code-block:: console - source - module use + source /path/to/etc/lmod-setup.sh + module use /path/to/ufs-srweather-app/modulefiles module load wflow_ -Users running a csh/tcsh shell would run ``source `` in place of the first command above. +Users running a csh/tcsh shell would run ``source /path/to/etc/lmod-setup.csh `` in place of the first command above. After loading the workflow, users should follow the instructions printed to the console. Usually, the instructions will tell the user to run ``conda activate regional_workflow``. Configure the Verification Sample Case ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -Once the regional workflow is loaded, copy the out-of-the-box configuration: +Once the workflow environment is loaded, copy the out-of-the-box configuration: .. code-block:: console - cd + cd /path/to/ufs-srweather-app/ush cp config.community.yaml config.yaml -where ```` is replaced by the actual path to the ``ufs-srweather-app/ush`` directory on the user's system. +where ``/path/to/ufs-srweather-app/ush`` is replaced by the actual path to the ``ufs-srweather-app/ush`` directory on the user's system. Then, edit the configuration file (``config.yaml``) to include the variables and values in the sample configuration excerpt below (variables not listed below do not need to be changed or removed). Users must be sure to substitute values in ``<>`` with values appropriate to their system. .. note:: - Variables ``CCPA_OBS_DIR``, ``MRMS_OBS_DIR``, and ``NDAS_OBS_DIR`` are set in the ./ush/machine/.yaml configuration. Variables ``MET_INSTALL_DIR``, ``METPLUS_PATH``, ``MET_BIN_EXEC`` are set in modulefiles for met/xx.x.x and metplus/x.x.x, when the modules are loaded in ./modulefiles/tasks//run_vx.local (run_vx.local.lua modulefile) + Users working on a `Level 1 platform `__ do not need to add or update the following variables: ``MET_INSTALL_DIR``, ``METPLUS_PATH``, ``MET_BIN_EXEC``, ``CCPA_OBS_DIR``, ``MRMS_OBS_DIR``, and ``NDAS_OBS_DIR``. .. code-block:: console @@ -118,6 +118,13 @@ Then, edit the configuration file (``config.yaml``) to include the variables and ACCOUNT: platform: MODEL: FV3_GFS_v16_SUBCONUS_3km + MET_INSTALL_DIR: /path/to/met/x.x.x # Example: MET_INSTALL_DIR: /contrib/met/10.1.1 + METPLUS_PATH: /path/to/METplus/METplus-x.x.x # Example: METPLUS_PATH: /contrib/METplus/METplus-4.1.1 + # Add MET_BIN_EXEC variable to config.yaml + MET_BIN_EXEC: bin + CCPA_OBS_DIR: /path/to/Indy-Severe-Weather/obs_data/ccpa/proc + MRMS_OBS_DIR: /path/to/Indy-Severe-Weather/obs_data/mrms/proc + NDAS_OBS_DIR: /path/to/Indy-Severe-Weather/obs_data/ndas/proc workflow: EXPT_SUBDIR: DATE_FIRST_CYCL: '2019061500' @@ -128,11 +135,11 @@ Then, edit the configuration file (``config.yaml``) to include the variables and RUN_TASK_VX_POINTSTAT: true task_get_extrn_ics: # Add EXTRN_MDL_SOURCE_BASEDIR_ICS variable to config.yaml - EXTRN_MDL_SOURCE_BASEDIR_ICS: + EXTRN_MDL_SOURCE_BASEDIR_ICS: /path/to/Indy-Severe-Weather/input_model_data/FV3GFS/grib2/2019061500 USE_USER_STAGED_EXTRN_FILES: true task_get_extrn_lbcs: # Add EXTRN_MDL_SOURCE_BASEDIR_LBCS variable to config.yaml - EXTRN_MDL_SOURCE_BASEDIR_LBCS: + EXTRN_MDL_SOURCE_BASEDIR_LBCS: /path/to/Indy-Severe-Weather/input_model_data/FV3GFS/grib2/2019061500 USE_USER_STAGED_EXTRN_FILES: true task_run_fcst: WTIME_RUN_FCST: 05:00:00 @@ -147,7 +154,7 @@ Then, edit the configuration file (``config.yaml``) to include the variables and To modify the file, hit the ``i`` key and then make any changes required. To close and save, hit the ``esc`` key and type ``:wq``. Users may opt to use their preferred code editor instead. -For additional configuration guidance, refer to :numref:`Section %s `. +For additional configuration guidance, refer to the v2.1.0 release documentation on :ref:`configuring the SRW App `. Generate the Experiment ^^^^^^^^^^^^^^^^^^^^^^^^^^ @@ -186,13 +193,13 @@ The plots are created using the graphics generation script that comes with the S .. code-block:: console - cd - python plot_allvars.py 2019061500 0 60 6 / SUBCONUS_Ind_3km + cd /path/to/ufs-srweather-app/ush/Python + python plot_allvars.py 2019061500 0 60 6 /path/to/experiment/directory /path/to/NaturalEarth SUBCONUS_Ind_3km Compare ---------- -Once the experiment has completed (i.e., all tasks have "SUCCEEDED" and the end of the ``log.launch_FV3LAM_wflow`` file lists "Workflow status: SUCCESS"), users can compare their forecast results against the forecast results provided in the ``Indy-Severe-Weather`` directory downloaded in :numref:`Section %s `. This directory contains the forecast output and plots from NOAA developers under the ``postprd`` directory and METplus verification files under the ``metprd`` directory. +Once the experiment has completed (i.e., all tasks have "SUCCEEDED" and the end of the ``log.launch_FV3LAM_wflow`` file lists "Workflow status: SUCCESS"), users can compare their forecast results against the forecast results provided in the ``Indy-Severe-Weather`` directory downloaded in :numref:`Section %s `. This directory contains the forecast output and plots from NOAA developers under the ``postprd`` subdirectory and METplus verification files under the ``metprd`` subdirectory. Qualitative Comparison of the Plots ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ diff --git a/docs/UsersGuide/source/WE2Etests.rst b/docs/UsersGuide/source/BuildingRunningTesting/WE2Etests.rst similarity index 68% rename from docs/UsersGuide/source/WE2Etests.rst rename to docs/UsersGuide/source/BuildingRunningTesting/WE2Etests.rst index 185faa777d..9e03b451b2 100644 --- a/docs/UsersGuide/source/WE2Etests.rst +++ b/docs/UsersGuide/source/BuildingRunningTesting/WE2Etests.rst @@ -1,9 +1,9 @@ .. _WE2E_tests: -================================== -Workflow End-to-End (WE2E) Tests -================================== -The SRW App contains a set of end-to-end tests that exercise various workflow configurations of the SRW App. These are referred to as workflow end-to-end (WE2E) tests because they all use the Rocoto workflow manager to run their individual workflows from start to finish. The purpose of these tests is to ensure that new changes to the App do not break existing functionality and capabilities. +======================= +Testing the SRW App +======================= +The SRW App contains a set of end-to-end tests that exercise various workflow configurations of the SRW App. These are referred to as workflow end-to-end (WE2E) tests because they all use the Rocoto workflow manager to run their individual workflows from start to finish. The purpose of these tests is to ensure that new changes to the App do not break existing functionality and capabilities. However, these WE2E tests also provide users with additional sample cases and data beyond the basic ``config.community.yaml`` case. Note that the WE2E tests are not regression tests---they do not check whether current results are identical to previously established baselines. They also do @@ -11,11 +11,14 @@ not test the scientific integrity of the results (e.g., they do not check that v of output fields are reasonable). These tests only check that the tasks within each test's workflow complete successfully. They are, in essence, tests of the workflow generation, task execution (:term:`J-jobs`, :term:`ex-scripts`), and other auxiliary scripts to ensure that these scripts function correctly. Tested functions include creating and correctly arranging and naming directories and files, ensuring -that all input files are available and readable, calling executables with correct namelists and/or options, etc. Currently, it is up to the external repositories that the App clones (:numref:`Section %s `) to check that changes to those repositories do not change results, or, if they do, to ensure that the new results are acceptable. (At least two of these external repositories---``UFS_UTILS`` and ``ufs-weather-model``---do have such regression tests.) +that all input files are available and readable, calling executables with correct namelists and/or options, etc. Currently, it is up to the external repositories that the App clones (:numref:`Section %s `) to check that changes to those repositories do not change results, or, if they do, to ensure that the new results are acceptable. (At least two of these external repositories---``UFS_UTILS`` and ``ufs-weather-model``---do have such regression tests.) + +WE2E Test Categories +====================== -WE2E tests are grouped into two categories that are of interest to code developers: ``fundamental`` and ``comprehensive`` tests. "Fundamental" tests are a lightweight but wide-reaching set of tests designed to function as a cheap "`smoke test `__ for changes to the UFS SRW App. The fundamental suite of test runs common combinations of workflow tasks, physical domains, input data, physics suites, etc. -The comprehensive suite of tests covers a broader range of combinations of capabilities, configurations, and components, ideally including all capabilities that *can* be run on a given platform. Because some capabilities are not available on all platforms (*e.g.*, retrieving data directly from NOAA HPSS), the suite of comprehensive tests varies from machine to machine. -The list of fundamental and comprehensive tests can be viewed in the ``ufs-srweather-app/tests/WE2E/machine_suites/`` directory, and are described in more detail in :doc:`this table `. +WE2E tests are grouped into two categories that are of interest to code developers: ``fundamental`` and ``comprehensive`` tests. "Fundamental" tests are a lightweight but wide-reaching set of tests designed to function as a cheap "`smoke test `__" for changes to the UFS SRW App. The fundamental suite of tests runs common combinations of workflow tasks, physical domains, input data, physics suites, etc. +The comprehensive suite of tests covers a broader range of combinations of capabilities, configurations, and components, ideally including all capabilities that *can* be run on a given platform. Because some capabilities are not available on all platforms (e.g., retrieving data directly from NOAA HPSS), the suite of comprehensive tests varies from machine to machine. +The list of fundamental and comprehensive tests can be viewed in the ``ufs-srweather-app/tests/WE2E/machine_suites/`` directory and are described in more detail in :doc:`this table <../tables/Tests>`. .. note:: @@ -23,6 +26,9 @@ The list of fundamental and comprehensive tests can be viewed in the ``ufs-srwea For convenience, the WE2E tests are currently grouped into the following categories (under ``ufs-srweather-app/tests/WE2E/test_configs/``): +* ``custom_grids`` + This category tests custom grids aside from those specified in ``ufs-srweather-app/ush/predef_grid_params.yaml``. These tests help ensure a wide range of domain sizes, resolutions, and locations will work as expected. These test files can also serve as examples for how to set your own custom domain. + * ``default_configs`` This category tests example config files provided for user reference. They are symbolically linked from the ``ufs-srweather-app/ush/`` directory. @@ -41,31 +47,82 @@ For convenience, the WE2E tests are currently grouped into the following categor * ``wflow_features`` This category of tests ensures that the workflow completes successfully with particular features/capabilities activated. -Some tests are duplicated among the above categories via symbolic links, both for legacy reasons (when tests for different capabilities were consolidated) and for convenience when a user would like to run all tests for a specific category (*e.g.* verification tests). +Some tests are duplicated among the above categories via symbolic links, both for legacy reasons (when tests for different capabilities were consolidated) and for convenience when a user would like to run all tests for a specific category (e.g., verification tests). -The script to run the WE2E tests is named ``run_WE2E_tests.py`` and is located in the directory ``ufs-srweather-app/tests/WE2E``. Each WE2E test has an associated configuration file named ``config.${test_name}.yaml``, where ``${test_name}`` is the name of the corresponding test. These configuration files are subsets of the full range of ``config.yaml`` experiment configuration options. (See :numref:`Chapter %s ` for all configurable options and :numref:`Section %s ` for information on configuring ``config.yaml``.) For each test, the ``run_WE2E_tests.py`` script reads in the test configuration file and generates from it a complete ``config.yaml`` file. It then calls the ``generate_FV3LAM_wflow()`` function, which in turn reads in ``config.yaml`` and generates a new experiment for the test. The name of each experiment directory is set to that of the corresponding test, and a copy of ``config.yaml`` for each test is placed in its experiment directory. +Running the WE2E Tests +================================ -As with any other experiment within the App, the -Python modules required for experiment generation must be loaded before ``run_WE2E_tests.py`` -can be called. See :numref:`Section %s ` for information on loading the Python -environment on supported platforms. Note also that ``run_WE2E_tests.py`` assumes that all of -the executables have been built (see :numref:`Section %s `). If they have not, then ``run_WE2E_tests.py`` will still generate the experiment directories, but the workflows will fail. +The Test Script (``run_WE2E_tests.py``) +----------------------------------------- + +The script to run the WE2E tests is named ``run_WE2E_tests.py`` and is located in the directory ``ufs-srweather-app/tests/WE2E``. Each WE2E test has an associated configuration file named ``config.${test_name}.yaml``, where ``${test_name}`` is the name of the corresponding test. These configuration files are subsets of the full range of ``config.yaml`` experiment configuration options. (See :numref:`Section %s ` for all configurable options and :numref:`Section %s ` for information on configuring ``config.yaml``.) For each test, the ``run_WE2E_tests.py`` script reads in the test configuration file and generates from it a complete ``config.yaml`` file. It then calls the ``generate_FV3LAM_wflow()`` function, which in turn reads in ``config.yaml`` and generates a new experiment for the test. The name of each experiment directory is set to that of the corresponding test, and a copy of ``config.yaml`` for each test is placed in its experiment directory. .. note:: - The full list of WE2E tests is extensive and some larger, high-resolution tests are computationally expensive. Estimates of walltime and core-hour cost for each test are provided in :doc:`this table `. + The full list of WE2E tests is extensive, and some larger, high-resolution tests are computationally expensive. Estimates of walltime and core-hour cost for each test are provided in :doc:`this table <../tables/Tests>`. -Running the WE2E Tests -================================ +Using the Test Script +---------------------- + +.. attention:: + + These instructions assume that the user has already built the SRW App (as described in :numref:`Section %s `) and loaded the appropriate python environment (as described in :numref:`Section %s `). + +The test script has three required arguments: machine, account, and tests. + + * Users must indicate which machine they are on using the ``--machine`` or ``-m`` option. See ``ush/machine`` or :numref:`Section %s ` for valid values. + * Users must submit a valid account name using the ``--account`` or ``-a`` option to run submitted jobs. On systems where an account name is not required, users may simply use ``-a none``. + * Users must specify the set of tests to run using the ``--tests`` or ``-t`` option. Users may pass (in order of priority): + + #. The name of a single test or list of tests to the test script. + #. A test suite name (e.g., "fundamental", "comprehensive", "coverage", or "all"). + #. The name of a subdirectory under ``ufs-srweather-app/tests/WE2E/test_configs/`` + #. The name of a text file (full or relative path), such as ``my_tests.txt``, which contains a list of the WE2E tests to run (one per line). + +Users may run ``./run_WE2E_tests.py -h`` for additional (optional) usage instructions. + +Examples +^^^^^^^^^^^ + +.. attention:: + + * Users will need to adjust the machine name and account in these examples to run tests successfully. + * These commands assume that the user is working from the ``WE2E`` directory (``ufs-srweather-app/tests/WE2E/``). + +To run the ``custom_ESGgrid`` and ``pregen_grid_orog_sfc_climo`` tests on Jet, users could run: + +.. code-block:: console + + ./run_WE2E_tests.py -t custom_ESGgrid pregen_grid_orog_sfc_climo -m jet -a hfv3gfs -Users may specify the set of tests to run in one of three ways. First, users can pass the name of a single test or list of tests to the script. Secondly, they can pass an option to run the ``fundamental`` or ``comprehensive`` suite of tests. Finally, users can create a text file, such as ``my_tests.txt``, which contains a list of the WE2E tests to run (one per line). Any one of these options can be passed to the ``run_WE2E_tests.py`` script via the ``--tests`` or ``-t`` option. +Alternatively, to run the entire suite of fundamental tests on Hera, users might run: -For example, to run the tests ``custom_ESGgrid`` and ``grid_RRFS_CONUScompact_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16`` (from the ``wflow_features`` and ``grids_extrn_mdls_suites_community`` categories, respectively), users would enter the following commands from the ``WE2E`` working directory (``ufs-srweather-app/tests/WE2E/``): +.. code-block:: console + + ./run_WE2E_tests.py -t fundamental -m hera -a nems + +To run the tests ``custom_ESGgrid`` and ``grid_RRFS_CONUScompact_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16`` on NOAA Cloud, users would enter the following commands: .. code-block:: console echo "custom_ESGgrid" > my_tests.txt echo "grid_RRFS_CONUScompact_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16" >> my_tests.txt + ./run_WE2E_tests.py -t my_tests.txt -m noaacloud -a none + +By default, the experiment directory for a WE2E test has the same name as the test itself, and it is created in ``${HOMEdir}/../expt_dirs``, where ``HOMEdir`` is the top-level directory for the ``ufs-srweather-app`` repository (usually set to something like ``/path/to/ufs-srweather-app``). Thus, the ``custom_ESGgrid`` experiment directory would be located in ``${HOMEdir}/../expt_dirs/custom_ESGgrid``. + +**A More Complex Example:** To run the fundamental suite of tests on Orion in parallel, charging computational resources to the "gsd-fv3" account, and placing all the experiment directories into a directory named ``test_set_01``, run: + + .. code-block:: + + ./run_WE2E_tests.py -t fundamental -m orion -a gsd-fv3 --expt_basedir "test_set_01" -q -p 2 + + * ``--expt_basedir``: Useful for grouping sets of tests. If set to a relative path, the provided path will be appended to the default path. In this case, all of the fundamental tests will reside in ``${HOMEdir}/../expt_dirs/test_set_01/``. It can also take a full path as an argument, which will place experiments in the given location. + * ``-q``: Suppresses the output from ``generate_FV3LAM_wflow()`` and prints only important messages (warnings and errors) to the screen. The suppressed output will still be available in the ``log.run_WE2E_tests`` file. + * ``-p 2``: Indicates the number of parallel proceeses to run. By default, job monitoring and submission is serial, using a single task. Therefore, the script may take a long time to return to a given experiment and submit the next job when running large test suites. Depending on the machine settings, running in parallel can substantially reduce the time it takes to run all experiments. However, it should be used with caution on shared resources (such as HPC login nodes) due to the potential to overwhelm machine resources. + +Workflow Information +^^^^^^^^^^^^^^^^^^^^^^ For each specified test, ``run_WE2E_tests.py`` will generate a new experiment directory and, by default, launch a second function ``monitor_jobs()`` that will continuously monitor active jobs, submit new jobs, and track the success or failure status of the experiment in a ``.yaml`` file. Finally, when all jobs have finished running (successfully or not), the function ``print_WE2E_summary()`` will print a summary of the jobs to screen, including the job's success or failure, timing information, and (if on an appropriately configured platform) the number of core hours used. An example run would look like this: @@ -77,12 +134,12 @@ For each specified test, ``run_WE2E_tests.py`` will generate a new experiment di /user/home/ufs-srweather-app/tests/WE2E/test_configs/wflow_features/config.custom_ESGgrid.yaml /user/home/ufs-srweather-app/tests/WE2E/test_configs/grids_extrn_mdls_suites_community/config.grid_RRFS_CONUScompact_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16.yaml Calling workflow generation function for test custom_ESGgrid - + ... Workflow for test custom_ESGgrid successfully generated in /user/home/expt_dirs/custom_ESGgrid Calling workflow generation function for test grid_RRFS_CONUScompact_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16 - + ... Workflow for test grid_RRFS_CONUScompact_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16 successfully generated in /user/home/expt_dirs/grid_RRFS_CONUScompact_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16 @@ -113,14 +170,10 @@ For each specified test, ``run_WE2E_tests.py`` will generate a new experiment di All experiments are complete Summary of results available in WE2E_tests_20230418174042.yaml - -.. note:: - These examples assume that the user has already built the SRW App and loaded the appropriate python environment as described in :numref:`Section %s `. +As the script runs, detailed debug output is written to the file ``log.run_WE2E_tests``. This can be useful for debugging if something goes wrong. Adding the ``-d`` flag will print all this output to the screen during the run, but this can get quite cluttered. -As the script runs, detailed debug output is written to the file ``log.run_WE2E_tests``. This can be useful for debugging if something goes wrong. You can also use the ``-d`` flag to print all this output to screen during the run, but this can get quite cluttered. - -The final job summary is written by the ``print_WE2E_summary()``; this prints a short summary of experiments to screen, and prints a more detailed summary of all jobs for all experiments in the indicated ``.txt`` file. +The progress of ``monitor_jobs()`` is tracked in a file ``WE2E_tests_{datetime}.yaml``, where {datetime} is the date and time (in ``yyyymmddhhmmss`` format) that the file was created. The final job summary is written by the ``print_WE2E_summary()``; this prints a short summary of experiments to the screen and prints a more detailed summary of all jobs for all experiments in the indicated ``.txt`` file. .. code-block:: console @@ -182,7 +235,7 @@ The final job summary is written by the ``print_WE2E_summary()``; this prints a Total COMPLETE 15.52 -One might have noticed the line during the experiment run that reads "Use ctrl-c to pause job submission/monitoring". The ``monitor_jobs()`` function (called automatically after all experiments are generated) is designed to be easily paused and re-started if necessary. If you wish to stop actively submitting jobs, simply quitting the script using "ctrl-c" will stop the function, and give a short message on how to continue the experiment. +One might have noticed the line during the experiment run that reads "Use ctrl-c to pause job submission/monitoring". The ``monitor_jobs()`` function (called automatically after all experiments are generated) is designed to be easily paused and re-started if necessary. To stop actively submitting jobs, simply quit the script using ``ctrl-c`` to stop the function, and a short message will appear explaining how to continue the experiment: .. code-block:: console @@ -190,78 +243,15 @@ One might have noticed the line during the experiment run that reads "Use ctrl-c Use ctrl-c to pause job submission/monitoring ^C - User interrupted monitor script; to resume monitoring jobs run: ./monitor_jobs.py -y=WE2E_tests_20230418174042.yaml -p=1 -The full list of options for any of these scripts can be found by using the ``-h`` flag. The examples below demonstrate several of the more common options for ``run_WE2E_tests.py``. - -#. To run the tests listed in ``my_tests.txt`` on Hera and charge the computational - resources used to the "rtrr" account: - - .. code-block:: - - ./run_WE2E_tests.py --tests=my_tests.txt --machine=hera --account=rtrr - - This will create the experiment subdirectories for the two sample WE2E tests in the directory ``${HOMEdir}/../expt_dirs``, where ``HOMEdir`` is the top-level directory for the ufs-srweather-app repository (usually set to something like ``/path/to/ufs-srweather-app``). Thus, the following two experiment directories will be created: - - .. code-block:: - - ${HOMEdir}/../expt_dirs/custom_ESGgrid - ${HOMEdir}/../expt_dirs/grid_RRFS_CONUScompact_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16 - - Once these experiment directories are created, the script will call the ``monitor_jobs()`` function. This function runs ``rocotorun`` in the background to monitor the status of jobs in each experiment directory, tracking the status of jobs as they run and complete, and submitting new jobs when they are ready. The progress of ``monitor_jobs()`` is tracked in a file ``WE2E_tests_{datetime}.yaml``, where {datetime} is the date and time (in ``yyyymmddhhmmss`` format) that the file was created. - -#. Our second example will run the fundamental suite of tests on Orion, charging computational resources to the "gsd-fv3" account, and placing the experiment subdirectories in a subdirectory named ``test_set_01``: - - .. code-block:: - - ./run_WE2E_tests.py -t fundamental -m hera -a gsd-fv3 --expt_basedir "test_set_01" -q - - In this case, the full paths to the experiment directories will be: - - .. code-block:: - - ${HOMEdir}/../expt_dirs/test_set_01/grid_RRFS_CONUScompact_25km_ics_HRRR_lbcs_RAP_suite_RRFS_v1beta - ${HOMEdir}/../expt_dirs/test_set_01/nco_grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_timeoffset_suite_GFS_v16 - ${HOMEdir}/../expt_dirs/test_set_01/grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v15p2 - ${HOMEdir}/../expt_dirs/test_set_01/grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v17_p8 - ${HOMEdir}/../expt_dirs/test_set_01/grid_RRFS_CONUScompact_25km_ics_HRRR_lbcs_HRRR_suite_HRRR - ${HOMEdir}/../expt_dirs/test_set_01/grid_SUBCONUS_Ind_3km_ics_HRRR_lbcs_RAP_suite_WoFS_v0 - ${HOMEdir}/../expt_dirs/test_set_01/grid_RRFS_CONUS_25km_ics_NAM_lbcs_NAM_suite_GFS_v16 - - - The ``--expt_basedir`` option is useful for grouping various sets of tests. It can also be given a full path as an argument, which will place experiments in the given location. - - The ``-q`` flag (as used in the first example shown above), is helpful for keeping the screen less cluttered; this will suppress the output from ``generate_FV3LAM_wflow()``, only printing important messages (warnings and errors) to screen. As always, this output will still be available in the ``log.run_WE2E_tests`` file. - -#. By default, the job monitoring and submission process is serial, using a single task. For test suites that contain many experiments, this means that the script may take a long time to return to a given experiment and submit the next job, due to the amount of time it takes for the ``rocotorun`` command to complete. In order to speed this process up, provided you have access to a node with the appropriate availability (e.g., submitting from a compute node), you can run the job monitoring processes in parallel using the ``-p`` option: - - .. code-block:: - - ./run_WE2E_tests.py -m=jet -a=gsd-fv3-dev -t=all -q -p 6 - - Depending on your machine settings, this can reduce the time it takes to run all experiments substantially. However, it should be used with caution on shared resources (such as HPC login nodes) due to the potential to overwhelm machine resources. - -#. This example will run the single experiment "custom_ESGgrid" on Hera, charging computational resources to the "fv3lam" account. For this example, we submit the suite of tests using the legacy :term:`cron`-based system: - -.. note:: - - This option is not recommended, as it does not work on some machines and can cause system bottlenecks on others. - - .. code-block:: - - ./run_WE2E_tests.py -t=custom_ESGgrid -m=hera -a=fv3lam --use_cron_to_relaunch --cron_relaunch_intvl_mnts=1 - -The option ``--use_cron_to_relaunch`` means that, rather than calling the ``monitor_jobs()`` function, the ``generate_FV3LAM_wflow()`` function will create a new :term:`cron` job in the user's cron table that will launch the experiment with the workflow launch script (``launch_FV3LAM_wflow.sh``). By default this script is run every 2 minutes, but we have changed that to 1 minute with the ``--cron_relaunch_intvl_mnts=1`` argument. This script will run until the workflow either completes successfully (i.e., all tasks SUCCEEDED) or fails (i.e., at least one task fails). The cron job is then removed from the user's cron table. - - -Checking test status and summary +Checking Test Status and Summary ================================= -By default, ``./run_WE2E_tests.py`` will actively monitor jobs, printing to screen when jobs are complete (either successfully or with a failure), and print a summary file ``WE2E_summary_{datetime.now().strftime("%Y%m%d%H%M%S")}.txt``. -However, if the user is using the legacy crontab option, or would like to summarize one or more experiments that are either not complete or were not handled by the WE2E test scripts, this status/summary file can be generated manually using ``WE2E_summary.py``. -In this example, an experiment was generated using the crontab option, and has not yet finished running. +By default, ``./run_WE2E_tests.py`` will actively monitor jobs, printing to console when jobs are complete (either successfully or with a failure), and printing a summary file ``WE2E_summary_{datetime.now().strftime("%Y%m%d%H%M%S")}.txt``. +However, if the user is using the legacy crontab option (by submitting ``./run_WE2E_tests.py`` with the ``--launch cron`` option), or if the user would like to summarize one or more experiments that either are not complete or were not handled by the WE2E test scripts, this status/summary file can be generated manually using ``WE2E_summary.py``. +In this example, an experiment was generated using the crontab option and has not yet finished running. We use the ``-e`` option to point to the experiment directory and get the current status of the experiment: .. code-block:: @@ -301,7 +291,36 @@ We use the ``-e`` option to point to the experiment directory and get the curren Detailed summary written to WE2E_summary_20230306173013.txt -As with all python scripts in the App, additional options for this script can be viewed by calling with the ``-h`` argument. +As with all python scripts in the SRW App, additional options for this script can be viewed by calling with the ``-h`` argument. + +The "Status" as specified by the above summary is explained below: + +* ``CREATED`` + The experiment directory has been created, but the monitor script has not yet begun submitting jobs. This is immediately overwritten at the beginning of the "monitor_jobs" function, so this status should not be seen unless the experiment has not yet been started. + +* ``SUBMITTING`` + All jobs are in status SUBMITTING or SUCCEEDED (as reported by the Rocoto workflow manager). This is a normal state; we will continue to monitor this experiment. + +* ``DYING`` + One or more tasks have died (status "DEAD"), so this experiment has had an error. We will continue to monitor this experiment until all tasks are either status DEAD or status SUCCEEDED (see next entry). + +* ``DEAD`` + One or more tasks are at status DEAD, and the rest are either DEAD or SUCCEEDED. We will no longer monitor this experiment. + +* ``ERROR`` + Could not read the rocoto database file. This will require manual intervention to solve, so we will no longer monitor this experiment. + +* ``RUNNING`` + One or more jobs are at status RUNNING, and the rest are either status QUEUED, SUBMITTED, or SUCCEEDED. This is a normal state; we will continue to monitor this experiment. + +* ``QUEUED`` + One or more jobs are at status QUEUED, and some others may be at status SUBMITTED or SUCCEEDED. This is a normal state; we will continue to monitor this experiment. + +* ``SUCCEEDED`` + All jobs are status SUCCEEDED; we will monitor for one more cycle in case there are unsubmitted jobs remaining. + +* ``COMPLETE`` + All jobs are status SUCCEEDED, and we have monitored this job for an additional cycle to ensure there are no un-submitted jobs. We will no longer monitor this experiment. .. _WE2ETestInfoFile: @@ -309,7 +328,7 @@ As with all python scripts in the App, additional options for this script can be WE2E Test Information File ================================== -If the user wants to see consolidated test information, they can generate a file that can be imported into a spreadsheet program (Google Sheets, Microsoft Excel, etc.) that summarizes each test. This file, named ``WE2E_test_info.txt`` by default, is delimited by the ``|`` character, and can be created either by running the ``./print_test_info.py`` script, or by generating an experiment using ``./run_WE2E_tests.py`` with the ``--print_test_info`` flag. +If the user wants to see consolidated test information, they can generate a file that can be imported into a spreadsheet program (Google Sheets, Microsoft Excel, etc.) that summarizes each test. This file, named ``WE2E_test_info.txt`` by default, is delimited by the ``|`` character and can be created either by running the ``./print_test_info.py`` script, or by generating an experiment using ``./run_WE2E_tests.py`` with the ``--print_test_info`` flag. The rows of the file/sheet represent the full set of available tests (not just the ones to be run). The columns contain the following information (column titles are included in the CSV file): @@ -337,9 +356,9 @@ The rows of the file/sheet represent the full set of available tests (not just t | Here, ``nx`` and ``ny`` are the number of grid points in the horizontal (``x`` and ``y``) directions, ``num_time_steps`` is the number of time steps in one forecast, and ``num_fcsts`` is the number of forecasts the - test runs (see Column 5 below). [Note that this cost calculation does - not (yet) differentiate between different physics suites.] The relative - cost ``rel_cost`` is then calculated using + test runs (see Column 5 below). (Note that this cost calculation does + not (yet) differentiate between different physics suites.) The relative + cost ``rel_cost`` is then calculated using: .. code-block:: @@ -348,7 +367,7 @@ The rows of the file/sheet represent the full set of available tests (not just t | where ``abs_cost_ref`` is the absolute cost of running the reference forecast described above, i.e., a single (``num_fcsts = 1``) 6-hour forecast (``FCST_LEN_HRS = 6``) on the ``RRFS_CONUS_25km grid`` (which currently has - ``nx = 219``, ``ny = 131``, and ``DT_ATMOS = 40 sec`` (so that ``num_time_steps + ``nx = 219``, ``ny = 131``, and ``DT_ATMOS = 40 sec`` (so that ``num_time_steps = FCST_LEN_HRS*3600/DT_ATMOS = 6*3600/40 = 540``). Therefore, the absolute cost reference is calculated as: .. code-block:: @@ -383,8 +402,8 @@ The rows of the file/sheet represent the full set of available tests (not just t Modifying the WE2E System ============================ -This section describes various ways in which the WE2E testing system can be modified -to suit specific testing needs. + +Users may wish to modify the WE2E testing system to suit specific testing needs. .. _ModExistingTest: @@ -403,10 +422,10 @@ Adding a New Test --------------------- To add a new test named, e.g., ``new_test01``, to one of the existing test categories, such as ``wflow_features``: -#. Choose an existing test configuration file in any one of the category directories that matches most closely the new test to be added. Copy that file to ``config.new_test01.yaml`` and, if necessary, move it to the ``wflow_features`` category directory. +#. Choose an existing test configuration file that most closely matches the new test to be added. It could come from any one of the category directories. + +#. Copy that file to ``config.new_test01.yaml`` and, if necessary, move it to the ``wflow_features`` category directory. #. Edit the header comments in ``config.new_test01.yaml`` so that they properly describe the new test. #. Edit the contents of ``config.new_test01.yaml`` by modifying existing experiment variable values and/or adding new variables such that the test runs with the intended configuration. - - diff --git a/docs/UsersGuide/source/BuildingRunningTesting/index.rst b/docs/UsersGuide/source/BuildingRunningTesting/index.rst new file mode 100644 index 0000000000..5c0efc3c64 --- /dev/null +++ b/docs/UsersGuide/source/BuildingRunningTesting/index.rst @@ -0,0 +1,15 @@ +Building, Running, and Testing the SRW App +============================================ + +.. toctree:: + :maxdepth: 3 + + + Quickstart + ContainerQuickstart + BuildSRW + RunSRW + WE2Etests + Tutorial + VXCases + AQM diff --git a/docs/UsersGuide/source/Components.rst b/docs/UsersGuide/source/Components.rst deleted file mode 100644 index b5e03eec85..0000000000 --- a/docs/UsersGuide/source/Components.rst +++ /dev/null @@ -1,86 +0,0 @@ -.. _Components: - -============================ -SRW Application Components -============================ - -The SRW Application assembles a variety of components, including: - -* Pre-processor Utilities & Initial Conditions -* UFS Weather Forecast Model -* Unified Post Processor -* Visualization Examples -* Build System and Workflow - -These components are documented within this User's Guide and supported through the `GitHub Discussions `__ forum. - -.. _Utils: - -Pre-processor Utilities and Initial Conditions -============================================== - -The SRW Application includes a number of pre-processing utilities that initialize and prepare the model. Since the SRW App provides forecast predictions over a limited area (rather than globally), it is necessary to first generate a regional grid (``regional_esg_grid/make_hgrid``) along with :term:`orography` (``orog``) and surface climatology (``sfc_climo_gen``) files on that grid. Grids include a strip, or "halo," of six cells that surround the regional grid and feed in lateral boundary condition data. Since different grid and orography files require different numbers of :term:`halo` cells, additional utilities handle topography filtering and shave the number of halo points (based on downstream workflow component requirements). The pre-processing software :term:`chgres_cube` is used to convert the raw external model data into initial and lateral boundary condition files in :term:`netCDF` format. These are needed as input to the :term:`FV3`-:term:`LAM`. Additional information about the UFS pre-processor utilities can be found in the `UFS_UTILS Technical Documentation `__ and in the `UFS_UTILS Scientific Documentation `__. - -The SRW Application can be initialized from a range of operational initial condition files. It is possible to initialize the model from the Global Forecast System (:term:`GFS`), North American Mesoscale (:term:`NAM`) Forecast System, Rapid Refresh (:term:`RAP`), and High-Resolution Rapid Refresh (:term:`HRRR`) files in Gridded Binary v2 (:term:`GRIB2`) format. GFS files also come in :term:`NEMSIO` format for past dates. - -.. WARNING:: - For GFS data, dates prior to 1 January 2018 may work but are not guaranteed. Public archives of model data can be accessed through the `NOAA Operational Model Archive and Distribution System `__ (NOMADS). Raw external model data may be pre-staged on disk by the user. - - -Forecast Model -============== - -The prognostic atmospheric model in the UFS SRW Application is the Finite-Volume Cubed-Sphere (:term:`FV3`) dynamical core configured with a Limited Area Model (:term:`LAM`) capability :cite:`BlackEtAl2021`. The :term:`dynamical core` is the computational part of a model that solves the equations of fluid motion. A User's Guide for the UFS Weather Model can be accessed `here `__. - -Supported model resolutions in this release include 3-, 13-, and 25-km predefined contiguous U.S. (:term:`CONUS`) domains, each with 127 vertical levels. Preliminary tools for users to define their own domain are also available in the release with full, formal support of these tools to be provided in future releases. The Extended Schmidt Gnomonic (ESG) grid is used with the FV3-LAM, which features relatively uniform grid cells across the entirety of the domain. Additional information about the FV3 dynamical core can be found in the `scientific documentation `__, the `technical documentation `__, and on the `NOAA Geophysical Fluid Dynamics Laboratory website `__. - -Interoperable atmospheric physics, along with various land surface model options, are supported through the Common Community Physics Package (CCPP), described `here `__. Atmospheric physics are a set of numerical methods describing small-scale processes such as clouds, turbulence, radiation, and their interactions. There are four physics suites supported as of the SRW App v2.1.0 release. The first is the FV3_RRFS_v1beta physics suite, which is being tested for use in the future operational implementation of the Rapid Refresh Forecast System (:term:`RRFS`) planned for 2023-2024, and the second is an updated version of the physics suite used in the operational Global Forecast System (GFS) v16. Additionally, FV3_WoFS_v0, FV3_HRRR, and FV3_RAP are supported. A detailed list of CCPP updates since the SRW App v2.0.0 release is available :ref:`here `. A full scientific description of CCPP parameterizations and suites can be found in the `CCPP Scientific Documentation `__, and CCPP technical aspects are described in the `CCPP Technical Documentation `__. The model namelist has many settings beyond the physics options that can optimize various aspects of the model for use with each of the supported suites. Additional information on Stochastic Physics options is available `here `__. Additionally, a CCPP single-column model (`CCPP-SCM `__) option has also been developed as a child repository. Users can refer to the `CCPP Single Column Model User and Technical Guide `__ for more details. This CCPP-SCM user guide contains a Quick Start Guide with instructions for obtaining the code, compiling, and running test cases, which include five standard test cases and two additional FV3 replay cases (refer to section 5.2 in the CCPP-SCM user guide for more details). Moreover, the CCPP-SCM supports a precompiled version in a docker container, allowing it to be easily executed on NOAA's cloud computing platforms without any issues (see section 2.5 in the CCPP-SCM user guide for more details). - -.. note:: - SPP is currently only available for specific physics schemes used in the RAP/HRRR physics suite. Users need to be aware of which physics suite definition file (:term:`SDF`) is chosen when turning this option on. Among the supported physics suites, the full set of parameterizations can only be used with the ``FV3_HRRR`` option for ``CCPP_PHYS_SUITE``. - -The SRW App supports the use of both :term:`GRIB2` and :term:`NEMSIO` input data. The UFS Weather Model ingests initial and lateral boundary condition files produced by :term:`chgres_cube` and outputs files in netCDF format on a specific projection (e.g., Lambert Conformal) in the horizontal direction and model levels in the vertical direction. - -Post Processor -============== - -The SRW Application is distributed with the Unified Post Processor (:term:`UPP`) included in the workflow as a way to convert the netCDF output on the native model grid to :term:`GRIB2` format on standard isobaric vertical coordinates. The UPP can also be used to compute a variety of useful diagnostic fields, as described in the `UPP User's Guide `__. - -Output from UPP can be used with visualization, plotting, and verification packages or in further downstream post-processing (e.g., statistical post-processing techniques). - -.. _MetplusComponent: - -METplus Verification Suite -============================= - -The enhanced Model Evaluation Tools (`METplus `__) verification system has been integrated into the SRW App to facilitate forecast evaluation. METplus is a verification framework that spans a wide range of temporal scales (warn-on-forecast to climate) and spatial scales (storm to global). It is supported by the `Developmental Testbed Center (DTC) `__. - -METplus *installation* is not included as part of the build process for the most recent release of the SRW App. However, METplus is preinstalled on many `Level 1 & 2 `__ systems; existing builds can be viewed `here `__. - -METplus can be installed on other systems individually or as part of :term:`HPC-Stack` installation. Users on systems without a previous installation of METplus can follow the `MET Installation Guide `__ and `METplus Installation Guide `__ for individual installation. Currently, METplus *installation* is not a supported feature for this release of the SRW App. However, METplus *use* is supported on systems with a functioning METplus installation. - -The core components of the METplus framework include the statistical driver, MET, the associated database and display systems known as METviewer and METexpress, and a suite of Python wrappers to provide low-level automation and examples, also called use cases. MET is a set of verification tools developed for use by the :term:`NWP` community. It matches up grids with either gridded analyses or point observations and applies configurable methods to compute statistics and diagnostics. Extensive documentation is available in the `METplus User's Guide `__ and `MET User's Guide `__. Documentation for all other components of the framework can be found at the Documentation link for each component on the METplus `downloads `__ page. - -Among other techniques, MET provides the capability to compute standard verification scores for comparing deterministic gridded model data to point-based and gridded observations. It also provides ensemble and probabilistic verification methods for comparing gridded model data to point-based or gridded observations. Verification tasks to accomplish these comparisons are defined in the SRW App in :numref:`Table %s `. Currently, the SRW App supports the use of :term:`NDAS` observation files (which include conventional point-based surface and upper-air data) in `prepBUFR format `__ for point-based verification. It also supports gridded Climatology-Calibrated Precipitation Analysis (:term:`CCPA`) data for accumulated precipitation evaluation and Multi-Radar/Multi-Sensor (:term:`MRMS`) gridded analysis data for composite reflectivity and :term:`echo top` verification. - -METplus is being actively developed by :term:`NCAR`/Research Applications Laboratory (RAL), NOAA/Earth Systems Research Laboratories (ESRL), and NOAA/Environmental Modeling Center (:term:`EMC`), and it is open to community contributions. - -Build System and Workflow -========================= - -The SRW Application has a portable build system and a user-friendly, modular, and expandable workflow framework. - -An umbrella CMake-based build system is used for building the components necessary for running the end-to-end SRW Application, including the UFS Weather Model and the pre- and post-processing software. Additional libraries necessary for the application (e.g., :term:`NCEPLIBS-external` and :term:`NCEPLIBS`) are not included in the SRW Application build system but are available pre-built on pre-configured platforms. On other systems, they can be installed via the HPC-Stack (see :doc:`HPC-Stack Documentation `). There is a small set of system libraries and utilities that are assumed to be present on the target computer: the CMake build software; a Fortran, C, and C++ compiler; and an :term:`MPI` library. - -Once built, the provided experiment generator script can be used to create a Rocoto-based -workflow file that will run each task in the system in the proper sequence (see :numref:`Chapter %s ` or the `Rocoto documentation `__ for more information on Rocoto). If Rocoto and/or a batch system is not present on the available platform, the individual components can be run in a stand-alone, command line fashion with provided run scripts. The generated namelist for the atmospheric model can be modified in order to vary settings such as forecast starting and ending dates, forecast length hours, the :term:`CCPP` physics suite, integration time step, history file output frequency, and more. It also allows for configuration of other elements of the workflow; for example, users can choose whether to run some or all of the pre-processing, forecast model, and post-processing steps. - -An optional Python plotting task is also included to create basic visualizations of the model output. The task outputs graphics in PNG format for several standard meteorological variables on the pre-defined :term:`CONUS` domain. A difference plotting option is also included to visually compare two runs for the same domain and resolution. These plots may be used to perform a visual check to verify that the application is producing reasonable results. Configuration instructions are provided in :numref:`Section %s `. - -The latest SRW Application release has been tested on a variety of platforms widely used by researchers, such as the NOAA Research and Development High-Performance Computing Systems (RDHPCS), including Hera, Orion, and Jet; the National Center for Atmospheric Research (:term:`NCAR`) Cheyenne system; and generic Linux and MacOS systems using Intel and GNU compilers. Four `levels of support `__ have been defined for the SRW Application, including pre-configured (Level 1), configurable (Level 2), limited-test (Level 3), and build-only (Level 4) platforms. Each level is further described below. - -On pre-configured (Level 1) computational platforms, all the required libraries for building the SRW Application are available in a central place. That means bundled libraries (NCEPLIBS) and third-party libraries (NCEPLIBS-external) have both been built. The SRW Application is expected to build and run out-of-the-box on these pre-configured platforms. - -A few additional computational platforms are considered configurable for the SRW Application release. Configurable platforms (Level 2) are platforms where all of the required libraries for building the SRW Application are expected to install successfully but are not available in a central location. Applications and models are expected to build and run once the required bundled libraries (e.g., NCEPLIBS) and third-party libraries (e.g., NCEPLIBS-external) are built. - -Limited-Test (Level 3) and Build-Only (Level 4) computational platforms are those in which the developers have built the code but little or no pre-release testing has been conducted, respectively. A complete description of the levels of support, along with a list of preconfigured and configurable platforms can be found in the `SRW Application Wiki `__. diff --git a/docs/UsersGuide/source/ConfigWorkflow.rst b/docs/UsersGuide/source/CustomizingTheWorkflow/ConfigWorkflow.rst similarity index 97% rename from docs/UsersGuide/source/ConfigWorkflow.rst rename to docs/UsersGuide/source/CustomizingTheWorkflow/ConfigWorkflow.rst index 717640e8a2..4fa1fea891 100644 --- a/docs/UsersGuide/source/ConfigWorkflow.rst +++ b/docs/UsersGuide/source/CustomizingTheWorkflow/ConfigWorkflow.rst @@ -127,9 +127,6 @@ METplus Parameters :ref:`METplus ` is a scientific verification framework that spans a wide range of temporal and spatial scales. Many of the METplus parameters are described below, but additional documentation for the METplus components is available on the `METplus website `__. -``MODEL``: (Default: "") - A descriptive name of the user's choice for the model being verified. - .. _METParamNote: .. note:: @@ -144,31 +141,31 @@ METplus Parameters ``CCPA_OBS_DIR``: (Default: "") User-specified location of top-level directory where CCPA hourly precipitation files used by METplus are located. This parameter needs to be set for both user-provided observations and for observations that are retrieved from the NOAA :term:`HPSS` (if the user has access) via the ``get_obs_ccpa`` task. (This task is activated in the workflow by using the taskgroup file ``parm/wflow/verify_pre.yaml``). - METplus configuration files require the use of a predetermined directory structure and file names. If the CCPA files are user-provided, they need to follow the anticipated naming structure: ``{YYYYMMDD}/ccpa.t{HH}z.01h.hrap.conus.gb2``, where YYYYMMDD and HH are as described in the note :ref:`above `. When pulling observations from NOAA HPSS, the data retrieved will be placed in the ``CCPA_OBS_DIR`` directory. This path must be defind as ``//ccpa/proc``. METplus is configured to verify 01-, 03-, 06-, and 24-h accumulated precipitation using hourly CCPA files. + METplus configuration files require the use of a predetermined directory structure and file names. If the CCPA files are user-provided, they need to follow the anticipated naming structure: ``{YYYYMMDD}/ccpa.t{HH}z.01h.hrap.conus.gb2``, where YYYYMMDD and HH are as described in the note :ref:`above `. When pulling observations from NOAA HPSS, the data retrieved will be placed in the ``CCPA_OBS_DIR`` directory. METplus is configured to verify 01-, 03-, 06-, and 24-h accumulated precipitation using hourly CCPA files. .. note:: - There is a problem with the valid time in the metadata for files valid from 19 - 00 UTC (i.e., files under the "00" directory). The script to pull the CCPA data from the NOAA HPSS (``scripts/exregional_get_obs_ccpa.sh``) has an example of how to account for this and organize the data into a more intuitive format. When a fix is provided, it will be accounted for in the ``exregional_get_obs_ccpa.sh`` script. + There is a problem with the valid time in the metadata for files valid from 19 - 00 UTC (i.e., files under the "00" directory) for dates up until 2021-05-04. The script to pull the CCPA data from the NOAA HPSS (``scripts/exregional_get_verif_obs.sh``) has an example of how to account for this and organize the data into a more intuitive format. ``NOHRSC_OBS_DIR``: (Default: "") User-specified location of top-level directory where NOHRSC 06- and 24-hour snowfall accumulation files (available every 6 and 12 hours respectively) used by METplus are located. This parameter needs to be set for both user-provided observations and for observations that are retrieved from the NOAA :term:`HPSS` (if the user has access) via the ``get_obs_nohrsc`` task. (This task is activated in the workflow by using the taskgroup file ``parm/wflow/verify_pre.yaml``). - METplus configuration files require the use of a predetermined directory structure and file names. If the NOHRSC files are user-provided, they need to follow the anticipated naming structure: ``{YYYYMMDD}/sfav2_CONUS_{AA}h_{YYYYMMDD}{HH}_grid184.grb2``, where AA is the 2-digit accumulation duration, and YYYYMMDD and HH are as described in the note :ref:`above `. When pulling observations from NOAA HPSS, the data retrieved will be placed in the ``NOHRSC_OBS_DIR`` directory. This path must be defind as ``//nohrsc/proc``. METplus is configured to verify 06-, and 24-h accumulated precipitation using NOHRSC files. + METplus configuration files require the use of a predetermined directory structure and file names. If the NOHRSC files are user-provided, they need to follow the anticipated naming structure: ``{YYYYMMDD}/sfav2_CONUS_{AA}h_{YYYYMMDD}{HH}_grid184.grb2``, where AA is the 2-digit accumulation duration, and YYYYMMDD and HH are as described in the note :ref:`above `. When pulling observations from NOAA HPSS, the data retrieved will be placed in the ``NOHRSC_OBS_DIR`` directory. METplus is configured to verify 06-, and 24-h accumulated precipitation using NOHRSC files. .. note:: Due to limited availability of NOHRSC observation data on NOAA HPSS, and the likelihood that snowfall acumulation verification will not be desired outside of winter cases, this verification option is currently not present in the workflow by default. In order to use it, the verification environment variable VX_FIELDS should be updated to include ``ASNOW``. This will allow the related workflow tasks to be run. ``MRMS_OBS_DIR``: (Default: "") - User-specified location of top-level directory where MRMS composite reflectivity files used by METplus are located. This parameter needs to be set for both user-provided observations and for observations that are retrieved from the NOAA :term:`HPSS` (if the user has access) via the ``get_obs_mrms`` task (activated in the workflow automatically when using the taskgroup file ``parm/wflow/verify_pre.yaml``). When pulling observations directly from NOAA HPSS, the data retrieved will be placed in this directory. Please note, this path must be defind as ``//mrms/proc``. - + User-specified location of top-level directory where MRMS composite reflectivity files used by METplus are located. This parameter needs to be set for both user-provided observations and for observations that are retrieved from the NOAA :term:`HPSS` (if the user has access) via the ``get_obs_mrms`` task (activated in the workflow automatically when using the taskgroup file ``parm/wflow/verify_pre.yaml``). When pulling observations directly from NOAA HPSS, the data retrieved will be placed in this directory. + METplus configuration files require the use of a predetermined directory structure and file names. Therefore, if the MRMS files are user-provided, they need to follow the anticipated naming structure: ``{YYYYMMDD}/MergedReflectivityQCComposite_00.50_{YYYYMMDD}-{HH}{mm}{SS}.grib2``, where YYYYMMDD and {HH}{mm}{SS} are as described in the note :ref:`above `. .. note:: - METplus is configured to look for a MRMS composite reflectivity file for the valid time of the forecast being verified; since MRMS composite reflectivity files do not always exactly match the valid time, a script (within the main script that retrieves MRMS data from the NOAA HPSS) is used to identify and rename the MRMS composite reflectivity file to match the valid time of the forecast. The script to pull the MRMS data from the NOAA HPSS has an example of the expected file-naming structure: ``scripts/exregional_get_obs_mrms.sh``. This script calls the script used to identify the MRMS file closest to the valid time: ``ush/mrms_pull_topofhour.py``. + METplus is configured to look for a MRMS composite reflectivity file for the valid time of the forecast being verified, which is why the minutes and seconds of the filename are hard-coded as "0000". Because MRMS composite reflectivity files do not typically match the valid time exactly, a script (``ush/mrms_pull_topofhour.py``) is called from within the MRMS task that identifies and renames the MRMS file nearest to the valid time to match the valid time of the forecast. This script can also be called separately for staging data independently of the workflow. ``NDAS_OBS_DIR``: (Default: "") - User-specified location of the top-level directory where NDAS prepbufr files used by METplus are located. This parameter needs to be set for both user-provided observations and for observations that are retrieved from the NOAA :term:`HPSS` (if the user has access) via the ``get_obs_ndas`` task (activated in the workflow automatically when using the taskgroup file ``parm/wflow/verify_pre.yaml``). When pulling observations directly from NOAA HPSS, the data retrieved will be placed in this directory. Please note, this path must be defined as ``//ndas/proc``. METplus is configured to verify near-surface variables hourly and upper-air variables at 00 and 12 UTC with NDAS prepbufr files. - - METplus configuration files require the use of predetermined file names. Therefore, if the NDAS files are user-provided, they need to follow the anticipated naming structure: ``prepbufr.ndas.{YYYYMMDDHH}``, where YYYYMMDDHH is as described in the note :ref:`above `. The script to pull the NDAS data from the NOAA HPSS (``scripts/exregional_get_obs_ndas.sh``) has an example of how to rename the NDAS data into a more intuitive format with the valid time listed in the file name. + User-specified location of the top-level directory where NDAS prepbufr files used by METplus are located. This parameter needs to be set for both user-provided observations and for observations that are retrieved from the NOAA :term:`HPSS` (if the user has access) via the ``get_obs_ndas`` task (activated in the workflow automatically when using the taskgroup file ``parm/wflow/verify_pre.yaml``). When pulling observations directly from NOAA HPSS, the data retrieved will be placed in this directory. METplus is configured to verify near-surface variables hourly and upper-air variables at 00 and 12 UTC with NDAS prepbufr files. + + METplus configuration files require the use of predetermined file names. Therefore, if the NDAS files are user-provided, they need to follow the anticipated naming structure: ``prepbufr.ndas.{YYYYMMDDHH}``, where YYYYMMDDHH is as described in the note :ref:`above `. The script to pull the NDAS data from the NOAA HPSS (``scripts/exregional_get_verif_obs.sh``) has an example of how to rename the NDAS data into a more intuitive format with the valid time listed in the file name. Test Directories ---------------------- @@ -322,7 +319,7 @@ Grid Generation Parameters ``GRID_GEN_METHOD``: (Default: "") This variable specifies which method to use to generate a regional grid in the horizontal plane. The values that it can take on are: - * ``"ESGgrid"``: The "ESGgrid" method will generate a regional version of the Extended Schmidt Gnomonic (ESG) grid using the map projection developed by Jim Purser of EMC (:cite:t:`Purser_2020`). "ESGgrid" is the preferred grid option. + * ``"ESGgrid"``: The "ESGgrid" method will generate a regional version of the Extended Schmidt Gnomonic (ESG) grid using the map projection developed by Jim Purser of EMC (:cite:t:`Purser_2020`). "ESGgrid" is the preferred grid option. More information on the ESG grid is available at https://github.com/ufs-community/ufs-srweather-app/wiki/Purser_UIFCW_2023.pdf. * ``"GFDLgrid"``: The "GFDLgrid" method first generates a "parent" global cubed-sphere grid. Then a portion from tile 6 of the global grid is used as the regional grid. This regional grid is referred to in the grid generation scripts as "tile 7," even though it does not correspond to a complete tile. The forecast is run only on the regional grid (i.e., on tile 7, not on tiles 1 through 6). Note that the "GFDLgrid" method is the legacy grid generation method. It is not supported in *all* predefined domains. @@ -375,12 +372,6 @@ Compiler ``COMPILER``: (Default: "intel") Type of compiler invoked during the build step. Currently, this must be set manually; it is not inherited from the build system in the ``ufs-srweather-app`` directory. Valid values: ``"intel"`` | ``"gnu"`` -Verification Parameters ---------------------------- - -``GET_OBS``: (Default: "get_obs") - Set the name of the Rocoto workflow task used to load proper module files for ``get_obs_*`` tasks. Users typically do not need to change this value. - .. _NCOModeParms: @@ -942,10 +933,10 @@ These parameters are associated with the fixed (i.e., static) files. On `Level 1 ``FIXshp``: (Default: "") System directory where the graphics shapefiles are located. On Level 1 systems, these are set within the machine files. Users on other systems will need to provide the path to the directory that contains the *Natural Earth* shapefiles. -``TOPO_DIR``: (Default: "") +``FIXorg``: (Default: "") The location on disk of the static input files used by the ``make_orog`` task (i.e., ``orog.x`` and ``shave.x``). Can be the same as ``FIXgsm``. -``SFC_CLIMO_INPUT_DIR``: (Default: "") +``FIXsfc``: (Default: "") The location on disk of the static surface climatology input fields, used by ``sfc_climo_gen``. These files are only used if the ``MAKE_SFC_CLIMO`` task is meant to run. ``SYMLINK_FIX_FILES``: (Default: true) diff --git a/docs/UsersGuide/source/DefineWorkflow.rst b/docs/UsersGuide/source/CustomizingTheWorkflow/DefineWorkflow.rst similarity index 75% rename from docs/UsersGuide/source/DefineWorkflow.rst rename to docs/UsersGuide/source/CustomizingTheWorkflow/DefineWorkflow.rst index 470d04ea95..4ea8f7052d 100644 --- a/docs/UsersGuide/source/DefineWorkflow.rst +++ b/docs/UsersGuide/source/CustomizingTheWorkflow/DefineWorkflow.rst @@ -4,21 +4,20 @@ Defining an SRW App Workflow ============================= +Many predefined workflows with optional variants exist within the Short-Range Weather Application, but the Application also includes the ability to define a new workflow from scratch. This functionality allows users to add tasks to the workflow to meet their scientific exploration needs. -Rocoto is the primary workflow manager software used by the UFS SRW App. Because the SRW App supports a variety of research and operational configurations, it is important to incorporate the ability to build a static Rocoto XML from scratch. This means a user will be able to specify exactly which tasks are included in the workflow definition from a YAML configuration file. While many predefined workflows will exist with optional variants in the SRW App, additional tasks may be added by individuals for their scientific exploration needs. - -This guide explains how the Rocoto XML is built using a Jinja2 template (`Jinja docs here `__) and structured YAML files. The YAML follows the requirements in the `Rocoto documentation `__ with a few exceptions or additions outlined in this documentation. +Rocoto is the primary workflow manager software used by the UFS SRW App. Rocoto workflows are defined in an XML file (``FV3LAM_wflow.xml``) based on parameters set during experiment generation. This section explains how the Rocoto XML is built using a Jinja2 template (`Jinja docs here `__) and structured YAML files. The YAML follows the requirements in the `Rocoto documentation `__ with a few exceptions or additions outlined in this documentation. The Jinja2 Template =================== -In previous versions of the SRW App, the Jinja2 template to create the Rocoto XML was tightly coupled to specific configuration settings of the SRW App. It was built from a limited, pre-defined set of specific tasks, defining switches for those tasks to be included or not in the rendered XML. +In previous versions of the SRW Application, the Jinja2 template to create the Rocoto XML was tightly coupled to specific configuration settings of the SRW App. It was built from a limited, pre-defined set of specific tasks, defining switches for those tasks to be included or not in the rendered XML. -Now, the Jinja2 template is entirely agnostic to Short-Range Weather Application decisions and has been developed to wrap the features of Rocoto in an extensible, configurable format. +Now, the Jinja2 template is entirely agnostic to SRW Application decisions and has been developed to wrap the features of Rocoto in an extensible, configurable format. -The ``rocoto`` section of ``config.py`` -======================================= +The ``rocoto`` section of ``config.yaml`` +========================================== The structured YAML file that defines the Rocoto XML is meant to reflect the sections required by any Rocoto XML. That structure looks like this, with some example values filled in: .. code-block:: console @@ -32,10 +31,10 @@ The structured YAML file that defines the Rocoto XML is meant to reflect the sec taskthrottle: cycledefs: groupname: - !startstopfreq ['2022102000', ‘2023102018’, ‘06:00:00’] - groupname2: - !startstopfreq ['2022102000', ‘2023102018’, ‘24:00:00’] - !startstopfreq ['2022102006', ‘2023102018’, ‘24:00:00’] + - !startstopfreq ['2022102000', ‘2023102018’, ‘06:00:00’] + groupname2: + - !startstopfreq ['2022102000', ‘2023102018’, ‘24:00:00’] + - !startstopfreq ['2022102006', ‘2023102018’, ‘24:00:00’] entities: foo: 1 bar: “/some/path” @@ -45,22 +44,22 @@ The structured YAML file that defines the Rocoto XML is meant to reflect the sec task_*: metatask_*: -In addition to the structured YAML itself, the SRW App enables additional functionality in defining a YAML file. Often, PyYAML features are introduced and documented `here `__. In the above example, the ``!startstopfreq`` is an example of a PyYAML constructor. Supported constructors will be outlined :ref:`below `. There are also examples of using PyYAML anchors and aliases in the definition of groups of tasks in the SRW App. Please see `this documentation `__ for the behavior of PyYAML anchors and aliases. - -The use of Jinja2 templates inside the values of entries allows for the reference to other keys, mathematical operations, Jinja2 control structures, and the use of user-defined filters. Here, the ``include`` filter in the ``taskgroups`` entry is a user-defined filter. The supported filters are described in a section :ref:`below `. - Under the Rocoto section, several subentries are required. They are described here using similar language as in the Rocoto documentation. ``attrs``: Any of the attributes to the ``workflow`` tag in Rocoto. This is meant to contain a nested dictionary defining any of the Rocoto-supported attributes, where the key is the name of the attribute, and the value is what Rocoto expects. -``cycledefs``: A dictionary with each key defining a group name for a ``cycledef`` tag, where the key’s value is a list of acceptable ``cycledef`` formatted strings. The PyYAML constructor ``!startstopfreq`` has been included here to help with the automated construction of a tag of that nature. The preferred option for the SRW App is to use the “start, stop, step” method. +``cycledefs``: A dictionary in which each key defines a group name for a ``cycledef`` tag; the key’s value is a list of acceptable ``cycledef`` formatted strings. The PyYAML constructor ``!startstopfreq`` has been included here to help with the automated construction of a tag of that nature. The preferred option for the SRW App is to use the “start, stop, step” method. -``entities``: A dictionary with each key defining the name of a Rocoto entity and its value. These variables are referenceable throughout the workflow with the ‘&foo;’ notation. +``entities``: A dictionary in which each key defines the name of a Rocoto entity and its value. These variables are referenceable throughout the workflow with the ``&foo;`` notation. ``log``: The path to the log file. This corresponds to the ```` tag. ``tasks``: This section is where the defined tasks and metatasks are created. This is the main portion of the workflow that will most commonly differ from experiment to experiment with different configurations. +In addition to the structured YAML itself, the SRW App enables additional functionality when defining a YAML file. Often, PyYAML features are introduced and documented `here `__. In the above example, the ``!startstopfreq`` is an example of a PyYAML constructor. Supported constructors will be outlined :ref:`below `. There are also examples of using PyYAML anchors and aliases in the definition of groups of tasks in the SRW App. Please see `this documentation `__ for the behavior of PyYAML anchors and aliases. + +The use of Jinja2 templates inside the values of entries allows for the reference to other keys, mathematical operations, Jinja2 control structures, and the use of user-defined filters. Here, the ``include`` filter in the ``taskgroups`` entry is a user-defined filter. The supported filters are described in a section :ref:`below `. + .. _tasks: The ``tasks`` Subsection @@ -68,9 +67,9 @@ The ``tasks`` Subsection ``taskgroups``: This entry is not a standard Rocoto entry. It defines a set of files that will be included to build a workflow from predefined groups of tasks. The supported groups are included under ``parm/wflow`` for the SRW App, but the paths can point to any location on your local disk. The resulting order of the tasks will be in the same order as defined in this list. The syntax for the “include” is included as a Jinja2 filter. -``task_*``: This is a section header to add a task. The task name will be whatever the section key has defined after the first underscore. For example, “task_run_fcst” will be named “run_fcst” in the resulting workflow. More information about defining a task is included :ref:`below `. +``task_*``: This is a section header to add a task. The task name will be whatever the section key has defined after the first underscore. For example, ``task_run_fcst`` will be named ``run_fcst`` in the resulting workflow. More information about defining a task is included :ref:`below `. -``metatask_*``: This is a section header to add a metatask. The metatask name will be whatever the section key has defined after the first underscore. For example “metatask_run_ensemble” will be named “run_ensemble” in the resulting workflow. More information about defining a metatask is included :ref:`below `. +``metatask_*``: This is a section header to add a metatask. The metatask name will be whatever the section key has defined after the first underscore. For example ``metatask_run_ensemble`` will be named ``run_ensemble`` in the resulting workflow. More information about defining a metatask is included :ref:`below `. .. _defining_tasks: @@ -107,7 +106,7 @@ Each task supports any of the tags that are defined in the Rocoto documentation. The following sections are constructs of the interface, while all others are direct translations to tags available in Rocoto. Any tag that allows for attributes to the XML tag will take an ``attrs`` nested dictionary entry. -``attrs``: Any of the attributes to the task tag in Rocoto. This is meant to be a subdictionary defining any of the Rocoto-supported attributes, where the key is the name of the attribute, and the value is what Rocoto expects. This might include any combination of the following: cycledefs, maxtries, throttle, or final. +``attrs``: Any of the attributes to the task tag in Rocoto. This is meant to be a subdictionary defining any of the Rocoto-supported attributes, where the key is the name of the attribute, and the value is what Rocoto expects. Attributes might include any combination of the following: cycledefs, maxtries, throttle, or final. ``envars``: A dictionary of keys that map to variable names that will be exported for the job. These will show up as the set of ```` tags in the XML. The value will be the value of the defined variable when it is exported. @@ -121,7 +120,7 @@ The dependency entry will be an arbitrarily deep nested dictionary of key, value Because we are representing these entries as a dictionary, which requires hashable keys (no repeats at the same level), some tags may need to be differentiated where XML may not differentiate at all. In these instances, it is best practice to name them something descriptive. For example, you might have multiple “or” dependencies at the same level that could be named “or_files_exist” and “or_task_ran”. This style can be adopted whether or not differentiation is needed. -The text entry on some dependencies is for those dependency tags that need the information to come between two flags, as in a data dependency. +The ``text`` entry on some dependencies is for those dependency tags that need the information to come between two flags, as in a data dependency. Otherwise, all dependencies follow the same naming conventions as defined in Rocoto with ``attrs`` dictionaries included to define any of the tag attributes that may be accepted by Rocoto. @@ -220,16 +219,16 @@ There is a specific order of precedence imposed when the SRW App loads configura #. Load ``config_defaults.yaml`` file. #. Load the user’s ``config.yaml`` file. -#. Load the workflow defaults YAML file. -#. At this point, all anchors and references will be resolved. -#. All PyYAML constructors will also be called for the data provided in that entry. +#. Load the ``default_workflow.yaml`` file. + + * At this point, all anchors and references will be resolved. + * All PyYAML constructors will also be called for the data provided in that entry. #. Call ``update_dict`` function to remove any null entries from default tasks using the PyYAML anchors. #. Load all files from the ``taskgroups:`` entry from the user’s config or from the default if not overridden. This is achieved with a call to the ``extend_yaml()`` function. #. Add the contents of the files to the ``task:`` section. #. Update the existing workflow configuration with any user-specified entries (removing the ones that are null entries). #. Add a ``jobname:`` entry to every task in the workflow definition section. - -#. Incorporate other default configuration settings from machine files, constants, etc. into the default config dictionary in memory. +#. Incorporate other default configuration settings from machine files, constants, etc. into the default configuration dictionary in memory. #. Apply all user settings last to take highest precedence. #. Call ``extend_yaml()`` to render templates that are available. NOTE: This is the one that is likely to trip up any settings that ``setup.py`` will make. References to other defaults that get changed during the course of validation may be rendered here earlier than desired. diff --git a/docs/UsersGuide/source/InputOutputFiles.rst b/docs/UsersGuide/source/CustomizingTheWorkflow/InputOutputFiles.rst similarity index 73% rename from docs/UsersGuide/source/InputOutputFiles.rst rename to docs/UsersGuide/source/CustomizingTheWorkflow/InputOutputFiles.rst index 0b6b364613..e179219e7a 100644 --- a/docs/UsersGuide/source/InputOutputFiles.rst +++ b/docs/UsersGuide/source/CustomizingTheWorkflow/InputOutputFiles.rst @@ -19,20 +19,20 @@ Initial and Boundary Condition Files The external model files needed for initializing an experiment can be obtained in a number of ways, including: - * pulled directly from `NOMADS `__ (limited timespan for data availability), - * pulled from the NOAA High Performance Storage System (HPSS) during the workflow execution (requires user access), or - * obtained and staged by the user from a different source. + * Pulled from the `SRW App Data Bucket `__, + * Pulled from the NOAA High Performance Storage System (:term:`HPSS`) during the workflow execution (requires user access), or + * Obtained and staged by the user from a different source. The data format for these files can be :term:`GRIB2` or :term:`NEMSIO`. More information on downloading and setting up the external model data can be found in :numref:`Section %s `. Once the data is set up, the end-to-end application will run the system and write output files to disk. Pre-processing (UFS_UTILS) --------------------------- -When a user generates the regional workflow as described in :numref:`Section %s `, the workflow generation script links the input data for the pre-processing utilities to the experiment directory. The pre-processing utilities use many different datasets to create grids and to generate model input datasets from the external model files. A detailed description of the input files for the pre-processing utilities can be found in the UFS_UTILS `Technical Documentation `__ and `Scientific Documentation `__. +When a user generates the SRW App workflow as described in :numref:`Section %s `, the workflow generation script links the input data for the pre-processing utilities to the experiment directory. The pre-processing utilities use many different datasets to create grids and to generate model input datasets from the external model files. A detailed description of the input files for the pre-processing utilities can be found in the UFS_UTILS `Technical Documentation `__ and `Scientific Documentation `__. UFS Weather Model ----------------- The input files for the UFS Weather Model include both static (fixed) files and grid- and date-specific files (terrain, initial conditions, boundary conditions, etc). The static fix(ed) files -must be staged by the user unless the user is running on a `Level 1/pre-configured `__ platform, in which case users can link to the existing copy of the data on their machine. See :numref:`Section %s ` for instructions. The workflow scripts link the static, grid, and date-specific files in the experiment directory. An extensive description of the input files for the Weather Model can be found in the `UFS Weather Model User's Guide `__. The namelists and configuration files for the SRW Application are created from templates by the workflow generation script, as described in :numref:`Section %s `. +must be staged by the user unless the user is running on a `Level 1/pre-configured `__ platform, in which case users can link to the existing copy of the data on their machine. (See :numref:`Section %s ` for instructions on staging the data on a new machine and :numref:`Section %s ` for data locations on Level 1 machines.) The workflow scripts link the static, grid, and date-specific files to the experiment directory. An extensive description of the input files for the Weather Model can be found in the `UFS Weather Model User's Guide `__. The namelists and configuration files for the SRW Application are created from templates by the workflow generation script, as described in :numref:`Section %s `. Unified Post Processor (UPP) ---------------------------- @@ -53,59 +53,42 @@ and are shown in :numref:`Table %s `. .. _TemplateFiles: -.. table:: Template Files for the Regional Workflow - - +-----------------------------+--------------------------------------------------------------+ - | **File Name** | **Description** | - +=============================+==============================================================+ - | data_table | :term:`Cycle-independent` file that the forecast model | - | | reads in at the start of each forecast. It is an empty file. | - | | No need to change. | - +-----------------------------+--------------------------------------------------------------+ - | diag_table.[CCPP] | File specifying the output fields of the forecast model. | - | | A different ``diag_table`` may be configured for different | - | | :term:`CCPP` suites. | - +-----------------------------+--------------------------------------------------------------+ - | field_table.[CCPP] | :term:`Cycle-independent` file that the forecast model | - | | reads in at the start of each forecast. It specifies the | - | | :term:`tracers ` that the forecast model will | - | | :term:`advect`. A different ``field_table`` may be needed | - | | for different CCPP suites. | - +-----------------------------+--------------------------------------------------------------+ - | FV3.input.yml | YAML configuration file containing the forecast model's | - | | namelist settings for various physics suites. The values | - | | specified in this file update the corresponding values in | - | | the ``input.nml`` file. This file may be modified for the | - | | specific namelist options of your experiment. | - +-----------------------------+--------------------------------------------------------------+ - | FV3LAM_wflow.xml | Rocoto XML file to run the workflow. It is filled in using | - | | the ``fill_template.py`` python script that is called in | - | | ``generate_FV3LAM_wflow.py``. | - +-----------------------------+--------------------------------------------------------------+ - | input.nml.FV3 | Namelist file for the Weather Model. | - +-----------------------------+--------------------------------------------------------------+ - | model_configure | Settings and configurations for the | - | | :term:`NUOPC`/:term:`ESMF` main component. | - +-----------------------------+--------------------------------------------------------------+ - | nems.configure | :term:`NEMS` (NOAA Environmental Modeling System) | - | | configuration file. No need to change because it is an | - | | atmosphere-only model in the SRW Application. | - +-----------------------------+--------------------------------------------------------------+ - | regional_grid.nml | Namelist settings for the code that generates an :term:`ESG` | - | | grid. | - +-----------------------------+--------------------------------------------------------------+ - | README.xml_templating.md | Instructions for Rocoto XML templating with Jinja. | - +-----------------------------+--------------------------------------------------------------+ +.. list-table:: Template Files for the SRW App Workflow + :widths: 20 50 + :header-rows: 1 + + * - File Name + - Description + * - data_table + - :term:`Cycle-independent` file that the forecast model reads in at the start of each forecast. It is an empty file. No need to change. + * - diag_table.[CCPP] + - File specifying the output fields of the forecast model. A different ``diag_table`` may be configured for different :term:`CCPP` suites. + * - field_table.[CCPP] + - :term:`Cycle-independent` file that the forecast model reads in at the start of each forecast. It specifies the :term:`tracers ` that the forecast model will :term:`advect`. A different ``field_table`` may be needed for different CCPP suites. + * - FV3.input.yml + - YAML configuration file containing the forecast model's namelist settings for various physics suites. The values specified in this file update the corresponding values in the ``input.nml`` file. This file may be modified for the specific namelist options of your experiment. + * - FV3LAM_wflow.xml + - Rocoto XML file to run the workflow. It is filled in using the ``fill_template.py`` python script that is called in ``generate_FV3LAM_wflow.py``. + * - input.nml.FV3 + - Namelist file for the Weather Model. + * - model_configure + - Settings and configurations for the :term:`NUOPC`/:term:`ESMF` main component. + * - nems.configure + - :term:`NEMS` (NOAA Environmental Modeling System) configuration file. No need to change because the usual SRW App configuration is atmosphere-only, and UFS-AQM settings handle any configuration/templating required for that configuration. + * - regional_grid.nml + - Namelist settings for the code that generates an :term:`ESG` grid. + * - README.xml_templating.md + - Instructions for Rocoto XML templating with Jinja. Additional information related to ``diag_table.[CCPP]``, ``field_table.[CCPP]``, ``input.nml.FV3``, ``model_configure``, and ``nems.configure`` can be found in the `UFS Weather Model User's Guide `__, while information on ``regional_grid.nml`` options can be found in the `UFS_UTILS Technical Documentation `__. Migratory Route of the Input Files in the Workflow ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -:numref:`Figure %s ` shows how the input files in the template directory (``ufs-srweather-app/parm``) flow to the experiment directory. First, the CCPP physics suite is specified in the configuration file. The template input files corresponding to the selected physics suite, such as ``field_table.[CCPP]`` and ``nems.configure_[CCPP]``, are copied to the experiment directory (``$EXPTDIR``). Additionally, the namelist file of the Weather Model (``input.nml``) is created from the ``input.nml.FV3`` and ``FV3.input.yml`` files by running the workflow generation script. While running the ``RUN_FCST`` task in the regional workflow as shown in :numref:`Figure %s `, the ``field_table``, ``nems.configure``, and ``input.nml`` files, located in ``$EXPTDIR``, are linked to the cycle directory (``$CYCLE_DIR``). Additionally, ``diag_table`` and ``model_configure`` are copied from the ``parm`` directory. Finally, these files are updated with the variables specified in ``var_defn.sh``. +:numref:`Figure %s ` shows how the input files in the template directory (``ufs-srweather-app/parm``) flow to the experiment directory. First, the CCPP physics suite is specified in the configuration file. The template input files corresponding to the selected physics suite, such as ``field_table.[CCPP]`` and ``nems.configure_[CCPP]``, are copied to the experiment directory (``$EXPTDIR``). Additionally, the namelist file of the Weather Model (``input.nml``) is created from the ``input.nml.FV3`` and ``FV3.input.yml`` files by running the workflow generation script. While running the ``RUN_FCST`` task in the SRW App workflow as shown in :numref:`Figure %s `, the ``field_table``, ``nems.configure``, and ``input.nml`` files, located in ``$EXPTDIR``, are linked to the cycle directory (``$CYCLE_DIR``). Additionally, ``diag_table`` and ``model_configure`` are copied from the ``parm`` directory. Finally, these files are updated with the variables specified in ``var_defn.sh``. .. _MigratoryRoute: -.. figure:: _static/SRW_wflow_input_path.png +.. figure:: https://github.com/ufs-community/ufs-srweather-app/wiki/SRW_wflow_input_path.png :alt: Flowchart showing how information from the physics suite travels from the configuration file to the setup file to the workflow generation script to the run forecast ex-script. As this information is fed from one file to the next, file paths and variables required for workflow execution are set. *Migratory Route of Input Files* @@ -115,13 +98,15 @@ Migratory Route of the Input Files in the Workflow Output Files ============== -Output files from each workflow task are written to a subdirectory within the experiment directory (``$EXPTDIR/YYYYMMDDHH``), named based on the settings in ``config.yaml``. +Output files from each workflow task are written to a subdirectory within the experiment directory (``$EXPTDIR/YYYYMMDDHH``), named based on the settings in ``config.yaml``. These files may then be used as input to future tasks. Initial and boundary condition files ------------------------------------ The external model data used by ``chgres_cube`` (as part of the pre-processing utilities) are located in the experiment directory under ``$EXPTDIR/YYYYMMDDHH/EXTRN_MDL_NAME/{for_ICS/for_LBCS}``. +.. COMMENT: This is confusing bc it sounds like these are input files, not output files. Does chgres_cube output these? In which tasks? + Pre-processing (UFS_UTILS) -------------------------- The files output by the other pre-processing utilities reside in the ``INPUT`` directory under the @@ -146,7 +131,7 @@ These output files are used as inputs for the UFS Weather Model and are describe UFS Weather Model ------------------ -As stated in :numref:`Section %s `, the workflow can be run in "community" mode or "nco" mode, which determines the location and names of the output files. Weather Model output files can be in :term:`netCDF` or :term:`NEMSIO` format. The output file format is set in the ``model_configure`` file (see :numref:`Table %s `) using the ``output_file`` variable. At this time, due to limitations in the post-processing component, only netCDF output is recommended as output for the SRW Application. +As stated in :numref:`Section %s `, the workflow can be run in "community" mode or "nco" mode, which determines the location and names of the output files. Weather Model output files can be in :term:`netCDF` or :term:`NEMSIO` format. The output file format is set in the ``model_configure`` file using the ``output_file`` variable (see :ref:`UFS WM Documentation `). At this time, due to limitations in the post-processing component, only netCDF output is recommended as output for the SRW Application. .. note:: The fully supported options for this release include running in "community" mode with netCDF-formatted output files. @@ -191,11 +176,11 @@ After creating the new flat text file to reflect the changes, users will need to .. code-block:: console USE_CUSTOM_POST_CONFIG_FILE: true - CUSTOM_POST_CONFIG_FP: + CUSTOM_POST_CONFIG_FP: /path/to/custom/postxconfig-NT-fv3lam.txt which tells the workflow to use the custom file located in the user-defined path. The path should include the filename. If ``USE_CUSTOM_POST_CONFIG_FILE`` is set to true, but the file path is not found, then an error will occur when trying to generate the SRW Application workflow. -Users may then start their experiment workflow as usual, and the UPP will use the new flat ``*.txt`` file. +After successfully generating the workflow, users may run/monitor their experiment as usual, and the UPP will use the new flat ``*.txt`` file. .. _SatelliteProducts: @@ -217,7 +202,7 @@ Modify the ``config.yaml`` file to include the following lines: .. code-block:: console USE_CRTM: true - CRTM_DIR: + CRTM_DIR: /path/to/top/crtm/dir By setting ``USE_CRTM`` to true, the workflow will use the path defined in ``CRTM_DIR`` to link the necessary coefficient files to the working directory at runtime. Otherwise, it is assumed that no satellite fields are being requested in the UPP configuration. ``CRTM_DIR`` should point to the top CRTM directory where the fix files are located. @@ -245,13 +230,15 @@ Static files are available in the `"fix" directory `__ of the SRW Data Bucket using the ``wget`` command for each required file. A list of ``wget`` commands with links is provided :ref:`here ` for the release v2.1.0 fix file data. Users will need to create an appropriate directory structure for the files when downloading them individually. The best solution is to download the files into directories that mirror the structure of the `Data Bucket `__. -The environment variables ``FIXgsm``, ``TOPO_DIR``, and ``SFC_CLIMO_INPUT_DIR`` indicate the path to the directories where the static files are located. After downloading the experiment data, users must set the paths to the files in ``config.yaml``. Add the following code to the ``task_run_fcst:`` section of the ``config.yaml`` file, and alter the variable paths accordingly: +.. COMMENT: Update release file list above for next SRW release. + +The environment variables ``FIXgsm``, ``FIXorg``, and ``FIXsfc`` indicate the path to the directories where the static files are located. After downloading the experiment data, users must set the paths to the files in ``config.yaml``. Add the following code to the ``task_run_fcst:`` section of the ``config.yaml`` file, and alter the variable paths accordingly: .. code-block:: console - FIXgsm: - TOPO_DIR: - SFC_CLIMO_INPUT_DIR: + FIXgsm: /path/to/fix/fix_am + FIXorg: /path/to/fix/fix_orog + FIXsfc: /path/to/fix/sfc_climo/ .. _InitialConditions: @@ -279,14 +266,12 @@ The paths to ``EXTRN_MDL_SOURCE_BASEDIR_ICS`` and ``EXTRN_MDL_SOURCE_BASEDIR_LBC task_get_extrn_ics: USE_USER_STAGED_EXTRN_FILES: true - EXTRN_MDL_SOURCE_BASEDIR_ICS: - EXTRN_MDL_DATA_STORES: disk + EXTRN_MDL_SOURCE_BASEDIR_ICS: /path/to/ufs-srweather-app/input_model_data/FV3GFS/grib2/YYYYMMDDHH task_get_extrn_lbcs: USE_USER_STAGED_EXTRN_FILES: true - EXTRN_MDL_SOURCE_BASEDIR_LBCS: - EXTRN_MDL_DATA_STORES: disk + EXTRN_MDL_SOURCE_BASEDIR_LBCS: /path/to/ufs-srweather-app/input_model_data/FV3GFS/grib2/YYYYMMDDHH -The two ``EXTRN_MDL_SOURCE_BASEDIR_*CS`` variables describe where the :term:`IC ` and :term:`LBC ` file directories are located, respectively. For ease of reusing ``config.yaml`` across experiments, it is recommended that users set up the raw :term:`IC/LBC ` file paths to include the model name (e.g., FV3GFS, GEFS, GDAS, NAM, RAP, HRRR), data format (e.g., grib2, nemsio), and date (in ``YYYYMMDDHH`` format). For example: ``/path-to/input_model_data/FV3GFS/grib2/2019061518/``. While there is flexibility to modify these settings, this structure will provide the most reusability for multiple dates when using the SRW Application workflow. +The two ``EXTRN_MDL_SOURCE_BASEDIR_*CS`` variables describe where the :term:`IC ` and :term:`LBC ` file directories are located, respectively. For ease of reusing ``config.yaml`` across experiments, it is recommended that users set up the raw :term:`IC/LBC ` file paths to include the model name (e.g., FV3GFS, GEFS, GDAS, NAM, RAP, HRRR), data format (e.g., grib2, nemsio), and date (in ``YYYYMMDDHH`` format). For example: ``/path/to/input_model_data/FV3GFS/grib2/2019061518/``. While there is flexibility to modify these settings, this structure will provide the most reusability for multiple dates when using the SRW Application workflow. When files are pulled from NOAA :term:`HPSS` (rather than downloaded from the data bucket), the naming convention looks something like: @@ -388,8 +373,8 @@ It is recommended that users have a separate directory for each file format if t .. code-block:: console - /path-to/input_model_data/FV3GFS/grib2/YYYYMMDDHH - /path-to/input_model_data/FV3GFS/nemsio/YYYYMMDDHH + /path/to/input_model_data/FV3GFS/grib2/YYYYMMDDHH + /path/to/input_model_data/FV3GFS/nemsio/YYYYMMDDHH Additionally, users must set the following environment variables if they plan to use GRIB2-formatted files for FV3GFS: diff --git a/docs/UsersGuide/source/LAMGrids.rst b/docs/UsersGuide/source/CustomizingTheWorkflow/LAMGrids.rst similarity index 95% rename from docs/UsersGuide/source/LAMGrids.rst rename to docs/UsersGuide/source/CustomizingTheWorkflow/LAMGrids.rst index 4c69ec0e21..e4ddaa29b1 100644 --- a/docs/UsersGuide/source/LAMGrids.rst +++ b/docs/UsersGuide/source/CustomizingTheWorkflow/LAMGrids.rst @@ -58,7 +58,7 @@ The 3-km CONUS domain is ideal for running the ``FV3_RRFS_v1beta`` physics suite .. _RRFS_CONUS_3km: -.. figure:: _static/RRFS_CONUS_3km.sphr.native_wrtcmp.png +.. figure:: https://github.com/ufs-community/ufs-srweather-app/wiki/RRFS_CONUS_3km.sphr.native_wrtcmp.png :alt: Map of the continental United States 3 kilometer domain. The computational grid boundaries appear in red and the write-component grid appears just inside the computational grid boundaries in blue. *The boundary of the RRFS_CONUS_3km computational grid (red) and corresponding write-component grid (blue).* @@ -74,7 +74,7 @@ Predefined SUBCONUS Grid Over Indianapolis .. _SUBCONUS_Ind_3km: -.. figure:: _static/SUBCONUS_Ind_3km.png +.. figure:: https://github.com/ufs-community/ufs-srweather-app/wiki/SUBCONUS_Ind_3km.png :alt: Map of Indiana and portions of the surrounding states. The map shows the boundaries of the continental United States sub-grid centered over Indianapolis. The computational grid boundaries appear in red and the write-component grid appears just inside the computational grid boundaries in blue. *The boundary of the SUBCONUS_Ind_3km computational grid (red) and corresponding write-component grid (blue).* @@ -86,7 +86,7 @@ Predefined 13-km Grid .. _RRFS_CONUS_13km: -.. figure:: _static/RRFS_CONUS_13km.sphr.native_wrtcmp.png +.. figure:: https://github.com/ufs-community/ufs-srweather-app/wiki/RRFS_CONUS_13km.sphr.native_wrtcmp.png :alt: Map of the continental United States 13 kilometer domain. The computational grid boundaries appear in red and the write-component grid appears just inside the computational grid boundaries in blue. *The boundary of the RRFS_CONUS_13km computational grid (red) and corresponding write-component grid (blue).* @@ -98,7 +98,7 @@ Predefined 25-km Grid .. _RRFS_CONUS_25km: -.. figure:: _static/RRFS_CONUS_25km.sphr.native_wrtcmp.png +.. figure:: https://github.com/ufs-community/ufs-srweather-app/wiki/RRFS_CONUS_25km.sphr.native_wrtcmp.png :alt: Map of the continental United States 25 kilometer domain. The computational grid boundaries appear in red and the write-component grid appears just inside the computational grid boundaries in blue. *The boundary of the RRFS_CONUS_25km computational grid (red) and corresponding write-component grid (blue).* @@ -121,7 +121,7 @@ scripts that handle the workflow and experiment generation (see :numref:`Figure With those caveats in mind, this section provides instructions for adding a new predefined grid to the FV3-LAM workflow that will be generated using the "ESGgrid" method (i.e., using the ``regional_esg_grid`` code -in the `UFS_UTILS `__ repository, where ESG stands for "Extended Schmidt Gnomonic"). We assume here that the grid to be generated covers a domain that (1) does not contain either of the poles and (2) does not cross the -180 deg --> +180 deg discontinuity in longitude near the international date line. Instructions for domains that do not have these restrictions will be provided in a future release. +in the `UFS_UTILS `__ repository, where ESG stands for "Extended Schmidt Gnomonic"). We assume here that the grid to be generated covers a domain that (1) does not contain either of the poles and (2) does not cross the -180 deg --> +180 deg discontinuity in longitude near the international date line. More information on the ESG grid is available `here `__. Instructions for domains that do not have these restrictions will be provided in a future release. The steps to add such a grid to the workflow are as follows: diff --git a/docs/UsersGuide/source/TemplateVars.rst b/docs/UsersGuide/source/CustomizingTheWorkflow/TemplateVars.rst similarity index 96% rename from docs/UsersGuide/source/TemplateVars.rst rename to docs/UsersGuide/source/CustomizingTheWorkflow/TemplateVars.rst index 2677888f5a..f22d915a62 100644 --- a/docs/UsersGuide/source/TemplateVars.rst +++ b/docs/UsersGuide/source/CustomizingTheWorkflow/TemplateVars.rst @@ -5,7 +5,7 @@ Template Variables ====================== The SRW App's experiment configuration system supports the use of template variables -in ``config_defaults.yaml`` and ``config.yaml``. A template variable --- or "template" --- is an experiment configuration variable that contains references to values of other variables. +in ``config_defaults.yaml`` and ``config.yaml``. A template variable is an experiment configuration variable that contains references to values of other variables. These references are **not** set to the values of the referenced variables (or "expanded") when the experiment's variable definitions file (``var_defns.sh``) is generated or sourced. Instead, they are expanded and evaluated **at run time** when bash's ``eval`` command is used on the template. diff --git a/docs/UsersGuide/source/CustomizingTheWorkflow/index.rst b/docs/UsersGuide/source/CustomizingTheWorkflow/index.rst new file mode 100644 index 0000000000..680021f5cf --- /dev/null +++ b/docs/UsersGuide/source/CustomizingTheWorkflow/index.rst @@ -0,0 +1,12 @@ +Customizing the Workflow +=========================== + +.. toctree:: + :maxdepth: 3 + + ConfigWorkflow + InputOutputFiles + LAMGrids + DefineWorkflow + TemplateVars + diff --git a/docs/UsersGuide/source/FAQ.rst b/docs/UsersGuide/source/FAQ.rst deleted file mode 100644 index 35d60797ec..0000000000 --- a/docs/UsersGuide/source/FAQ.rst +++ /dev/null @@ -1,173 +0,0 @@ -.. _FAQ: - -**** -FAQ -**** - -* :ref:`How do I define an experiment name? ` -* :ref:`How do I change the Physics Suite Definition File (SDF)? ` -* :ref:`How do I change the grid? ` -* :ref:`How do I turn on/off the cycle-independent workflow tasks? ` -* :ref:`How do I restart a DEAD task? ` -* :ref:`How can I clean up the SRW App code if something went wrong? ` -* :ref:`How do I run a new experiment? ` - -.. _DefineExptName: - -==================================== -How do I define an experiment name? -==================================== - -The name of the experiment is set in the ``workflow:`` section of the ``config.yaml`` file using the variable ``EXPT_SUBDIR``. -See :numref:`Section %s ` and/or :numref:`Section %s ` for more details. - -.. _ChangePhysics: - -========================================================= -How do I change the Physics Suite Definition File (SDF)? -========================================================= - -The SDF is set in the ``workflow:`` section of the ``config.yaml`` file using the variable ``CCPP_PHYS_SUITE``. The five supported physics suites for the SRW Application as of the v2.1.0 release are: - -.. code-block:: console - - FV3_GFS_v16 - FV3_RRFS_v1beta - FV3_HRRR - FV3_WoFS_v0 - FV3_RAP - -When users run the ``generate_FV3LAM_wflow.py`` script, the SDF file is copied from its location in the forecast -model directory to the experiment directory ``$EXPTDIR``. For more information on the :term:`CCPP` physics suite parameters, see :numref:`Section %s `. - -.. _ChangeGrid: - -=========================== -How do I change the grid? -=========================== - -To change the predefined grid, modify the ``PREDEF_GRID_NAME`` variable in the ``task_run_fcst:`` section of the ``config.yaml`` script (see :numref:`Section %s ` for details on creating and modifying the ``config.yaml`` file). The four supported predefined grids as of the SRW Application v2.1.0 release are: - -.. code-block:: console - - RRFS_CONUS_3km - RRFS_CONUS_13km - RRFS_CONUS_25km - SUBCONUS_Ind_3km - -However, users can choose from a variety of predefined grids listed in :numref:`Section %s `. An option also exists to create a user-defined grid, with information available in :numref:`Chapter %s `. However, the user-defined grid option is not fully supported as of the v2.1.0 release and is provided for informational purposes only. - -.. _CycleInd: - -=========================================================== -How do I turn on/off the cycle-independent workflow tasks? -=========================================================== - -The first three pre-processing tasks ``make_grid``, ``make_orog``, and ``make_sfc_climo`` -are :term:`cycle-independent`, meaning that they only need to be run once per experiment. If the -grid, orography, and surface climatology files that these tasks generate are already -available (e.g., from a previous experiment that used the same grid as the current experiment), then -these tasks can be skipped, and the workflow can use those pre-generated files. This -can be done by adding the following parameters to the appropriate sections of the ``config.yaml`` script before running ``generate_FV3LAM_wflow.py``: - -.. code-block:: console - - workflow_switches: - RUN_TASK_MAKE_GRID: false - RUN_TASK_MAKE_OROG: false - RUN_TASK_MAKE_SFC_CLIMO: false - task_make_grid: - GRID_DIR: /path/to/directory/containing/grid/files - task_make_orog: - OROG_DIR: /path/to/directory/containing/orography/files - task_make_sfc_climo: - SFC_CLIMO_DIR: /path/to/directory/containing/surface/climatology/files - -The ``RUN_TASK_MAKE_GRID``, ``RUN_TASK_MAKE_OROG``, and ``RUN_TASK_MAKE_SFC_CLIMO`` flags disable their respective tasks. ``GRID_DIR``, ``OROG_DIR``, and ``SFC_CLIMO_DIR`` -specify the directories where pre-generated grid, orography, and surface climatology files are located (all -three sets of files *may* be placed in the same directory location). By default, the ``RUN_TASK_MAKE_*`` -flags are set to true in ``config_defaults.yaml``. This means that the workflow will -run the ``make_grid``, ``make_orog``, and ``make_sfc_climo`` tasks by default. - -.. _RestartTask: - -============================= -How do I restart a DEAD task? -============================= - -On platforms that utilize Rocoto workflow software (such as NCAR's Cheyenne machine), if something goes wrong with the workflow, a task may end up in the DEAD state: - -.. code-block:: console - - rocotostat -w FV3SAR_wflow.xml -d FV3SAR_wflow.db -v 10 - CYCLE TASK JOBID STATE EXIT STATUS TRIES DURATION - ================================================================================= - 201906151800 make_grid 9443237 QUEUED - 0 0.0 - 201906151800 make_orog - - - - - - 201906151800 make_sfc_climo - - - - - - 201906151800 get_extrn_ics 9443293 DEAD 256 3 5.0 - -This means that the dead task has not completed successfully, so the workflow has stopped. Once the issue -has been identified and fixed (by referencing the log files in ``$EXPTDIR/log``), users can re-run the failed task using the ``rocotorewind`` command: - -.. code-block:: console - - rocotorewind -w FV3LAM_wflow.xml -d FV3LAM_wflow.db -v 10 -c 201906151800 -t get_extrn_ics - -where ``-c`` specifies the cycle date (first column of rocotostat output) and ``-t`` represents the task name -(second column of rocotostat output). After using ``rocotorewind``, the next time ``rocotorun`` is used to -advance the workflow, the job will be resubmitted. - -.. _CleanUp: - -=============================================================== -How can I clean up the SRW App code if something went wrong? -=============================================================== - -The ``ufs-srweather-app`` repository contains a ``devclean.sh`` convenience script. This script can be used to clean up code if something goes wrong when checking out externals or building the application. To view usage instructions and to get help, run with the ``-h`` flag: - -.. code-block:: console - - ./devclean.sh -h - -To remove the ``build`` directory, run: - -.. code-block:: console - - ./devclean.sh --remove - -To remove all build artifacts (including ``build``, ``exec``, ``lib``, and ``share``), run: - -.. code-block:: console - - ./devclean.sh --clean - OR - ./devclean.sh -a - -To remove external submodules, run: - -.. code-block:: console - - ./devclean.sh --sub-modules - -Users will need to check out the external submodules again before building the application. - -In addition to the options above, many standard terminal commands can be run to remove unwanted files and directories (e.g., ``rm -rf expt_dirs``). A complete explanation of these options is beyond the scope of this User's Guide. - -.. _NewExpt: - -================================== -How can I run a new experiment? -================================== - -To run a new experiment at a later time, users need to rerun the commands in :numref:`Section %s ` that reactivate the regional workflow python environment: - -.. code-block:: console - - source - module use - module load wflow_ - -Follow any instructions output by the console. - -Then, users can configure a new experiment by updating the environment variables in ``config.yaml`` to reflect the desired experiment configuration. Detailed instructions can be viewed in :numref:`Section %s `. Parameters and valid values are listed in :numref:`Chapter %s `. After adjusting the configuration file, generate the new experiment by running ``./generate_FV3LAM_wflow.py``. Check progress by navigating to the ``$EXPTDIR`` and running ``rocotostat -w FV3LAM_wflow.xml -d FV3LAM_wflow.db -v 10``. diff --git a/docs/UsersGuide/source/Introduction.rst b/docs/UsersGuide/source/Introduction.rst deleted file mode 100644 index d8dd8e68a8..0000000000 --- a/docs/UsersGuide/source/Introduction.rst +++ /dev/null @@ -1,484 +0,0 @@ -.. _Introduction: - -============== -Introduction -============== - -The Unified Forecast System (:term:`UFS`) is a community-based, coupled, comprehensive Earth modeling system. NOAA's operational model suite for numerical weather prediction (:term:`NWP`) is quickly transitioning to the UFS from a number of different modeling systems. The UFS enables research, development, and contribution opportunities within the broader :term:`Weather Enterprise` (including government, industry, and academia). For more information about the UFS, visit the `UFS Portal `__. - -The UFS includes `multiple applications `__ that support different forecast durations and spatial domains. This documentation describes the UFS Short-Range Weather (SRW) Application, which targets predictions of atmospheric behavior on a limited spatial domain and on time scales from minutes to several days. The SRW Application v2.1.0 release includes a prognostic atmospheric model, pre- and post-processing, and a community workflow for running the system end-to-end. These components are documented within this User's Guide and supported through the `GitHub Discussions `__ forum. New and improved capabilities for the v2.0.0 release included the addition of a verification package (METplus) for both deterministic and ensemble simulations and support for four stochastically perturbed physics schemes. Additions for the v2.1.0 release included: - - * Bug fixes since the v2.0.0 release - * Conversion to a Python workflow (from the former shell workflow) - * Improved container support, including the option to run across compute nodes using Rocoto (see :numref:`Chapter %s `) - * Updates to :term:`CCPP` that target the top of the ``main`` branch (which is ahead of CCPP v6.0.0). See :ref:`this page ` for a detailed summary of updates. - * Support for the :term:`UPP` inline post option (see :ref:`here `) - * Addition of a multi-purpose code clean-up script (``devclean.sh``) (see :numref:`Section %s `) - * Documentation updates to reflect the changes above - -This documentation provides: - - * A :ref:`Quick Start Guide ` designed for use on `Level 1 systems `__ or as an overview of the workflow - * A :ref:`Container-Based Quick Start Guide ` for running the SRW Application in a container - * Detailed chapters on :ref:`building ` and :ref:`running ` the SRW App on any supported platform - * An overview of the :ref:`release components ` and details on how to customize or modify different portions of the workflow - -The SRW App v2.1.0 citation is as follows and should be used when presenting results based on research conducted with the App: - -UFS Development Team. (2022, Nov. 17). Unified Forecast System (UFS) Short-Range Weather (SRW) Application (Version v2.1.0). Zenodo. https://doi.org/10.5281/zenodo.7277602 - -How to Use This Document -======================== - -This guide instructs both novice and experienced users on downloading, building, and running the SRW Application. Please post questions in the `GitHub Discussions `__ forum. - -.. code-block:: console - - Throughout the guide, this presentation style indicates shell commands and options, code examples, etc. - -Variables presented as ``AaBbCc123`` in this User's Guide typically refer to variables in scripts, names of files, or directories. - -File paths and code that include angle brackets (e.g., ``build__``) indicate that users should insert options appropriate to their SRW App configuration (e.g., ``build_orion_intel``). - -.. hint:: - * To get started with the SRW App, users have a few options: - - #. View :numref:`Chapter %s ` for a quick overview of the workflow steps. - #. To build the application in a container, which provides a more uniform work environment, users can refer to the :ref:`Container-Based Quick Start Guide `. - #. For detailed instructions on building and running the SRW App, users can refer to :numref:`Chapter %s: Building the SRW App ` and :numref:`Chapter %s: Running the SRW App `. - - * For background information on the SRW App code repositories and directory structure, see :numref:`Section %s ` below. - * For an outline of SRW App components, see section :numref:`Section %s ` below or refer to :numref:`Chapter %s ` for a more in-depth treatment. - - -.. _SRWPrerequisites: - -Prerequisites for Using the SRW Application -=============================================== - -Background Knowledge Prerequisites --------------------------------------- - -The instructions in this documentation assume that users have certain background knowledge: - -* Familiarity with LINUX/UNIX systems -* Command line basics -* System configuration knowledge (e.g., compilers, environment variables, paths, etc.) -* Numerical Weather Prediction (concepts of parameterizations: physical, microphysical, convective) -* Meteorology (in particular, meteorology at the scales being predicted: 25km, 13km, and 3km resolutions) - -Additional background knowledge in the following areas could be helpful: - -* High-Performance Computing (HPC) Systems (for those running the SRW App on an HPC system) -* Programming (particularly Python) for those interested in contributing to the SRW App code -* Creating an SSH Tunnel to access HPC systems from the command line -* Containerization -* Workflow Managers/Rocoto - - -Software/Operating System Requirements ------------------------------------------ -The UFS SRW Application has been designed so that any sufficiently up-to-date machine with a UNIX-based operating system should be capable of running the application. SRW App `Level 1 & 2 systems `__ already have these prerequisites installed. However, users working on other systems must ensure that the following requirements are installed on their system: - -**Minimum Platform Requirements:** - -* POSIX-compliant UNIX-style operating system - -* >82 GB disk space - - * 53 GB input data for a standard collection of global data, or "fix" file data (topography, climatology, observational data) for a short 12-hour test forecast on the :term:`CONUS` 25km domain. See data download instructions in :numref:`Section %s `. - * 8 GB for full :term:`HPC-Stack` installation - * 3 GB for ``ufs-srweather-app`` installation - * 1 GB for boundary conditions for a short 12-hour test forecast on the CONUS 25km domain. See data download instructions in :numref:`Section %s `. - * 17 GB for a 12-hour test forecast on the CONUS 25km domain, with model output saved hourly. - -* Fortran compiler released since 2018 - - * gfortran v9+ or ifort v18+ are the only ones tested, but others may work. - -* C and C++ compilers compatible with the Fortran compiler - - * gcc v9+, ifort v18+, and clang v9+ (macOS, native Apple clang, LLVM clang, GNU) have been tested - -* Python v3.6+, including prerequisite packages ``jinja2``, ``pyyaml``, and ``f90nml`` - - * Python packages ``scipy``, ``matplotlib``, ``pygrib``, ``cartopy``, and ``pillow`` are required for users who would like to use the provided graphics scripts. - -* Perl 5 - -* git v2.12+ - -* curl - -* wget - -* Lmod - - -The following software is also required to run the SRW Application, but the :term:`HPC-Stack` (which contains the software libraries necessary for building and running the SRW App) can be configured to build these requirements: - -* CMake v3.20+ - -* :term:`MPI` (MPICH, OpenMPI, or other implementation) - - * Only **MPICH** or **OpenMPI** can be built with HPC-Stack. Other implementations must be installed separately by the user (if desired). - -For MacOS systems, some additional software packages are needed. When possible, it is recommended that users install and/or upgrade this software (along with software listed above) using the `Homebrew `__ package manager for MacOS. See :doc:`HPC-Stack Documentation: Chapter 3 ` and :numref:`Chapter %s ` for further guidance on installing these prerequisites on MacOS. - -* bash v4.x -* GNU compiler suite v11 or higher with gfortran -* cmake -* make -* coreutils -* gsed - -Optional but recommended prerequisites for all systems: - -* Conda for installing/managing Python packages -* Bash v4+ -* Rocoto Workflow Management System (1.3.1) -* Python packages ``scipy``, ``matplotlib``, ``pygrib``, ``cartopy``, and ``pillow`` for graphics - -.. _ComponentsOverview: - -SRW App Components Overview -============================== - -Pre-Processor Utilities and Initial Conditions ------------------------------------------------- - -The SRW Application includes a number of pre-processing utilities that initialize and prepare the model. Tasks include generating a regional grid along with :term:`orography` and surface climatology files for that grid. Additional information about the pre-processor utilities can be found in :numref:`Chapter %s `, in the `UFS_UTILS Technical Documentation `__, and in the `UFS_UTILS Scientific Documentation `__. - -Forecast Model ------------------ - -Atmospheric Model -^^^^^^^^^^^^^^^^^^^^^^ - -The prognostic atmospheric model in the UFS SRW Application is the Finite-Volume Cubed-Sphere -(:term:`FV3`) dynamical core configured with a Limited Area Model (:term:`LAM`) capability (:cite:t:`BlackEtAl2021`). The :term:`dynamical core` is the computational part of a model that solves the equations of fluid motion. A User's Guide for the UFS :term:`Weather Model` can be found `here `__. - -Common Community Physics Package -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -The `Common Community Physics Package `__ (:term:`CCPP`) supports interoperable atmospheric physics and land surface model options. Atmospheric physics are a set of numerical methods describing small-scale processes such as clouds, turbulence, radiation, and their interactions. The most recent SRW App release includes four supported physics suites. - -Data Format -^^^^^^^^^^^^^^^^^^^^^^ - -The SRW App supports the use of external model data in :term:`GRIB2`, :term:`NEMSIO`, and :term:`netCDF` format when generating initial and boundary conditions. The UFS Weather Model ingests initial and lateral boundary condition files produced by :term:`chgres_cube`. - - -Unified Post-Processor (UPP) --------------------------------- - -The `Unified Post Processor `__ (:term:`UPP`) processes raw output from a variety of numerical weather prediction (:term:`NWP`) models. In the SRW App, it converts data output from netCDF format to GRIB2 format. The UPP can also be used to compute a variety of useful diagnostic fields, as described in the `UPP User's Guide `__. - -METplus Verification Suite ------------------------------- - -The Model Evaluation Tools (MET) package is a set of statistical verification tools developed by the `Developmental Testbed Center `__ (DTC) for use by the :term:`NWP` community to help them assess and evaluate the performance of numerical weather predictions. MET is the core component of the enhanced METplus verification framework. The suite also includes the associated database and display systems called METviewer and METexpress. METplus spans a wide range of temporal and spatial scales. It is intended to be extensible through additional capabilities developed by the community. More details about METplus can be found in :numref:`Chapter %s ` and on the `METplus website `__. - -Build System and Workflow ----------------------------- - -The SRW Application has a portable CMake-based build system that packages together all the components required to build the SRW Application. Once built, users can generate a Rocoto-based workflow that will run each task in the proper sequence (see the `Rocoto documentation `__ for more on workflow management). Individual workflow tasks can also be run in a stand-alone, command line fashion. - -The SRW Application allows for configuration of various elements of the workflow. For example, users can modify the parameters of the atmospheric model, such as start and end dates, duration, time step, and the physics suite used for the simulation. More information on how to do this is available in :numref:`Section %s `. - -The SRW Application has been tested on a variety of platforms widely used by researchers, including NOAA High-Performance Computing (HPC) systems (e.g., Hera, Orion), cloud environments, and generic Linux and MacOS systems. Four `levels of support `__ have been defined for the SRW Application. Preconfigured (Level 1) systems already have the required external libraries available in a central location (via :term:`HPC-Stack`). The SRW Application is expected to build and run out-of-the-box on these systems, and users can :ref:`download the SRW App code ` without first installing prerequisites. On other platforms (Levels 2-4), the SRW App can be :ref:`run within a container ` that includes the prerequisite software; otherwise, the required libraries will need to be installed as part of the :ref:`SRW Application build ` process. Once these prerequisite libraries are installed, applications and models should build and run successfully. However, users may need to perform additional troubleshooting on Level 3 or 4 systems since little or no pre-release testing has been conducted on these systems. - - - -.. _SRWStructure: - -Code Repositories and Directory Structure -========================================= - -.. _HierarchicalRepoStr: - -Hierarchical Repository Structure ------------------------------------ -The :term:`umbrella repository` for the SRW Application is named ``ufs-srweather-app`` and is available on GitHub at https://github.com/ufs-community/ufs-srweather-app. An umbrella repository is a repository that houses external code, called "externals," from additional repositories. The SRW Application includes the ``manage_externals`` tool and a configuration file called ``Externals.cfg``, which tags the appropriate versions of the external repositories associated with the SRW App (see :numref:`Table %s `). - -.. _top_level_repos: - -.. table:: List of top-level repositories that comprise the UFS SRW Application - - +---------------------------------+---------------------------------------------------------+ - | **Repository Description** | **Authoritative repository URL** | - +=================================+=========================================================+ - | Umbrella repository for the UFS | https://github.com/ufs-community/ufs-srweather-app | - | Short-Range Weather Application | | - +---------------------------------+---------------------------------------------------------+ - | Repository for | https://github.com/ufs-community/ufs-weather-model | - | the UFS Weather Model | | - +---------------------------------+---------------------------------------------------------+ - | Repository for UFS utilities, | https://github.com/ufs-community/UFS_UTILS | - | including pre-processing, | | - | chgres_cube, and more | | - +---------------------------------+---------------------------------------------------------+ - | Repository for the Unified Post | https://github.com/NOAA-EMC/UPP | - | Processor (UPP) | | - +---------------------------------+---------------------------------------------------------+ - -The UFS Weather Model contains a number of sub-repositories, which are documented `here `__. - -.. note:: - The prerequisite libraries (including NCEP Libraries and external libraries) are not included in the UFS SRW Application repository. The `HPC-Stack `__ repository assembles these prerequisite libraries. The HPC-Stack has already been built on `preconfigured (Level 1) platforms `__. However, it must be built on other systems. See the :doc:`HPC-Stack Documentation ` for details on installing the HPC-Stack. - - -.. _TopLevelDirStructure: - -Directory Structure ----------------------- -The ``ufs-srweather-app`` :term:`umbrella repository` structure is determined by the ``local_path`` settings contained within the ``Externals.cfg`` file. After ``manage_externals/checkout_externals`` is run (see :numref:`Section %s `), the specific GitHub repositories described in :numref:`Table %s ` are cloned into the target subdirectories shown below. Directories that will be created as part of the build process appear in parentheses and will not be visible until after the build is complete. Some directories have been removed for brevity. - -.. code-block:: console - - ufs-srweather-app - ├── (build) - ├── docs - │ └── UsersGuide - ├── etc - ├── (exec) - ├── (include) - ├── jobs - ├── (lib) - ├── manage_externals - ├── modulefiles - ├── parm - ├── (share) - ├── scripts - ├── sorc - │ ├── CMakeLists.txt - │ ├── (gsi) - │ ├── (rrfs_utl) - │ ├── (UPP) - │ │ ├── parm - │ │ └── sorc - │ │ └── ncep_post.fd - │ ├── (UFS_UTILS) - │ │ ├── sorc - │ │ │ ├── chgres_cube.fd - │ │ │ ├── fre-nctools.fd - │ │ │ ├── grid_tools.fd - │ │ │ ├── orog_mask_tools.fd - │ │ │ └── sfc_climo_gen.fd - │ │ └── ush - │ └── (ufs-weather-model) - │ └── FV3 - │ ├── atmos_cubed_sphere - │ └── ccpp - ├── tests/WE2E - ├── ush - │ ├── bash_utils - │ ├── machine - │ ├── Python - │ ├── python_utils - │ ├── test_data - │ └── wrappers - └── versions - -SRW App SubDirectories -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -:numref:`Table %s ` describes the contents of the most important subdirectories. :numref:`Table %s ` provides an in-depth explanation of the ``ufs-srweather-app`` directories. - -.. _Subdirectories: - -.. table:: Subdirectories of the regional workflow - - +-------------------------+----------------------------------------------------+ - | **Directory Name** | **Description** | - +=========================+====================================================+ - | jobs | J-job scripts launched by Rocoto | - +-------------------------+----------------------------------------------------+ - | modulefiles | Files used to load modules needed for building and | - | | running the workflow | - +-------------------------+----------------------------------------------------+ - | scripts | Scripts launched by the J-jobs | - +-------------------------+----------------------------------------------------+ - | tests | Tests for baseline experiment configurations | - +-------------------------+----------------------------------------------------+ - | ush | Utility scripts used by the workflow | - +-------------------------+----------------------------------------------------+ - -.. _ExperimentDirSection: - -Experiment Directory Structure --------------------------------- -When the user generates an experiment using the ``generate_FV3LAM_wflow.py`` script (:numref:`Step %s `), a user-defined experiment directory (``$EXPTDIR``) is created based on information specified in the ``config.yaml`` file. :numref:`Table %s ` shows the contents of the experiment directory before running the experiment workflow. - -.. _ExptDirStructure: - -.. table:: Files and subdirectory initially created in the experiment directory - :widths: 33 67 - - +---------------------------+--------------------------------------------------------------------------------------------------------------+ - | **File Name** | **Description** | - +===========================+==============================================================================================================+ - | config.yaml | User-specified configuration file, see :numref:`Section %s ` | - +---------------------------+--------------------------------------------------------------------------------------------------------------+ - | data_table | :term:`Cycle-independent` input file (empty) | - +---------------------------+--------------------------------------------------------------------------------------------------------------+ - | field_table | :term:`Tracers ` in the `forecast model | - | | `__ | - +---------------------------+--------------------------------------------------------------------------------------------------------------+ - | FV3LAM_wflow.xml | Rocoto XML file to run the workflow | - +---------------------------+--------------------------------------------------------------------------------------------------------------+ - | input.nml | :term:`Namelist` for the `UFS Weather Model | - | | `__ | - +---------------------------+--------------------------------------------------------------------------------------------------------------+ - | launch_FV3LAM_wflow.sh | Symlink to the ``ufs-srweather-app/ush/launch_FV3LAM_wflow.sh`` shell script, | - | | which can be used to (re)launch the Rocoto workflow. | - | | Each time this script is called, it appends information to a log | - | | file named ``log.launch_FV3LAM_wflow``. | - +---------------------------+--------------------------------------------------------------------------------------------------------------+ - | log.generate_FV3LAM_wflow | Log of the output from the experiment generation script | - | | (``generate_FV3LAM_wflow.py``) | - +---------------------------+--------------------------------------------------------------------------------------------------------------+ - | nems.configure | See `NEMS configuration file | - | | `__ | - +---------------------------+--------------------------------------------------------------------------------------------------------------+ - | suite_{CCPP}.xml | :term:`CCPP` suite definition file (:term:`SDF`) used by the forecast model | - +---------------------------+--------------------------------------------------------------------------------------------------------------+ - | var_defns.sh | Shell script defining the experiment parameters. It contains all | - | | of the primary parameters specified in the default and | - | | user-specified configuration files plus many secondary parameters | - | | that are derived from the primary ones by the experiment | - | | generation script. This file is sourced by various other scripts | - | | in order to make all the experiment variables available to these | - | | scripts. | - +---------------------------+--------------------------------------------------------------------------------------------------------------+ - | YYYYMMDDHH | Cycle directory (empty) | - +---------------------------+--------------------------------------------------------------------------------------------------------------+ - -In addition, running the SRW App in *community* mode creates the ``fix_am`` and ``fix_lam`` directories (see :numref:`Table %s `) in ``$EXPTDIR``. The ``fix_lam`` directory is initially empty but will contain some *fix* (time-independent) files after the grid, orography, and/or surface climatology generation tasks run. - -.. _FixDirectories: - -.. table:: Description of the fix directories - - +-------------------------+----------------------------------------------------------+ - | **Directory Name** | **Description** | - +=========================+==========================================================+ - | fix_am | Directory containing the global fix (time-independent) | - | | data files. The experiment generation script symlinks | - | | these files from a machine-dependent system directory. | - +-------------------------+----------------------------------------------------------+ - | fix_lam | Directory containing the regional fix (time-independent) | - | | data files that describe the regional grid, orography, | - | | and various surface climatology fields, as well as | - | | symlinks to pre-generated files. | - +-------------------------+----------------------------------------------------------+ - -Once the Rocoto workflow is launched, several files and directories are generated. A log file named ``log.launch_FV3LAM_wflow`` will be created (unless it already exists) in ``$EXPTDIR``. The first several workflow tasks (i.e., ``make_grid``, ``make_orog``, ``make_sfc_climo``, ``get_extrn_ics``, and ``get_extrn_lbcs``) are preprocessing tasks, and these tasks also result in the creation of new files and subdirectories, described in :numref:`Table %s `. - -.. _CreatedByWorkflow: - -.. table:: New directories and files created when the workflow is launched - :widths: 30 70 - - +---------------------------+--------------------------------------------------------------------+ - | **Directory/File Name** | **Description** | - +===========================+====================================================================+ - | YYYYMMDDHH | This is a “cycle directory” that is updated when the first | - | | cycle-specific workflow tasks (``get_extrn_ics`` and | - | | ``get_extrn_lbcs``) are run. These tasks are launched | - | | simultaneously for each cycle in the experiment. Cycle directories | - | | are created to contain cycle-specific files for each cycle that | - | | the experiment runs. If ``DATE_FIRST_CYCL`` and ``DATE_LAST_CYCL`` | - | | are different in the ``config.yaml`` file, more than one cycle | - | | directory will be created under the experiment directory. | - +---------------------------+--------------------------------------------------------------------+ - | grid | Directory generated by the ``make_grid`` task to store grid files | - | | for the experiment | - +---------------------------+--------------------------------------------------------------------+ - | log | Contains log files generated by the overall workflow and by its | - | | various tasks. View the files in this directory to determine why | - | | a task may have failed. | - +---------------------------+--------------------------------------------------------------------+ - | orog | Directory generated by the ``make_orog`` task containing the | - | | orography files for the experiment | - +---------------------------+--------------------------------------------------------------------+ - | sfc_climo | Directory generated by the ``make_sfc_climo`` task containing the | - | | surface climatology files for the experiment | - +---------------------------+--------------------------------------------------------------------+ - | FV3LAM_wflow.db | Database files that are generated when Rocoto is called (by the | - | FV3LAM_wflow_lock.db | launch script) to launch the workflow | - +---------------------------+--------------------------------------------------------------------+ - | log.launch_FV3LAM_wflow | The ``launch_FV3LAM_wflow.sh`` script appends its output to this | - | | log file each time it is called. View the last several | - | | lines of this file to check the status of the workflow. | - +---------------------------+--------------------------------------------------------------------+ - -The output files for an experiment are described in :numref:`Section %s `. -The workflow tasks are described in :numref:`Section %s `. - - -User Support, Documentation, and Contributions to Development -=============================================================== - -The SRW App's `GitHub Discussions `__ forum provides online support for UFS users and developers to post questions and exchange information. - -A list of available documentation is shown in :numref:`Table %s `. - -.. _list_of_documentation: - -.. table:: Centralized list of documentation - - +----------------------------+---------------------------------------------------------------------------------+ - | **Documentation** | **Location** | - +============================+=================================================================================+ - | UFS SRW Application | https://ufs-srweather-app.readthedocs.io/en/develop/ | - | User's Guide | | - +----------------------------+---------------------------------------------------------------------------------+ - | UFS_UTILS Technical | https://noaa-emcufs-utils.readthedocs.io/en/latest | - | Documentation | | - +----------------------------+---------------------------------------------------------------------------------+ - | UFS_UTILS Scientific | https://ufs-community.github.io/UFS_UTILS/index.html | - | Documentation | | - +----------------------------+---------------------------------------------------------------------------------+ - | UFS Weather Model | https://ufs-weather-model.readthedocs.io/en/latest | - | User's Guide | | - +----------------------------+---------------------------------------------------------------------------------+ - | HPC-Stack Documentation | https://hpc-stack.readthedocs.io/en/latest/ | - +----------------------------+---------------------------------------------------------------------------------+ - | FV3 Scientific | https://repository.library.noaa.gov/view/noaa/30725 | - | Documentation | | - +----------------------------+---------------------------------------------------------------------------------+ - | FV3 Technical | https://noaa-emc.github.io/FV3_Dycore_ufs-v2.0.0/html/index.html | - | Documentation | | - +----------------------------+---------------------------------------------------------------------------------+ - | CCPP Scientific | https://dtcenter.ucar.edu/GMTB/v6.0.0/sci_doc/index.html | - | Documentation | | - +----------------------------+---------------------------------------------------------------------------------+ - | CCPP Technical | https://ccpp-techdoc.readthedocs.io/en/latest/ | - | Documentation | | - +----------------------------+---------------------------------------------------------------------------------+ - | Stochastic Physics | https://stochastic-physics.readthedocs.io/en/latest/ | - | Documentation | | - +----------------------------+---------------------------------------------------------------------------------+ - | ESMF manual | https://earthsystemmodeling.org/docs/release/latest/ESMF_usrdoc/ | - +----------------------------+---------------------------------------------------------------------------------+ - | Unified Post Processor | https://upp.readthedocs.io/en/latest/ | - +----------------------------+---------------------------------------------------------------------------------+ - -The UFS community is encouraged to contribute to the development effort of all related -utilities, model code, and infrastructure. Users can post issues in the related GitHub repositories to report bugs or to announce upcoming contributions to the code base. For code to be accepted into the authoritative repositories, users must follow the code management rules of each UFS component repository. These rules are usually outlined in the User's Guide (see :numref:`Table %s `) or wiki for each respective repository (see :numref:`Table %s `). Contributions to the `ufs-srweather-app `__ repository should follow the guidelines contained in the `SRW App Contributor's Guide `__. - -Future Direction -================= - -Users can expect to see incremental improvements and additional capabilities in upcoming releases of the SRW Application to enhance research opportunities and support operational forecast implementations. Planned enhancements include: - -* A more extensive set of supported developmental physics suites. -* A larger number of pre-defined domains/resolutions and a *fully supported* capability to create a user-defined domain. -* Add user-defined vertical levels (number and distribution). -* Inclusion of data assimilation and forecast restart/cycling capabilities. - - -.. bibliography:: references.bib - - - diff --git a/docs/UsersGuide/source/Reference/FAQ.rst b/docs/UsersGuide/source/Reference/FAQ.rst new file mode 100644 index 0000000000..61e63edd63 --- /dev/null +++ b/docs/UsersGuide/source/Reference/FAQ.rst @@ -0,0 +1,269 @@ +.. _FAQ: + +**** +FAQ +**** + +* :ref:`How do I define an experiment name? ` +* :ref:`How do I change the Physics Suite Definition File (SDF)? ` +* :ref:`How do I change the grid? ` +* :ref:`How can I select which workflow tasks to run? ` +* :ref:`How do I turn on/off the cycle-independent workflow tasks? ` +* :ref:`How do I restart a DEAD task? ` +* :ref:`How can I clean up the SRW App code if something went wrong? ` +* :ref:`How do I run a new experiment? ` +* :ref:`How can I add a physics scheme (e.g., YSU PBL) to the UFS SRW App? ` +* :ref:`How can I troubleshoot issues related to ICS/LBCS generation for a predefined 3-km grid? ` + +.. _DefineExptName: + +==================================== +How do I define an experiment name? +==================================== + +The name of the experiment is set in the ``workflow:`` section of the ``config.yaml`` file using the variable ``EXPT_SUBDIR``. +See :numref:`Section %s ` and/or :numref:`Section %s ` for more details. + +.. _ChangePhysics: + +========================================================= +How do I change the Physics Suite Definition File (SDF)? +========================================================= + +The SDF is set in the ``workflow:`` section of the ``config.yaml`` file using the variable ``CCPP_PHYS_SUITE``. The five supported physics suites for the SRW Application are: + +.. code-block:: console + + FV3_GFS_v16 + FV3_RRFS_v1beta + FV3_HRRR + FV3_WoFS_v0 + FV3_RAP + +When users run the ``generate_FV3LAM_wflow.py`` script, the SDF file is copied from its location in the forecast +model directory to the experiment directory ``$EXPTDIR``. For more information on the :term:`CCPP` physics suite parameters, see :numref:`Section %s `. + +.. _ChangeGrid: + +=========================== +How do I change the grid? +=========================== + +To change the predefined grid, modify the ``PREDEF_GRID_NAME`` variable in the ``task_run_fcst:`` section of the ``config.yaml`` script (see :numref:`Section %s ` for details on creating and modifying the ``config.yaml`` file). The four supported predefined grids as of the SRW Application v2.1.0 release are: + +.. code-block:: console + + RRFS_CONUS_3km + RRFS_CONUS_13km + RRFS_CONUS_25km + SUBCONUS_Ind_3km + +However, users can choose from a variety of predefined grids listed in :numref:`Section %s `. An option also exists to create a user-defined grid, with information available in :numref:`Section %s `. However, the user-defined grid option is not fully supported as of the v2.1.0 release and is provided for informational purposes only. + +.. _SetTasks: + +=============================================== +How can I select which workflow tasks to run? +=============================================== + +The ``/parm/wflow`` directory contains several ``YAML`` files that configure different workflow task groups. Each task group file contains a number of tasks that are typically run together. :numref:`Table %s ` describes each of the task groups. + +.. _task-group-files: + +.. list-table:: Task group files + :widths: 20 50 + :header-rows: 1 + + * - File + - Function + * - aqm_post.yaml + - SRW-AQM post-processing tasks + * - aqm_prep.yaml + - SRW-AQM pre-processing tasks + * - coldstart.yaml + - Tasks required to run a cold-start forecast + * - da_data_preproc.yaml + - Preprocessing tasks for RRFS `DA `. + * - plot.yaml + - Plotting tasks + * - post.yaml + - Post-processing tasks + * - prdgen.yaml + - Horizontal map projection processor that creates smaller domain products from the larger domain created by the UPP. + * - prep.yaml + - Pre-processing tasks + * - verify_det.yaml + - Deterministic verification tasks + * - verify_ens.yaml + - Ensemble verification tasks + * - verify_pre.yaml + - Verification pre-processing tasks + +The default workflow task groups are set in ``parm/wflow/default_workflow.yaml`` and include ``prep.yaml``, ``coldstart.yaml``, and ``post.yaml``. Changing this list of task groups in the user configuration file (``config.yaml``) will override the default and run only the task groups listed. For example, to omit :term:`cycle-independent` tasks and run plotting tasks, users would delete ``prep.yaml`` from the list of tasks and add ``plot.yaml``: + +.. code-block:: console + + rocoto: + tasks: + taskgroups: '{{ ["parm/wflow/coldstart.yaml", "parm/wflow/post.yaml", "parm/wflow/plot.yaml"]|include }}' + +Users may need to make additional adjustments to ``config.yaml`` depending on which task groups they add or remove. For example, when plotting, the user should add the plotting increment (``PLOT_FCST_INC``) for the plotting tasks in ``task_plot_allvars``. + +Users can omit specific tasks from a task group by including them under the list of tasks as an empty entry. For example, if a user wanted to run only ``task_pre_post_stat`` from ``aqm_post.yaml``, the taskgroups list would include ``aqm_post.yaml``, and the tasks that the user wanted to omit would be listed with no value: + +.. code-block:: console + + rocoto: + tasks: + taskgroups: '{{ ["parm/wflow/prep.yaml", "parm/wflow/coldstart.yaml", "parm/wflow/post.yaml", "parm/wflow/aqm_post.yaml"]|include }}' + task_post_stat_o3: + task_post_stat_pm25: + task_bias_correction_o3: + task_bias_correction_pm25: + +.. _CycleInd: + +=========================================================== +How do I turn on/off the cycle-independent workflow tasks? +=========================================================== + +The first three pre-processing tasks ``make_grid``, ``make_orog``, and ``make_sfc_climo`` +are :term:`cycle-independent`, meaning that they only need to be run once per experiment. +By default, the the workflow will run these tasks. However, if the +grid, orography, and surface climatology files that these tasks generate are already +available (e.g., from a previous experiment that used the same grid as the current experiment), then +these tasks can be skipped, and the workflow can use those pre-generated files. + +To skip these tasks, remove ``parm/wflow/prep.yaml`` from the list of task groups in the Rocoto section of the configuration file (``config.yaml``): + +.. code-block:: console + + rocoto: + tasks: + taskgroups: '{{ ["parm/wflow/coldstart.yaml", "parm/wflow/post.yaml"]|include }}' + +Then, add the paths to the previously generated grid, orography, and surface climatology files under the appropariate tasks in ``config.yaml``: + +.. code-block:: console + + task_make_grid: + GRID_DIR: /path/to/directory/containing/grid/files + task_make_orog: + OROG_DIR: /path/to/directory/containing/orography/files + task_make_sfc_climo: + SFC_CLIMO_DIR: /path/to/directory/containing/surface/climatology/files + +All three sets of files *may* be placed in the same directory location (and would therefore have the same path), but they can also reside in different directories and use different paths. + +.. _RestartTask: + +============================= +How do I restart a DEAD task? +============================= + +On platforms that utilize Rocoto workflow software (such as NCAR's Cheyenne machine), if something goes wrong with the workflow, a task may end up in the DEAD state: + +.. code-block:: console + + rocotostat -w FV3SAR_wflow.xml -d FV3SAR_wflow.db -v 10 + CYCLE TASK JOBID STATE EXIT STATUS TRIES DURATION + ================================================================================= + 201906151800 make_grid 9443237 QUEUED - 0 0.0 + 201906151800 make_orog - - - - - + 201906151800 make_sfc_climo - - - - - + 201906151800 get_extrn_ics 9443293 DEAD 256 3 5.0 + +This means that the dead task has not completed successfully, so the workflow has stopped. Once the issue +has been identified and fixed (by referencing the log files in ``$EXPTDIR/log``), users can re-run the failed task using the ``rocotorewind`` command: + +.. code-block:: console + + rocotorewind -w FV3LAM_wflow.xml -d FV3LAM_wflow.db -v 10 -c 201906151800 -t get_extrn_ics + +where ``-c`` specifies the cycle date (first column of ``rocotostat`` output) and ``-t`` represents the task name +(second column of ``rocotostat`` output). After using ``rocotorewind``, the next time ``rocotorun`` is used to +advance the workflow, the job will be resubmitted. + +.. _CleanUp: + +=============================================================== +How can I clean up the SRW App code if something went wrong? +=============================================================== + +The ``ufs-srweather-app`` repository contains a ``devclean.sh`` convenience script. This script can be used to clean up code if something goes wrong when checking out externals or building the application. To view usage instructions and to get help, run with the ``-h`` flag: + +.. code-block:: console + + ./devclean.sh -h + +To remove the ``build`` directory, run: + +.. code-block:: console + + ./devclean.sh --remove + +To remove all build artifacts (including ``build``, ``exec``, ``lib``, and ``share``), run: + +.. code-block:: console + + ./devclean.sh --clean + OR + ./devclean.sh -a + +To remove external submodules, run: + +.. code-block:: console + + ./devclean.sh --sub-modules + +Users will need to check out the external submodules again before building the application. + +In addition to the options above, many standard terminal commands can be run to remove unwanted files and directories (e.g., ``rm -rf expt_dirs``). A complete explanation of these options is beyond the scope of this User's Guide. + +.. _NewExpt: + +================================== +How can I run a new experiment? +================================== + +To run a new experiment at a later time, users need to rerun the commands in :numref:`Section %s ` that reactivate the *workflow_tools* environment: + +.. code-block:: console + + source /path/to/etc/lmod-setup.sh/or/lmod-setup.csh + module use /path/to/modulefiles + module load wflow_ + +Follow any instructions output by the console (e.g., ``conda activate workflow_tools``). + +Then, users can configure a new experiment by updating the environment variables in ``config.yaml`` to reflect the desired experiment configuration. Detailed instructions can be viewed in :numref:`Section %s `. Parameters and valid values are listed in :numref:`Section %s `. After adjusting the configuration file, generate the new experiment by running ``./generate_FV3LAM_wflow.py``. Check progress by navigating to the ``$EXPTDIR`` and running ``rocotostat -w FV3LAM_wflow.xml -d FV3LAM_wflow.db -v 10``. + +.. note:: + + If users have updated their clone of the SRW App (e.g., via ``git pull`` or ``git fetch``/``git merge``) since running their last experiement, and the updates include a change to ``Externals.cfg``, users will need to rerun ``checkout_externals`` (instructions :ref:`here `) and rebuild the SRW App according to the instructions in :numref:`Section %s `. + +.. _AddPhys: + +==================================================================== +How can I add a physics scheme (e.g., YSU PBL) to the UFS SRW App? +==================================================================== + +At this time, there are ten physics suites available in the SRW App, :ref:`five of which are fully supported `. However, several additional physics schemes are available in the UFS Weather Model (WM) and can be enabled in the SRW App. The CCPP Scientific Documentation details the various `namelist options `__ available in the UFS WM, including physics schemes, and also includes an `overview of schemes and suites `__. +To enable an additional physics scheme, such as the YSU PBL scheme, in the SRW App, users must modify ``ufs-srweather-app/parm/FV3.input.yml`` and set the variable corresponding to the desired physics scheme to *True* under the physics suite they would like to use (e.g., ``do_ysu = True``). + +It may be necessary to disable another physics scheme, too. For example, when using the YSU PBL scheme, users should disable the default SATMEDMF PBL scheme (*satmedmfvdifq*) by setting the ``satmedmf`` variable to *False* in the ``FV3.input.yml`` file. +Regardless, users will need to modify the suite definition file (SDF) and recompile the code. For example, to activate the YSU PBL scheme, users should replace the line ``satmedmfvdifq`` with ``ysuvdif`` and recompile the code. + +Users must ensure that they are using the same physics suite in their ``config.yaml`` file as the one they modified in ``FV3.input.yml``. Then, the user can run the ``generate_FV3LAM_wflow.py`` script to generate an experiment and navigate to the experiment directory. They should see ``do_ysu = .true.`` in the namelist file (or a similar statement, depending on the physics scheme selected), which indicates that the YSU PBL scheme is enabled. + +.. _IC-LBC-gen-issue: + +========================================================================================================== +How can I troubleshoot issues related to :term:`ICS`/:term:`LBCS` generation for a predefined 3-km grid? +========================================================================================================== + +If you encounter issues while generating ICS and LBCS for a predefined 3-km grid using the UFS SRW App, there are a number of troubleshooting options. The first step is always to check the log file for a failed task. This file will provide information on what went wrong. A log file for each task appears in the ``log`` subdirectory of the experiment directory (e.g., ``$EXPTDIR/log/make_ics``). + +Additionally, users can try increasing the number of processors or the wallclock time requested for the jobs. Sometimes jobs may fail without errors because the process is cut short. These settings can be adusted in one of the ``ufs-srweather-app/parm/wflow`` files. For ICs/LBCs tasks, these parameters are set in the ``coldstart.yaml`` file. + +Users can also update the hash of UFS_UTILS in the ``Externals.cfg`` file to the HEAD of that repository. There was a known memory issue with how ``chgres_cube`` was handling regridding of the 3-D wind field for large domains at high resolutions (see `UFS_UTILS PR #766 `__ and the associated issue for more information). If changing the hash in ``Externals.cfg``, users will need to rerun ``manage_externals`` and rebuild the code (see :numref:`Section %s `). \ No newline at end of file diff --git a/docs/UsersGuide/source/Glossary.rst b/docs/UsersGuide/source/Reference/Glossary.rst similarity index 91% rename from docs/UsersGuide/source/Glossary.rst rename to docs/UsersGuide/source/Reference/Glossary.rst index 3406181324..8a61939d18 100644 --- a/docs/UsersGuide/source/Glossary.rst +++ b/docs/UsersGuide/source/Reference/Glossary.rst @@ -102,12 +102,12 @@ Glossary GFS `Global Forecast System `_. The GFS is a National Centers for Environmental Prediction (:term:`NCEP`) weather forecast model that generates data for dozens of atmospheric and land-soil variables, including temperatures, winds, precipitation, soil moisture, and atmospheric ozone concentration. The system couples four separate models (atmosphere, ocean, land/soil, and sea ice) that work together to accurately depict weather conditions. + GOCART + NASA's Goddard Chemistry Aerosol Radiation and Transport (GOCART) model simulates the distribution of major tropospheric aerosol types, including sulfate, dust, organic carbon (OC), black carbon (BC), and sea salt aerosols. The UFS Weather Model integrates a prognostic aerosol component using GOCART. The code is publicly available on GitHub at https://github.com/GEOS-ESM/GOCART. + GRIB2 The second version of the World Meterological Organization's (WMO) standard for distributing gridded data. - GSI - `Gridpoint Statistical Interpolation `__ (GSI) is a variational data assimilation system, designed to be flexible, state-of-art, and run efficiently on various parallel computing platforms. It supports :term:`RRFS` features. GSI code is publicly available `on GitHub `__, and fix file data is publicly available `here `__. - GSL NOAA `Global Systems Laboratory `__ is one of ten NOAA Research laboratories and is located in Boulder, Colorado. Its research improves environmental prediction models, develops state-of-the-science decision support tools and visualization systems, and uses high-performance computing technology to support a Weather-Ready Nation. @@ -126,7 +126,7 @@ Glossary HRRR `High Resolution Rapid Refresh `__. The HRRR is a NOAA real-time 3-km resolution, hourly updated, cloud-resolving, convection-allowing atmospheric model initialized by 3km grids with 3km radar assimilation. Radar data is assimilated in the HRRR every 15 min over a 1-h period adding further detail to that provided by the hourly data assimilation from the 13km radar-enhanced Rapid Refresh. - IC/LBCs + ICs/LBCs Initial conditions/lateral boundary conditions ICs @@ -135,12 +135,20 @@ Glossary J-jobs Scripting layer (contained in ``ufs-srweather-app/jobs/``) that should be directly called for each workflow component (either on the command line or by the workflow manager) to run a specific task in the workflow. The different scripting layers are described in detail in the `NCO Implementation Standards document `__ + JEDI + The Joint Effort for Data assimilation Integration (`JEDI `__) is a unified and versatile data assimilation (DA) system for Earth System Prediction. It aims to enable efficient research and accelerated transition from research to operations by providing a framework that takes into account all components of the Earth system in a consistent manner. The JEDI software package can run on a variety of platforms and for a variety of purposes, and it is designed to readily accommodate new atmospheric and oceanic models and new observation systems. The `JEDI User's Guide `__ contains extensive information on the software. + + JEDI is developed and distributed by the `Joint Center for Satellite Data Assimilation `__, a multi-agency research center hosted by the University Corporation for Atmospheric Research (`UCAR `__). JCSDA is dedicated to improving and accelerating the quantitative use of research and operational satellite data in weather, ocean, climate, and environmental analysis and prediction systems. + LAM Limited Area Model (grid type), formerly known as the "Stand-Alone Regional Model," or SAR. LAM grids use a regional (rather than global) configuration of the :term:`FV3` :term:`dynamical core`. LBCs Lateral boundary conditions + MEGAN + The Model of Emissions of Gases and Aerosols from Nature (`MEGAN `__) is a modeling system for estimating the emission of gases and aerosols from terrestrial ecosystems into the atmosphere. It has been integrated into a number of chemistry and transport models, including :ref:`NEXUS `. + MERRA2 The `Modern-Era Retrospective analysis for Research and Applications, Version 2 `__ provides satellite observation data back to 1980. According to NASA, "It was introduced to replace the original MERRA dataset because of the advances made in the assimilation system that enable assimilation of modern hyperspectral radiance and microwave observations, along with GPS-Radio Occultation datasets. It also uses NASA's ozone profile observations that began in late 2004. Additional advances in both the GEOS model and the GSI assimilation system are included in MERRA-2. Spatial resolution remains about the same (about 50 km in the latitudinal direction) as in MERRA." @@ -248,8 +256,8 @@ Glossary Helicity measures the rotation in a storm's updraft (rising) air. Significant rotation increases the probability that the storm will produce severe weather, including tornadoes. See http://ww2010.atmos.uiuc.edu/(Gh)/guides/mtr/svr/modl/fcst/params/hel.rxml for more details on updraft helicity. UPP - The `Unified Post Processor `__ is software developed at :term:`NCEP` and used operationally to - post-process raw output from a variety of :term:`NCEP`'s :term:`NWP` models, including the :term:`FV3`. + The `Unified Post Processor `__ is software developed at :term:`NCEP` and used operationally to + post-process raw output from a variety of :term:`NCEP`'s :term:`NWP` models, including the :term:`FV3`. See https://epic.noaa.gov/unified-post-processor/ for more information. Weather Enterprise Individuals and organizations from public, private, and academic sectors that contribute to the research, development, and production of weather forecast products; primary consumers of these weather forecast products. diff --git a/docs/UsersGuide/source/RocotoInfo.rst b/docs/UsersGuide/source/Reference/RocotoInfo.rst similarity index 59% rename from docs/UsersGuide/source/RocotoInfo.rst rename to docs/UsersGuide/source/Reference/RocotoInfo.rst index ad0ec024c9..d7da7a16f1 100644 --- a/docs/UsersGuide/source/RocotoInfo.rst +++ b/docs/UsersGuide/source/Reference/RocotoInfo.rst @@ -3,21 +3,23 @@ ================================== Rocoto Introductory Information ================================== -The tasks in the SRW Application (:numref:`Table %s `) are typically run using -the Rocoto Workflow Manager. Rocoto is a Ruby program that communicates with the batch system on an +The tasks in the SRW Application are typically run using the Rocoto Workflow Manager +(see :numref:`Table %s ` for default tasks). +Rocoto is a Ruby program that communicates with the batch system on an :term:`HPC` system to run and manage dependencies between the tasks. Rocoto submits jobs to the HPC batch system as the task dependencies allow and runs one instance of the workflow for a set of user-defined -:term:`cycles `. More information about Rocoto can be found on the `Rocoto Wiki `__. +:term:`cycles `. More information about Rocoto can be found on the +`Rocoto Wiki `__. The SRW App workflow is defined in a Jinja-enabled Rocoto XML template called ``FV3LAM_wflow.xml``, -which resides in the ``parm`` directory. When the ``generate_FV3LAM_wflow.py`` -script is run, the ``set_template`` uwtool is called, and the parameters in the template file +which resides in the ``parm`` directory. When the ``generate_FV3LAM_wflow.py`` script is run, +the :ref:`Unified Workflow ` ``set_template`` tool is called, and the parameters in the template file are filled in. The completed file contains the workflow task names, parameters needed by the job scheduler, and task interdependencies. The generated XML file is then copied to the experiment directory: ``$EXPTDIR/FV3LAM_wflow.xml``. There are a number of Rocoto commands available to run and monitor the workflow; users can find more information in the -complete `Rocoto documentation `__. +complete `Rocoto documentation `__. Descriptions and examples of commonly used commands are discussed below. .. _RocotoRunCmd: @@ -29,12 +31,12 @@ automatically resubmit failed tasks and can recover from system outages without .. code-block:: console - rocotorun -w -d -v 10 + rocotorun -w /path/to/workflow/xml/file -d /path/to/workflow/database/file -v 10 where * ``-w`` specifies the name of the workflow definition file. This must be an XML file. -* ``-d`` specifies the name of the database file that stores the state of the workflow. The database file is a binary file created and used only by Rocoto. It need not exist prior to the first time the command is run. +* ``-d`` specifies the name of the database file that stores the state of the workflow. The database file is a binary file created and used only by Rocoto. It does not need to exist when the command is initially run. * ``-v`` (optional) specified level of verbosity. If no level is specified, a level of 1 is used. From the ``$EXPTDIR`` directory, the ``rocotorun`` command for the workflow would be: @@ -43,10 +45,12 @@ From the ``$EXPTDIR`` directory, the ``rocotorun`` command for the workflow woul rocotorun -w FV3LAM_wflow.xml -d FV3LAM_wflow.db +Users will need to include the absolute or relative path to these files when running the command from another directory. + It is important to note that the ``rocotorun`` process is iterative; the command must be executed many times before the entire workflow is completed, usually every 1-10 minutes. This command can be -placed in the user’s crontab, and cron will call it with a specified frequency. More information on -this command can be found in the `Rocoto documentation `__. +placed in the user’s :term:`crontab`, and cron will call it with a specified frequency. More information on +this command can be found in the `Rocoto documentation `__. The first time the ``rocotorun`` command is executed for a workflow, the files ``FV3LAM_wflow.db`` and ``FV3LAM_wflow_lock.db`` are created. There is usually no need for the user to modify these files. @@ -66,29 +70,29 @@ workflow using the ``rocotostat`` command: .. code-block:: console - rocotostat -w -d + rocotostat -w /path/to/workflow/xml/file -d /path/to/workflow/database/file Executing this command will generate a workflow status table similar to the following: .. code-block:: console - CYCLE TASK JOBID STATE EXIT STATUS TRIES DURATION - ============================================================================================================= - 201907010000 make_grid 175805 QUEUED - 0 0.0 - 201907010000 make_orog - - - - - - 201907010000 make_sfc_climo - - - - - - 201907010000 get_extrn_ics druby://hfe01:36261 SUBMITTING - 0 0.0 - 201907010000 get_extrn_lbcs druby://hfe01:36261 SUBMITTING - 0 0.0 - 201907010000 make_ics - - - - - - 201907010000 make_lbcs - - - - - - 201907010000 run_fcst - - - - - - 201907010000 run_post_f000 - - - - - - 201907010000 run_post_f001 - - - - - - 201907010000 run_post_f002 - - - - - - 201907010000 run_post_f003 - - - - - - 201907010000 run_post_f004 - - - - - - 201907010000 run_post_f005 - - - - - - 201907010000 run_post_f006 - - - - - + CYCLE TASK JOBID STATE EXIT STATUS TRIES DURATION + =============================================================================================================== + 201907010000 make_grid 175805 QUEUED - 0 0.0 + 201907010000 make_orog - - - - - + 201907010000 make_sfc_climo - - - - - + 201907010000 get_extrn_ics druby://hfe01:36261 SUBMITTING - 0 0.0 + 201907010000 get_extrn_lbcs druby://hfe01:36261 SUBMITTING - 0 0.0 + 201907010000 make_ics_mem000 - - - - - + 201907010000 make_lbcs_mem000 - - - - - + 201907010000 run_fcst_mem000 - - - - - + 201907010000 run_post__mem000_f000 - - - - - + 201907010000 run_post__mem000_f001 - - - - - + 201907010000 run_post__mem000_f002 - - - - - + 201907010000 run_post__mem000_f003 - - - - - + 201907010000 run_post__mem000_f004 - - - - - + 201907010000 run_post__mem000_f005 - - - - - + 201907010000 run_post__mem000_f006 - - - - - This table indicates that the ``make_grid`` task was sent to the batch system and is now queued, while the ``get_extrn_ics`` and ``get_extrn_lbcs`` tasks for the ``201907010000`` cycle are currently being @@ -105,27 +109,28 @@ on the grid size and computational resources available), the output of the ``roc .. code-block:: console - CYCLE TASK JOBID STATE EXIT STATUS TRIES DURATION - ==================================================================================================== - 201907010000 make_grid 175805 SUCCEEDED 0 1 10.0 - 201907010000 make_orog 175810 SUCCEEDED 0 1 27.0 - 201907010000 make_sfc_climo 175822 SUCCEEDED 0 1 38.0 - 201907010000 get_extrn_ics 175806 SUCCEEDED 0 1 37.0 - 201907010000 get_extrn_lbcs 175807 SUCCEEDED 0 1 53.0 - 201907010000 make_ics 175825 SUCCEEDED 0 1 99.0 - 201907010000 make_lbcs 175826 SUCCEEDED 0 1 90.0 - 201907010000 run_fcst 175937 RUNNING - 0 0.0 - 201907010000 run_post_f000 - - - - - - 201907010000 run_post_f001 - - - - - - 201907010000 run_post_f002 - - - - - - 201907010000 run_post_f003 - - - - - - 201907010000 run_post_f004 - - - - - - 201907010000 run_post_f005 - - - - - - 201907010000 run_post_f006 - - - - - + CYCLE TASK JOBID STATE EXIT STATUS TRIES DURATION + =================================================================================================== + 201907010000 make_grid 175805 SUCCEEDED 0 1 10.0 + 201907010000 make_orog 175810 SUCCEEDED 0 1 27.0 + 201907010000 make_sfc_climo 175822 SUCCEEDED 0 1 38.0 + 201907010000 get_extrn_ics 175806 SUCCEEDED 0 1 37.0 + 201907010000 get_extrn_lbcs 175807 SUCCEEDED 0 1 53.0 + 201907010000 make_ics_mem000 175825 SUCCEEDED 0 1 99.0 + 201907010000 make_lbcs_mem000 175826 SUCCEEDED 0 1 90.0 + 201907010000 run_fcst_mem000 175937 RUNNING - 0 0.0 + 201907010000 run_post__mem000_f000 - - - - - + 201907010000 run_post__mem000_f001 - - - - - + 201907010000 run_post__mem000_f002 - - - - - + 201907010000 run_post__mem000_f003 - - - - - + 201907010000 run_post__mem000_f004 - - - - - + 201907010000 run_post__mem000_f005 - - - - - + 201907010000 run_post__mem000_f006 - - - - - When the workflow runs to completion, all tasks will be marked as SUCCEEDED. The log file for each task is located in ``$EXPTDIR/log``. If any task fails, the corresponding log file can be checked for error -messages. Optional arguments for the ``rocotostat`` command can be found in the `Rocoto documentation `__. +messages. Optional arguments for the ``rocotostat`` command can be found in the +`Rocoto documentation `__. .. _rocotocheck: @@ -138,35 +143,35 @@ from the ``$EXPTDIR`` directory as follows: .. code-block:: console - rocotocheck -w -d file -c -t + rocotocheck -w FV3LAM_wflow.xml -d FV3LAM_wflow.db file -c -t where -* ``-c`` is the cycle to query in YYYYMMDDHHmm format -* ``-t`` is the task name (e.g., ``make_grid``, ``get_extrn_ics``, ``run_fcst``). +* ``-c`` is the cycle to query in YYYYMMDDHHmm format. +* ``-t`` is the task name (e.g., ``make_grid``, ``get_extrn_ics``, ``run_fcst_mem000``). -The cycle and task names appear in the first and second columns of the table output by ``rocotostat``. +The cycle and task names appear in the first and second columns of the table output by ``rocotostat``. Users will need to include the absolute or relative path to the workflow XML and database files when running the command from another directory. A specific example is: .. code-block:: console - rocotocheck -w FV3LAM_wflow.xml -d FV3LAM_wflow.db -v 10 -c 201907010000 -t run_fcst + rocotocheck -w /Users/John.Doe/expt_dirs/test_community/FV3LAM_wflow.xml -d /Users/John.Doe/expt_dirs/test_community/FV3LAM_wflow.db -v 10 -c 201907010000 -t run_fcst_mem000 Running ``rocotocheck`` will result in output similar to the following: .. code-block:: console :emphasize-lines: 8,19,34 - Task: run_fcst + Task: run_fcst_mem000 account: gsd-fv3 - command: /scratch2/BMC/det/$USER/ufs-srweather-app/ush/load_modules_run_task.sh "run_fcst" "/scratch2/BMC/det/$USER/ufs-srweather-app/jobs/JREGIONAL_RUN_FCST" + command: /scratch2/BMC/det/$USER/ufs-srweather-app/ush/load_modules_run_task.sh "run_fcst_mem000" "/scratch2/BMC/det/$USER/ufs-srweather-app/jobs/JREGIONAL_RUN_FCST" cores: 24 final: false jobname: run_FV3 - join: /scratch2/BMC/det/$USER/expt_dirs/test_community/log/run_fcst_2019070100.log + join: /scratch2/BMC/det/$USER/expt_dirs/test_community/log/run_fcst_mem000_2019070100.log maxtries: 3 - name: run_fcst + name: run_fcst_mem000 nodes: 1:ppn=24 queue: batch throttle: 9999999 @@ -212,7 +217,7 @@ command will rerun tasks in the workflow. The command line options are the same .. code-block:: console - rocotorewind -w -d file -c -t + rocotorewind -w /path/to/workflow/xml/file -d /path/to/workflow/database/ file -c -t Running this command will edit the Rocoto database file ``FV3LAM_wflow.db`` to remove evidence that the job has been run. ``rocotorewind`` is recommended over ``rocotoboot`` for restarting a task, since ``rocotoboot`` will force a specific @@ -222,13 +227,13 @@ command to rerun the forecast task from ``$EXPTDIR`` is: .. code-block:: console - rocotorewind -w FV3LAM_wflow.xml -d FV3LAM_wflow.db -v 10 -c 201907010000 -t run_fcst + rocotorewind -w FV3LAM_wflow.xml -d FV3LAM_wflow.db -v 10 -c 201907010000 -t run_fcst_mem000 rocotoboot =========== ``rocotoboot`` will force a specific task of a cycle in a Rocoto workflow to run. All dependencies and throttle limits are ignored, and it is generally recommended to use ``rocotorewind`` instead. An example of how to -use this command to rerun the ``make_ics`` task from the ``$EXPTDIR`` is: +use this command to rerun the ``make_ics`` task from ``$EXPTDIR`` is: .. code-block:: console diff --git a/docs/UsersGuide/source/Reference/index.rst b/docs/UsersGuide/source/Reference/index.rst new file mode 100644 index 0000000000..a62993b625 --- /dev/null +++ b/docs/UsersGuide/source/Reference/index.rst @@ -0,0 +1,9 @@ +Reference +============ + +.. toctree:: + :maxdepth: 3 + + RocotoInfo + FAQ + Glossary diff --git a/docs/UsersGuide/source/RunSRW.rst b/docs/UsersGuide/source/RunSRW.rst deleted file mode 100644 index b1cac7058c..0000000000 --- a/docs/UsersGuide/source/RunSRW.rst +++ /dev/null @@ -1,1253 +0,0 @@ -.. _RunSRW: - -=========================== -Running the SRW App -=========================== - -This chapter explains how to set up and run the "out-of-the-box" case for the SRW Application. However, the steps are relevant to any SRW App experiment and can be modified to suit user goals. This chapter assumes that users have already built the SRW App by following the steps in :numref:`Chapter %s `. These steps are also applicable to containerized versions of the SRW App and assume that the user has completed all of :numref:`Section %s `. - -The out-of-the-box SRW App case builds a weather forecast for June 15-16, 2019. Multiple convective weather events during these two days produced over 200 filtered storm reports. Severe weather was clustered in two areas: the Upper Midwest through the Ohio Valley and the Southern Great Plains. This forecast uses a predefined 25-km Continental United States (:term:`CONUS`) domain (RRFS_CONUS_25km), the Global Forecast System (:term:`GFS`) version 16 physics suite (FV3_GFS_v16 :term:`CCPP`), and :term:`FV3`-based GFS raw external model data for initialization. - -.. attention:: - - The SRW Application has `four levels of support `__. The steps described in this chapter will work most smoothly on preconfigured (Level 1) systems. This chapter can also serve as a starting point for running the SRW App on other systems (including generic Linux/Mac systems), but the user may need to perform additional troubleshooting. - - -The overall procedure for generating an experiment is shown in :numref:`Figure %s `, with the scripts to generate and run the workflow shown in red. Once the SRW App has been built, as described in :numref:`Chapter %s `, the steps to run a forecast are as follows: - - #. :ref:`Download and stage data ` - #. :ref:`Optional: Configure a new grid ` - #. :ref:`Generate a regional workflow experiment ` - - * :ref:`Load the python environment for the regional workflow ` - * :ref:`Set the experiment configuration parameters ` - * :ref:`Optional: Plot the output ` - * :ref:`Optional: Configure METplus Verification Suite ` - - #. :ref:`Run the regional workflow ` - -.. _AppOverallProc: - -.. figure:: _static/SRW_run_process.png - :alt: Flowchart describing the SRW App workflow steps. - - *Overall Layout of the SRW App Workflow* - -.. _Data: - -Download and Stage the Data -============================ - -The SRW App requires input files to run. These include static datasets, initial and boundary conditions files, and model configuration files. On Level 1 systems, the data required to run SRW App tests are already available in the following locations: - -.. _DataLocations: -.. table:: Data Locations for Level 1 Systems - - +--------------+------------------------------------------------------------------------------+ - | Machine | File location | - +==============+==============================================================================+ - | Cheyenne | /glade/work/epicufsrt/contrib/UFS_SRW_data/develop/input_model_data/ | - +--------------+------------------------------------------------------------------------------+ - | Gaea | /lustre/f2/dev/role.epic/contrib/UFS_SRW_data/develop/input_model_data/ | - +--------------+------------------------------------------------------------------------------+ - | Hera | /scratch1/NCEPDEV/nems/role.epic/UFS_SRW_data/develop/input_model_data/ | - +--------------+------------------------------------------------------------------------------+ - | Jet | /mnt/lfs4/HFIP/hfv3gfs/role.epic/UFS_SRW_data/develop/input_model_data/ | - +--------------+------------------------------------------------------------------------------+ - | NOAA Cloud | /contrib/EPIC/UFS_SRW_data/develop/input_model_data/ | - +--------------+------------------------------------------------------------------------------+ - | Orion | /work/noaa/epic-ps/role-epic-ps/UFS_SRW_data/develop/input_model_data/ | - +--------------+------------------------------------------------------------------------------+ - | WCOSS2 | /lfs/h2/emc/lam/noscrub/UFS_SRW_App/develop/input_model_data/ | - +--------------+------------------------------------------------------------------------------+ - -For Level 2-4 systems, the data must be added to the user's system. Detailed instructions on how to add the data can be found in :numref:`Section %s `. Sections :numref:`%s ` and :numref:`%s ` contain useful background information on the input and output files used in the SRW App. - -.. _GridSpecificConfig: - -Grid Configuration -======================= - -The SRW App officially supports the four predefined grids shown in :numref:`Table %s `. The out-of-the-box SRW App case uses the ``RRFS_CONUS_25km`` predefined grid option. More information on the predefined and user-generated grid options can be found in :numref:`Chapter %s `. Users who plan to utilize one of the four predefined domain (grid) options may continue to :numref:`Step %s `. Users who plan to create a new custom predefined grid should refer to :numref:`Section %s ` for instructions. At a minimum, these users will need to add the new grid name to the ``valid_param_vals.yaml`` file and add the corresponding grid-specific parameters in the ``predef_grid_params.yaml`` file. - -.. _PredefinedGrids: - -.. table:: Predefined Grids Supported in the SRW App - - +----------------------+-------------------+--------------------------------+ - | **Grid Name** | **Grid Type** | **Quilting (write component)** | - +======================+===================+================================+ - | RRFS_CONUS_25km | ESG grid | lambert_conformal | - +----------------------+-------------------+--------------------------------+ - | RRFS_CONUS_13km | ESG grid | lambert_conformal | - +----------------------+-------------------+--------------------------------+ - | RRFS_CONUS_3km | ESG grid | lambert_conformal | - +----------------------+-------------------+--------------------------------+ - | SUBCONUS_Ind_3km | ESG grid | lambert_conformal | - +----------------------+-------------------+--------------------------------+ - - -.. _GenerateForecast: - -Generate the Forecast Experiment -================================= -Generating the forecast experiment requires three steps: - -#. :ref:`Load the python environment for the regional workflow ` -#. :ref:`Set experiment configuration parameters ` -#. :ref:`Run a script to generate the experiment workflow ` - -The first two steps depend on the platform being used and are described here for each Level 1 platform. Users will need to adjust the instructions to reflect their machine configuration if they are working on a Level 2-4 platform. Information in :numref:`Chapter %s: Configuring the Workflow ` can help with this. - -.. _SetUpPythonEnv: - -Load the Conda/Python Environment for the Regional Workflow --------------------------------------------------------------- - -The workflow requires Python3 installed using conda, with the additional packages built in a separate conda evironment named ``regional_workflow``. This environment has the following additional packages: ``PyYAML``, ``Jinja2``, ``f90nml``, ``scipy``, ``matplotlib``, ``pygrib``, ``cartopy``. This conda/Python environment has already been set up on Level 1 platforms and can be activated in the following way: - -.. code-block:: console - - source - module use - module load wflow_ - -where ```` refers to a valid machine name (see :numref:`Section %s `). - -.. note:: - If users source the lmod-setup file on a system that doesn't need it, it will not cause any problems (it will simply do a ``module purge``). - -A brief recipe for building the regional workflow environment on a Linux or Mac system can be found in :numref:`Section %s `. - -The ``wflow_`` modulefile will then output instructions to activate the regional workflow. The user should run the commands specified in the modulefile output. The command may vary from system to system. For example, if the output says: - -.. code-block:: console - - Please do the following to activate conda: - > conda activate regional_workflow - -then the user should run ``conda activate regional_workflow``. This activates the ``regional_workflow`` conda environment, and the user typically sees ``(regional_workflow)`` in front of the Terminal prompt at this point. - -Preparing the Workflow Environment on Non-Level 1 Systems -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -Users on non-Level 1 systems can copy one of the provided ``wflow_`` files and use it as a template to create a ``wflow_`` file that works for their system. The ``wflow_macos`` and ``wflow_linux`` template modulefiles are provided in the ``modulefiles`` directory. Modifications are required to provide paths for python, miniconda modules, module loads, conda initialization, and the path for user's ``regional_workflow`` conda environment. After making modifications to a ``wflow_`` file, users can run the commands from :numref:`Step %s ` above to activate the regional workflow. - -.. note:: - ``conda`` needs to be initialized before running ``conda activate regional_workflow`` command. Depending on the user's system and login setup, this may be accomplished in a variety of ways. Conda initialization usually involves the following command: ``source /etc/profile.d/conda.sh``, where ```` is the base conda installation directory. - -.. _ExptConfig: - -Set Experiment Configuration Parameters ------------------------------------------- - -Each experiment requires certain basic information to run (e.g., date, grid, physics suite). This information is specified in ``config_defaults.yaml`` and in the user-specified ``config.yaml`` file. When generating a new experiment, the SRW App first reads and assigns default values from ``config_defaults.yaml``. Then, it reads and (re)assigns variables from the user's custom ``config.yaml`` file. - -For background info on ``config_defaults.yaml``, read :numref:`Section %s `, or jump to :numref:`Section %s ` to continue configuring the experiment. - -.. _DefaultConfigSection: - -Default configuration: ``config_defaults.yaml`` -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -.. note:: - This section provides background information on available parameters and how the SRW App uses the ``config_defaults.yaml`` file. It is informative, but users do not need to modify ``config_defaults.yaml`` to run the out-of-the-box case for the SRW App. Therefore, users may skip to :numref:`Step %s ` to continue configuring their experiment. - -Configuration parameters in the ``config_defaults.yaml`` file appear in :numref:`Table %s `. Some of these default values are intentionally invalid in order to ensure that the user assigns valid values in the user-specified ``config.yaml`` file. Any settings provided in ``config.yaml`` will override the settings in ``config_defaults.yaml``. There is usually no need for a user to modify the default configuration file. Additional information on the default settings can be found in the ``config_defaults.yaml`` file comments and in :numref:`Chapter %s `. - -.. _ConfigVarsDefault: - -.. table:: Configuration variables specified in the config_defaults.yaml script - - +-----------------------------+-----------------------------------------------------------------------+ - | **Group Name** | **Configuration variables** | - +=============================+=======================================================================+ - | User | RUN_ENVIR, MACHINE, MACHINE_FILE, ACCOUNT | - +-----------------------------+-----------------------------------------------------------------------+ - | Platform | WORKFLOW_MANAGER, NCORES_PER_NODE, BUILD_MOD_FN, WFLOW_MOD_FN, | - | | BUILD_VER_FN, RUN_VER_FN, SCHED, DOMAIN_PREGEN_BASEDIR, | - | | ENV_INIT_SCRIPTS_FPS, PRE_TASK_CMDS, PARTITION_DEFAULT, QUEUE_DEFAULT,| - | | PARTITION_HPSS, QUEUE_HPSS, PARTITION_FCST, QUEUE_FCST, | - | | RUN_CMD_UTILS, RUN_CMD_FCST, RUN_CMD_POST, SLURM_NATIVE_CMD, | - | | MODEL, CCPA_OBS_DIR, | - | | MRMS_OBS_DIR, NDAS_OBS_DIR, NOHRSC_OBS_DIR | - +-----------------------------+-----------------------------------------------------------------------+ - | Workflow | WORKFLOW_ID, USE_CRON_TO_RELAUNCH, CRON_RELAUNCH_INTVL_MNTS, | - | | EXPT_BASEDIR, EXPT_SUBDIR, EXEC_SUBDIR, DOT_OR_USCORE, | - | | EXPT_CONFIG_FN, CONSTANTS_FN, RGNL_GRID_NML_FN, | - | | FV3_NML_BASE_SUITE_FN, FV3_NML_YAML_CONFIG_FN, FV3_NML_BASE_ENS_FN, | - | | FV3_EXEC_FN, DIAG_TABLE_TMPL_FN, FIELD_TABLE_TMPL_FN, | - | | DATA_TABLE_TMPL_FN, MODEL_CONFIG_TMPL_FN, NEMS_CONFIG_TMPL_FN, | - | | FCST_MODEL, WFLOW_XML_FN, GLOBAL_VAR_DEFNS_FN, | - | | EXTRN_MDL_VAR_DEFNS_FN, WFLOW_LAUNCH_SCRIPT_FN, WFLOW_LAUNCH_LOG_FN, | - | | CCPP_PHYS_SUITE, GRID_GEN_METHOD, DATE_FIRST_CYCL, DATE_LAST_CYCL, | - | | INCR_CYCL_FREQ, FCST_LEN_HRS, GET_OBS, MAXTRIES_VX_ENSGRID_PROB_REFC, | - | | PREEXISTING_DIR_METHOD, VERBOSE, DEBUG, COMPILER | - +-----------------------------+-----------------------------------------------------------------------+ - | NCO | envir, NET, model_ver, RUN, OPSROOT | - +-----------------------------+-----------------------------------------------------------------------+ - | task_make_grid | GRID_DIR, ESGgrid_LON_CTR, ESGgrid_LAT_CTR, | - | | ESGgrid_DELX, ESGgrid_DELY, ESGgrid_NX, ESGgrid_NY, ESGgrid_PAZI, | - | | ESGgrid_WIDE_HALO_WIDTH, GFDLgrid_LON_T6_CTR, GFDLgrid_LAT_T6_CTR, | - | | GFDLgrid_NUM_CELLS, GFDLgrid_STRETCH_FAC, GFDLgrid_REFINE_RATIO, | - | | GFDLgrid_ISTART_OF_RGNL_DOM_ON_T6G, GFDLgrid_IEND_OF_RGNL_DOM_ON_T6G, | - | | GFDLgrid_JSTART_OF_RGNL_DOM_ON_T6G, GFDLgrid_JEND_OF_RGNL_DOM_ON_T6G, | - | | GFDLgrid_USE_NUM_CELLS_IN_FILENAMES | - +-----------------------------+-----------------------------------------------------------------------+ - | task_make_orog | KMP_AFFINITY_MAKE_OROG, OMP_NUM_THREADS_MAKE_OROG | - | | OMP_STACKSIZE_MAKE_OROG, OROG_DIR | - +-----------------------------+-----------------------------------------------------------------------+ - | task_make_sfc_climo | KMP_AFFINITY_MAKE_SFC_CLIMO, OMP_NUM_THREADS_MAKE_SFC_CLIMO, | - | | OMP_STACKSIZE_MAKE_SFC_CLIMO, SFC_CLIMO_DIR | - +-----------------------------+-----------------------------------------------------------------------+ - | task_get_extrn_ics | EXTRN_MDL_NAME_ICS, EXTRN_MDL_ICS_OFFSET_HRS, FV3GFS_FILE_FMT_ICS, | - | | EXTRN_MDL_SYSBASEDIR_ICS, USE_USER_STAGED_EXTRN_FILES, | - | | EXTRN_MDL_SOURCE_BASEDIR_ICS, EXTRN_MDL_FILES_ICS, | - | | EXTRN_MDL_FILES_ICS, EXTRN_MDL_FILES_ICS, EXTRN_MDL_DATA_STORES, | - | | NOMADS, NOMADS_file_type | - +-----------------------------+-----------------------------------------------------------------------+ - | task_get_extrn_lbcs | EXTRN_MDL_NAME_LBCS, | - | | LBC_SPEC_INTVL_HRS, EXTRN_MDL_LBCS_OFFSET_HRS, FV3GFS_FILE_FMT_LBCS, | - | | EXTRN_MDL_SYSBASEDIR_LBCS, USE_USER_STAGED_EXTRN_FILES, | - | | EXTRN_MDL_SOURCE_BASEDIR_LBCS, EXTRN_MDL_FILES_LBCS, | - | | EXTRN_MDL_DATA_STORE, NOMADS, NOMADS_file_type | - +-----------------------------+-----------------------------------------------------------------------+ - | task_make_ics | KMP_AFFINITY_MAKE_ICS, OMP_NUM_THREADS_MAKE_ICS, | - | | OMP_STACKSIZE_MAKE_ICS, USE_FVCOM, FVCOM_WCSTART, FVCOM_DIR, | - | | FVCOM_FILE | - +-----------------------------+-----------------------------------------------------------------------+ - | task_make_lbcs | KMP_AFFINITY_MAKE_LBCS, OMP_NUM_THREADS_MAKE_LBCS, | - | | OMP_STACKSIZE_MAKE_LBCS | - +-----------------------------+-----------------------------------------------------------------------+ - | task_run_fcst | KMP_AFFINITY_RUN_FCST, OMP_NUM_THREADS_RUN_FCST, | - | | OMP_STACKSIZE_RUN_FCST, DT_ATMOS, RESTART_INTERVAL, WRITE_DOPOST, | - | | LAYOUT_X, LAYOUT_Y, BLOCKSIZE, QUILTING, PRINT_ESMF, | - | | WRTCMP_write_groups, WRTCMP_write_tasks_per_group, | - | | WRTCMP_cen_lon, WRTCMP_cen_lat, WRTCMP_lon_lwr_left, | - | | WRTCMP_lat_lwr_left, WRTCMP_lon_upr_rght, WRTCMP_lat_upr_rght, | - | | WRTCMP_dlon, WRTCMP_dlat, WRTCMP_stdlat1, WRTCMP_stdlat2, WRTCMP_nx, | - | | WRTCMP_ny, WRTCMP_dx, WRTCMP_dy, PREDEF_GRID_NAME, USE_MERRA_CLIMO, | - | | SFC_CLIMO_FIELDS, FIXgsm, FIXaer, FIXlut, TOPO_DIR, | - | | SFC_CLIMO_INPUT_DIR, SYMLINK_FIX_FILES, FNGLAC, FNMXIC, FNTSFC, | - | | FNSNOC, FNZORC, FNAISC, FNSMCC, FNMSKH, FIXgsm_FILES_TO_COPY_TO_FIXam,| - | | FV3_NML_VARNAME_TO_FIXam_FILES_MAPPING, | - | | FV3_NML_VARNAME_TO_SFC_CLIMO_FIELD_MAPPING, | - | | CYCLEDIR_LINKS_TO_FIXam_FILES_MAPPING | - +-----------------------------+-----------------------------------------------------------------------+ - | task_run_post | KMP_AFFINITY_RUN_POST, OMP_NUM_THREADS_RUN_POST, | - | | OMP_STACKSIZE_RUN_POST, SUB_HOURLY_POST, DT_SUB_HOURLY_POST_MNTS, | - | | USE_CUSTOM_POST_CONFIG_FILE, CUSTOM_POST_CONFIG_FP, | - | | POST_OUTPUT_DOMAIN_NAME | - +-----------------------------+-----------------------------------------------------------------------+ - | Global | USE_CRTM, CRTM_DIR, DO_ENSEMBLE, NUM_ENS_MEMBERS, | - | | NEW_LSCALE, DO_SHUM, ISEED_SHUM, SHUM_MAG, SHUM_LSCALE, SHUM_TSCALE, | - | | SHUM_INT, DO_SPPT, ISEED_SPPT, SPPT_MAG, SPPT_LOGIT, SPPT_LSCALE, | - | | SPPT_TSCALE, SPPT_INT, SPPT_SFCLIMIT, USE_ZMTNBLCK, DO_SKEB, | - | | ISEED_SKEB, SKEB_MAG, SKEB_LSCALE, SKEP_TSCALE, SKEB_INT, SKEBNORM, | - | | SKEB_VDOF, DO_SPP, ISEED_SPP, SPP_VAR_LIST, SPP_MAG_LIST, SPP_LSCALE, | - | | SPP_TSCALE, SPP_SIGTOP1, SPP_SIGTOP2, SPP_STDDEV_CUTOFF, DO_LSM_SPP, | - | | LSM_SPP_TSCALE, LSM_SPP_LSCALE, ISEED_LSM_SPP, LSM_SPP_VAR_LIST, | - | | LSM_SPP_MAG_LIST, HALO_BLEND | - +-----------------------------+-----------------------------------------------------------------------+ - -.. _UserSpecificConfig: - -User-specific configuration: ``config.yaml`` -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -The user must specify certain basic experiment configuration information in a ``config.yaml`` file located in the ``ufs-srweather-app/ush`` directory. Two example templates are provided in that directory: ``config.community.yaml`` and ``config.nco.yaml``. The first file is a minimal example for creating and running an experiment in *community* mode (with ``RUN_ENVIR`` set to ``community``). The second is an example for creating and running an experiment in the *NCO* (operational) mode (with ``RUN_ENVIR`` set to ``nco``). The *community* mode is recommended in most cases and is fully supported for this release. The operational/NCO mode is typically used by developers at the Environmental Modeling Center (:term:`EMC`) and at the Global Systems Laboratory (:term:`GSL`) working on pre-implementation testing for the Rapid Refresh Forecast System (RRFS). :numref:`Table %s ` compares the configuration variables that appear in the ``config.community.yaml`` with their default values in ``config_default.yaml``. - -.. _ConfigCommunity: - -.. table:: Configuration variables specified in the config.community.yaml script - - +--------------------------------+-------------------+------------------------------------+ - | **Parameter** | **Default Value** | **config.community.yaml Value** | - +================================+===================+====================================+ - | RUN_ENVIR | "nco" | "community" | - +--------------------------------+-------------------+------------------------------------+ - | MACHINE | "BIG_COMPUTER" | "hera" | - +--------------------------------+-------------------+------------------------------------+ - | ACCOUNT | "project_name" | "an_account" | - +--------------------------------+-------------------+------------------------------------+ - | MODEL | "" | "FV3_GFS_v16_CONUS_25km" | - +--------------------------------+-------------------+------------------------------------+ - | CCPA_OBS_DIR | "" | "" | - +--------------------------------+-------------------+------------------------------------+ - | NOHRSC_OBS_DIR | "" | "" | - +--------------------------------+-------------------+------------------------------------+ - | MRMS_OBS_DIR | "" | "" | - +--------------------------------+-------------------+------------------------------------+ - | NDAS_OBS_DIR | "" | "" | - +--------------------------------+-------------------+------------------------------------+ - | EXPT_SUBDIR | "" | "test_community" | - +--------------------------------+-------------------+------------------------------------+ - | CCPP_PHYS_SUITE | "FV3_GFS_v16" | "FV3_GFS_v16" | - +--------------------------------+-------------------+------------------------------------+ - | DATE_FIRST_CYCL | "YYYYMMDDHH" | '2019061518' | - +--------------------------------+-------------------+------------------------------------+ - | DATE_LAST_CYCL | "YYYYMMDDHH" | '2019061518' | - +--------------------------------+-------------------+------------------------------------+ - | FCST_LEN_HRS | 24 | 12 | - +--------------------------------+-------------------+------------------------------------+ - | PREEXISTING_DIR_METHOD | "delete" | "rename" | - +--------------------------------+-------------------+------------------------------------+ - | VERBOSE | true | true | - +--------------------------------+-------------------+------------------------------------+ - | COMPILER | "intel" | "intel" | - +--------------------------------+-------------------+------------------------------------+ - | EXTRN_MDL_NAME_ICS | "FV3GFS" | "FV3GFS" | - +--------------------------------+-------------------+------------------------------------+ - | FV3GFS_FILE_FMT_ICS | "nemsio" | "grib2" | - +--------------------------------+-------------------+------------------------------------+ - | EXTRN_MDL_NAME_LBCS | "FV3GFS" | "FV3GFS" | - +--------------------------------+-------------------+------------------------------------+ - | FV3GFS_FILE_FMT_LBCS | "nemsio" | "grib2" | - +--------------------------------+-------------------+------------------------------------+ - | LBC_SPEC_INTVL_HRS | 6 | 6 | - +--------------------------------+-------------------+------------------------------------+ - | WTIME_RUN_FCST | "04:30:00" | "02:00:00" | - +--------------------------------+-------------------+------------------------------------+ - | QUILTING | true | true | - +--------------------------------+-------------------+------------------------------------+ - | PREDEF_GRID_NAME | "" | "RRFS_CONUS_25km" | - +--------------------------------+-------------------+------------------------------------+ - | DO_ENSEMBLE | false | false | - +--------------------------------+-------------------+------------------------------------+ - | NUM_ENS_MEMBERS | 1 | 2 | - +--------------------------------+-------------------+------------------------------------+ - - -To get started, make a copy of ``config.community.yaml``. From the ``ufs-srweather-app`` directory, run: - -.. code-block:: console - - cd /path/to/ufs-srweather-app/ush - cp config.community.yaml config.yaml - -The default settings in this file include a predefined 25-km :term:`CONUS` grid (RRFS_CONUS_25km), the :term:`GFS` v16 physics suite (FV3_GFS_v16 :term:`CCPP`), and :term:`FV3`-based GFS raw external model data for initialization. - -.. note:: - - Users who have a previous configuration using the former shell-script-based can convert their ``config.sh`` file to a ``config.yaml`` file by running: - - .. code-block:: console - - ./config_utils.py -c $PWD/config.sh -t $PWD/config_defaults.yaml -o yaml >config.yaml - - Use caution when upgrading your experiment in this way, as many intervening changes in the workflow have occurred since the python changeover. - -Next, users should edit the new ``config.yaml`` file to customize it for their machine. At a minimum, users must change the ``MACHINE`` and ``ACCOUNT`` variables. Then, they can choose a name for the experiment directory by setting ``EXPT_SUBDIR``. If users have pre-staged initialization data for the experiment, they can set ``USE_USER_STAGED_EXTRN_FILES: true``, and set the paths to the data for ``EXTRN_MDL_SOURCE_BASEDIR_ICS`` and ``EXTRN_MDL_SOURCE_BASEDIR_LBCS``. If the modulefile used to set up the build environment in :numref:`Section %s ` uses a GNU compiler, check that the line ``COMPILER: "gnu"`` appears in the ``workflow:`` section of the ``config.yaml`` file. On platforms where Rocoto and :term:`cron` are available, users can automate resubmission of their experiment workflow by adding the following lines to the ``workflow:`` section of the ``config.yaml`` file: - -.. code-block:: console - - USE_CRON_TO_RELAUNCH: true - CRON_RELAUNCH_INTVL_MNTS: 3 - -.. note:: - - Generic Linux and MacOS users should refer to :numref:`Section %s ` for additional details on configuring an experiment and python environment. - -Detailed information on additional parameter options can be viewed in :numref:`Chapter %s: Configuring the Workflow `. Additionally, information about the four predefined Limited Area Model (LAM) Grid options can be found in :numref:`Chapter %s: Limited Area Model (LAM) Grids `. - -On Level 1 systems, the following fields typically need to be updated or added to the appropriate section of the ``config.yaml`` file in order to run the out-of-the-box SRW App case: - -.. code-block:: console - - user: - MACHINE: hera - ACCOUNT: an_account - workflow: - EXPT_SUBDIR: test_community - task_get_extrn_ics: - USE_USER_STAGED_EXTRN_FILES: true - EXTRN_MDL_SOURCE_BASEDIR_ICS: "/path/to/UFS_SRW_App/develop/input_model_data///" - task_get_extrn_lbcs: - USE_USER_STAGED_EXTRN_FILES: true - EXTRN_MDL_SOURCE_BASEDIR_LBCS: "/path/to/UFS_SRW_App/develop/input_model_data///" - -where: - * ``MACHINE`` refers to a valid machine name (see :numref:`Section %s ` for options). - * ``ACCOUNT`` refers to a valid account name. Not all systems require a valid account name, but most do. - - .. hint:: - - To determine an appropriate ACCOUNT field for Level 1 systems, run ``groups``, and it will return a list of projects you have permissions for. Not all of the listed projects/groups have an HPC allocation, but those that do are potentially valid account names. - - * ``EXPT_SUBDIR`` is changed to an experiment name of the user's choice. - * ```` is the path to the SRW App data on the user's machine (see :numref:`Section %s `). - * ```` refers to a subdirectory containing the experiment data from a particular model. Valid values on Level 1 systems correspond to the valid values for ``EXTRN_MDL_NAME_ICS`` and ``EXTRN_MDL_NAME_LBCS`` (see :numref:`Chapter %s ` for options). - * ```` refers to one of 3 possible data formats: ``grib2``, ``nemsio``, or ``netcdf``. - * ```` refers to a subdirectory containing data for the :term:`cycle` date (in YYYYMMDDHH format). - -.. note:: - - On ``JET``, users should also add ``PARTITION_DEFAULT: xjet`` and ``PARTITION_FCST: xjet`` to the ``platform:`` section of the ``config.yaml`` file. - -For example, to run the out-of-the-box experiment on Gaea, add or modify variables in the ``user``, ``workflow``, ``task_get_extrn_ics``, and ``task_get_extrn_lbcs`` sections of ``config.yaml`` (unmodified variables are not shown in this example): - - .. code-block:: - - user: - MACHINE: gaea - ACCOUNT: hfv3gfs - workflow: - EXPT_SUBDIR: run_basic_srw - task_get_extrn_ics: - USE_USER_STAGED_EXTRN_FILES: true - EXTRN_MDL_SOURCE_BASEDIR_ICS: /lustre/f2/dev/role.epic/contrib/UFS_SRW_data/develop/input_model_data/FV3GFS/grib2/2019061518 - EXTRN_MDL_DATA_STORES: disk - task_get_extrn_lbcs: - USE_USER_STAGED_EXTRN_FILES: true - EXTRN_MDL_SOURCE_BASEDIR_LBCS: /lustre/f2/dev/role.epic/contrib/UFS_SRW_data/develop/input_model_data/FV3GFS/grib2/2019061518 - EXTRN_MDL_DATA_STORES: disk - -To determine whether the ``config.yaml`` file adjustments are valid, users can run the following script from the ``ush`` directory: - -.. code-block:: console - - ./config_utils.py -c $PWD/config.yaml -v $PWD/config_defaults.yaml - -A correct ``config.yaml`` file will output a ``SUCCESS`` message. A ``config.yaml`` file with problems will output a ``FAILURE`` message describing the problem. For example: - -.. code-block:: console - - INVALID ENTRY: EXTRN_MDL_FILES_ICS=[] - FAILURE - -.. note:: - - The regional workflow must be loaded for the ``config_utils.py`` script to validate the ``config.yaml`` file. - -Valid values for configuration variables should be consistent with those in the ``ush/valid_param_vals.yaml`` script. In addition, various sample configuration files can be found within the subdirectories of ``tests/WE2E/test_configs``. - -To configure an experiment and python environment for a general Linux or Mac system, see the :ref:`next section `. To configure an experiment to run METplus verification tasks, see :numref:`Section %s `. Otherwise, skip to :numref:`Section %s ` to generate the workflow. - -.. _PlotOutput: - -Plotting Configuration (optional) -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -An optional Python plotting task (PLOT_ALLVARS) can be activated in the workflow to generate plots for the :term:`FV3`-:term:`LAM` post-processed :term:`GRIB2` -output over the :term:`CONUS`. It generates graphics plots for a number of variables, including: - - * 2-m temperature - * 2-m dew point temperature - * 10-m winds - * 250 hPa winds - * Accumulated precipitation - * Composite reflectivity - * Surface-based :term:`CAPE`/:term:`CIN` - * Max/Min 2-5 km updraft helicity - * Sea level pressure (SLP) - -.. COMMENT: * 500 hPa heights, winds, and vorticity --> seems to be omitted? Why? - -This workflow task can produce both plots from a single experiment and difference plots that compare the same cycle from two experiments. When plotting the difference, the two experiments must be on the same domain and available for -the same cycle starting date/time and forecast hours. Other parameters may differ (e.g., the experiments may use different physics suites). - -.. _Cartopy: - -Cartopy Shapefiles -````````````````````` - -The Python plotting tasks require a path to the directory where the Cartopy Natural Earth shapefiles are located. The medium scale (1:50m) cultural and physical shapefiles are used to create coastlines and other geopolitical borders on the map. On `Level 1 `__ systems, this path is already set in the system's machine file using the variable ``FIXshp``. Users on other systems will need to download the shapefiles and update the path of ``$FIXshp`` in the machine file they are using (e.g., ``$SRW/ush/machine/macos.yaml`` for a generic MacOS system, where ``$SRW`` is the path to the ``ufs-srweather-app`` directory). The subset of shapefiles required for the plotting task can be obtained from the `SRW Data Bucket `__. The full set of medium-scale (1:50m) Cartopy shapefiles can be downloaded `here `__. - -Task Configuration -````````````````````` - -Users will need to add or modify certain variables in ``config.yaml`` to run the plotting task(s). At a minimum, to activate the ``plot_allvars`` tasks, users must add it to the default list of ``taskgroups`` under the ``rocoto: tasks:`` section. - -.. code-block:: console - - rocoto: - tasks: - taskgroups: '{{ ["parm/wflow/prep.yaml", "parm/wflow/coldstart.yaml", "parm/wflow/post.yaml", "parm/wflow/plot.yaml"]|include }}' - -Users may also wish to adjust the start, end, and increment value for the plotting task. For example: - -.. code-block:: console - - task_plot_allvars: - PLOT_FCST_START: 0 - PLOT_FCST_INC: 6 - PLOT_FCST_END: 12 - -If the user chooses not to set these values, the default values will be used (see :numref:`Section %s `). - -.. note:: - If a forecast starts at 18h, this is considered the 0th forecast hour, so "starting forecast hour" should be 0, not 18. - -When plotting output from a single experiment, no further adjustments are necessary. The output files (in ``.png`` format) will be located in the experiment directory under the ``$CDATE/postprd`` subdirectory where ``$CDATE`` -corresponds to the cycle date and hour in YYYYMMDDHH format (e.g., ``2019061518``). - -Plotting the Difference Between Two Experiments -"""""""""""""""""""""""""""""""""""""""""""""""""" - -When plotting the difference between two experiments (``expt1`` and ``expt2``), users must set the ``COMOUT_REF`` template variable in ``expt2``'s ``config.yaml`` file to point at forecast output from the ``expt1`` directory. For example, in *community* mode, users can set ``COMOUT_REF`` as follows in the ``expt2`` configuration file: - -.. code-block:: console - - task_plot_allvars: - COMOUT_REF: '${EXPT_BASEDIR}/expt1/${PDY}${cyc}/postprd' - -This will ensure that ``expt2`` can produce a difference plot comparing ``expt1`` and ``expt2``. In *community* mode, using default directory names and settings, ``$COMOUT_REF`` will resemble ``/path/to/expt_dirs/test_community/2019061518/postprd``. Additional details on the plotting variables are provided in :numref:`Section %s `. - -The output files (in ``.png`` format) will be located in the ``postprd`` directory for the experiment. - -.. _LinuxMacEnvConfig: - -User-Specific Configuration on a Generic Linux/MacOS System -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -The configuration process for Linux and MacOS systems is similar to the process for other systems, but it requires a few extra steps. - -.. note:: - Examples in this subsection presume that the user is running in the Terminal with a bash shell environment. If this is not the case, users will need to adjust the commands to fit their command line application and shell environment. - -.. _MacMorePackages: - -Install/Upgrade Mac-Specific Packages -```````````````````````````````````````` -MacOS requires the installation of a few additional packages and, possibly, an upgrade to bash. Users running on MacOS should execute the following commands: - -.. code-block:: console - - bash --version - brew install bash # or: brew upgrade bash - brew install coreutils - brew gsed # follow directions to update the PATH env variable - -.. _LinuxMacVEnv: - -Creating a *conda* Environment on Linux and Mac -`````````````````````````````````````````````````` - -Users need to create a conda ``regional_workflow`` environment. The environment can be stored in a local path, which could be a default location or a user-specified location (e.g. ``$HOME/condaenv/venvs/`` directory). (To determine the default location, use the ``conda info`` command, and look for the ``envs directories`` list.) A brief recipe for creating a virtual conda environment on non-Level 1 platforms: - -.. code-block:: console - - conda create --name regional_workflow python= - conda activate regional_workflow - conda install -c conda-forge f90nml - conda install jinja2 - conda install pyyaml - # install packages for graphics environment - conda install scipy - conda install matplotlib - conda install -c conda-forge pygrib - conda install cartopy - # verify the packages installed - conda list - conda deactivate - -where ```` is a numeric version (e.g. ``3.9.12``) in the conda base installation resulting from the query ``python3 --version``. - -.. _LinuxMacExptConfig: - -Configuring an Experiment on General Linux and MacOS Systems -`````````````````````````````````````````````````````````````` - -**Optional: Install Rocoto** - -.. note:: - Users may `install Rocoto `__ if they want to make use of a workflow manager to run their experiments. However, this option has not yet been tested on MacOS and has had limited testing on general Linux plaforms. - - -**Configure the SRW App:** - -Configure an experiment using a template. Copy the contents of ``config.community.yaml`` into ``config.yaml``: - -.. code-block:: console - - cd /path/to/ufs-srweather-app/ush - cp config.community.yaml config.yaml - -In the ``config.yaml`` file, set ``MACHINE: macos`` or ``MACHINE: linux``, and modify the account and experiment info. For example: - -.. code-block:: console - - user: - RUN_ENVIR: community - MACHINE: macos - ACCOUNT: user - workflow: - EXPT_SUBDIR: test_community - PREEXISTING_DIR_METHOD: rename - VERBOSE: true - COMPILER: gnu - task_get_extrn_ics: - USE_USER_STAGED_EXTRN_FILES: true - EXTRN_MDL_SOURCE_BASEDIR_ICS: /path/to/input_model_data/FV3GFS/grib2/2019061518 - EXTRN_MDL_DATA_STORES: disk - task_get_extrn_lbcs: - USE_USER_STAGED_EXTRN_FILES: true - EXTRN_MDL_SOURCE_BASEDIR_LBCS: /path/to/input_model_data/FV3GFS/grib2/2019061518 - EXTRN_MDL_DATA_STORES: disk - task_run_fcst: - PREDEF_GRID_NAME: RRFS_CONUS_25km - QUILTING: true - -Due to the limited number of processors on MacOS systems, users must also configure the domain decomposition parameters directly in the section of the ``predef_grid_params.yaml`` file pertaining to the grid they want to use. Domain decomposition needs to take into account the number of available CPUs and configure the variables ``LAYOUT_X``, ``LAYOUT_Y``, and ``WRTCMP_write_tasks_per_group`` accordingly. - -The example below is for systems with 8 CPUs: - -.. code-block:: console - - task_run_fcst: - LAYOUT_X: 3 - LAYOUT_Y: 2 - WRTCMP_write_tasks_per_group: 2 - -.. note:: - The number of MPI processes required by the forecast will be equal to ``LAYOUT_X`` * ``LAYOUT_Y`` + ``WRTCMP_write_tasks_per_group``. - -For a machine with 4 CPUs, the following domain decomposition could be used: - -.. code-block:: console - - task_run_fcst: - LAYOUT_X: 3 - LAYOUT_Y: 1 - WRTCMP_write_tasks_per_group: 1 - -**Configure the Machine File** - -Configure a ``macos.yaml`` or ``linux.yaml`` machine file in ``ufs-srweather-app/ush/machine`` based on the number of CPUs (``NCORES_PER_NODE``) in the system (usually 8 or 4 in MacOS; varies on Linux systems). Job scheduler (``SCHED``) options can be viewed :ref:`here `. Users must also set the path to the fix file directories. - -.. code-block:: console - - platform: - # Architecture information - WORKFLOW_MANAGER: none - NCORES_PER_NODE: 8 - SCHED: none - # Run commands for executables - RUN_CMD_FCST: 'mpirun -np ${PE_MEMBER01}' - RUN_CMD_POST: 'mpirun -np 4' - RUN_CMD_SERIAL: time - RUN_CMD_UTILS: 'mpirun -np 4' - # Commands to run at the start of each workflow task. - PRE_TASK_CMDS: '{ ulimit -a; }' - - task_make_orog: - # Path to location of static input files used by the make_orog task - TOPO_DIR: path/to/FIXgsm/files - - task_make_sfc_climo: - # Path to location of static surface climatology input fields used by sfc_climo_gen - SFC_CLIMO_INPUT_DIR: path/to/FIXgsm/files - - task_run_fcst: - FIXaer: /path/to/FIXaer/files - FIXgsm: /path/to/FIXgsm/files - FIXlut: /path/to/FIXlut/files - - data: - # Used by setup.py to set the values of EXTRN_MDL_SOURCE_BASEDIR_ICS and EXTRN_MDL_SOURCE_BASEDIR_LBCS - FV3GFS: /Users/username/DATA/UFS/FV3GFS - -The ``data:`` section of the machine file can point to various data sources that the user has pre-staged on disk. For example: - -.. code-block:: console - - data: - FV3GFS: - nemsio: /Users/username/DATA/UFS/FV3GFS/nemsio - grib2: /Users/username/DATA/UFS/FV3GFS/grib2 - netcdf: /Users/username/DATA/UFS/FV3GFS/netcdf - RAP: /Users/username/DATA/UFS/RAP/grib2 - HRRR: /Users/username/DATA/UFS/HRRR/grib2 - -This can be helpful when conducting multiple experiments with different types of data. - -.. _VXConfig: - -Configure METplus Verification Suite (Optional) -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -Users who want to use the METplus verification suite to evaluate their forecasts need to add additional information to their ``ush/machine/.yaml`` or ``config.yaml`` file. Other users may skip to the :ref:`next section `. - -.. note:: - If METplus users update their METplus installation, they must update the module load statements in ``ufs-srweather-app/modulefiles/tasks//run_vx.local`` file to correspond to their system's updated installation: - - .. code-block:: console - - module use -a - module load met/ - module load metplus/ - -.. note:: - PRELIMINARY CHANGES, NEEDS TO BE UPDATE IN A SECTION BELOW: for the recent changes in develop, there are several verify_*.yaml files, verify_pre.yaml, verify_ens.yaml, verify_det.yaml. Documentation below still mentions a single `veryfy.yaml` file. - -To use METplus verification, MET and METplus modules need to be installed. To turn on verification tasks in the workflow, include the ``parm/wflow/verify.yaml`` file in the ``rocoto: tasks: taskgroups:`` section of ``config.yaml``. - -.. code-block:: console - - rocoto: - tasks: - taskgroups: '{{ ["parm/wflow/prep.yaml", "parm/wflow/coldstart.yaml", "parm/wflow/post.yaml", "parm/wflow/verify.yaml"]|include }}' - -The ``verify.yaml`` file includes the definitions of several common verification tasks by default. They are independent of each other, so users may want to turn some off depending on the needs of their experiment. Note that the ENSGRID and ENSPOINT tasks apply only to ensemble model verification. Additional verification tasks appear in :numref:`Table %s `. - -To turn off a task, simply include its entry from ``verify.yaml`` as an empty YAML entry. For example, to turn off PointStat tasks: - -.. code-block:: console - - rocoto: - tasks: - taskgroups: '{{ ["parm/wflow/prep.yaml", "parm/wflow/coldstart.yaml", "parm/wflow/post.yaml", "parm/wflow/verify.yaml"]|include }}' - metatask_vx_ens_member: - metatask_PointStat_mem#mem#: - - -More information about configuring the ``rocoto:`` section can be found in :numref:`Section %s `. - -If users have access to NOAA :term:`HPSS` but have not pre-staged the data, the default ``verify.yaml`` taskgroup will activate the tasks, and the workflow will attempt to download the appropriate data from NOAA HPSS. In this case, the ``*_OBS_DIR`` paths must be set to the location where users want the downloaded data to reside. - -Users who do not have access to NOAA HPSS and do not have the data on their system will need to download :term:`CCPA`, :term:`MRMS`, and :term:`NDAS` data manually from collections of publicly available data, such as the ones listed `here `__. - -Users who have already staged the observation data needed for METplus (i.e., the :term:`CCPA`, :term:`MRMS`, and :term:`NDAS` data) on their system should set the path to this data and turn off the corresponding task by including them with no entry in ``config.yaml``. - -.. code-block:: console - - platform: - CCPA_OBS_DIR: /path/to/UFS_SRW_App/develop/obs_data/ccpa/proc - NOHRSC_OBS_DIR: /path/to/UFS_SRW_App/develop/obs_data/nohrsc/proc - MRMS_OBS_DIR: /path/to/UFS_SRW_App/develop/obs_data/mrms/proc - NDAS_OBS_DIR: /path/to/UFS_SRW_App/develop/obs_data/ndas/proc - rocoto: - tasks: - taskgroups: '{{ ["parm/wflow/prep.yaml", "parm/wflow/coldstart.yaml", "parm/wflow/post.yaml", "parm/wflow/verify.yaml"]|include }}' - task_get_obs_ccpa: - task_get_obs_nohrsc: - task_get_obs_mrms: - task_get_obs_ndas: - - - -.. _GenerateWorkflow: - -Generate the Regional Workflow -------------------------------------------- - -Run the following command from the ``ufs-srweather-app/ush`` directory to generate the workflow: - -.. code-block:: console - - ./generate_FV3LAM_wflow.py - -The last line of output from this script, starting with ``*/1 * * * *`` or ``*/3 * * * *``, can be saved and :ref:`used later ` to automatically run portions of the workflow if users have the Rocoto workflow manager installed on their system. - -This workflow generation script creates an experiment directory and populates it with all the data needed to run through the workflow. The flowchart in :numref:`Figure %s ` describes the experiment generation process. First, ``generate_FV3LAM_wflow.py`` runs the ``setup.py`` script to set the configuration parameters. Second, it symlinks the time-independent (fix) files and other necessary data input files from their location to the experiment directory (``$EXPTDIR``). Third, it creates the input namelist file ``input.nml`` based on the ``input.nml.FV3`` file in the ``parm`` directory. Lastly, it creates the workflow XML file ``FV3LAM_wflow.xml`` that is executed when running the experiment with the Rocoto workflow manager. - -The ``setup.py`` script reads three other configuration scripts in order: (1) ``config_defaults.yaml`` (:numref:`Section %s `), (2) ``config.yaml`` (:numref:`Section %s `), and (3) ``set_predef_grid_params.py``. If a parameter is specified differently in these scripts, the file containing the last defined value will be used. - -The generated workflow will appear in ``$EXPTDIR``, where ``EXPTDIR=${EXPT_BASEDIR}/${EXPT_SUBDIR}``. These variables were specified in ``config_defaults.yaml`` and ``config.yaml`` in :numref:`Step %s `. The settings for these paths can also be viewed in the console output from the ``./generate_FV3LAM_wflow.py`` script or in the ``log.generate_FV3LAM_wflow`` file, which can be found in ``$EXPTDIR``. - -.. _WorkflowGeneration: - -.. figure:: _static/SRW_regional_workflow_gen.png - :alt: Flowchart of the workflow generation process. Scripts are called in the following order: source_util_funcs.sh (which calls bash_utils), then set_FV3nml_sfc_climo_filenames.py, set_FV3nml_ens_stoch_seeds.py, create_diag_table_file.py, and setup.py. setup.py calls several scripts: set_cycle_dates.py, set_grid_params_GFDLgrid.py, set_grid_params_ESGgrid.py, link_fix.py, set_ozone_param.py, set_thompson_mp_fix_files.py, config_defaults.yaml, config.yaml, and valid_param_vals.yaml. Then, it sets a number of variables, including FIXgsm, TOPO_DIR, and SFC_CLIMO_INPUT_DIR variables. Next, set_predef_grid_params.py is called, and the FIXam and FIXLAM directories are set, along with the forecast input files. The setup script also calls set_extrn_mdl_params.py, sets the GRID_GEN_METHOD with HALO, checks various parameters, and generates shell scripts. Then, the workflow generation script produces a YAML configuration file and generates the actual Rocoto workflow XML file from the template file (by calling uwtools set_template). The workflow generation script checks the crontab file and, if applicable, copies certain fix files to the experiment directory. Then, it copies templates of various input files to the experiment directory and sets parameters for the input.nml file. Finally, it generates the workflow. Additional information on each step appears in comments within each script. - - *Experiment Generation Description* - -.. _WorkflowTaskDescription: - -Description of Workflow Tasks --------------------------------- - -.. note:: - This section gives a general overview of workflow tasks. To begin running the workflow, skip to :numref:`Step %s ` - -:numref:`Figure %s ` illustrates the overall workflow. Individual tasks that make up the workflow are specified in the ``FV3LAM_wflow.xml`` file. :numref:`Table %s ` describes the function of each baseline task. The first three pre-processing tasks; ``MAKE_GRID``, ``MAKE_OROG``, and ``MAKE_SFC_CLIMO``; are optional. If the user stages pre-generated grid, orography, and surface climatology fix files, these three tasks can be skipped by removing the ``prep.yaml`` file from the default ``taskgroups`` entry in the ``config.yaml`` file before running the ``generate_FV3LAM_wflow.py`` script: - -.. code-block:: console - - rocoto: - tasks: - taskgroups: '{{ ["parm/wflow/coldstart.yaml", "parm/wflow/post.yaml"]|include }}' - -.. _WorkflowTasksFig: - -.. figure:: _static/SRW_wflow_flowchart.png - :alt: Flowchart of the workflow tasks. If the make_grid, make_orog, and make_sfc_climo tasks are toggled off, they will not be run. If toggled on, make_grid, make_orog, and make_sfc_climo will run consecutively by calling the corresponding exregional script in the scripts directory. The get_ics, get_lbcs, make_ics, make_lbcs, and run_fcst tasks call their respective exregional scripts. The run_post task will run, and if METplus verification tasks have been configured, those will run during post-processing by calling their exregional scripts. - - *Flowchart of the Workflow Tasks* - - -The ``FV3LAM_wflow.xml`` file runs the specific j-job scripts (``jobs/JREGIONAL_[task name]``) in the prescribed order when the experiment is launched via the ``launch_FV3LAM_wflow.sh`` script or the ``rocotorun`` command. Each j-job task has its own source script (or "ex-script") named ``exregional_[task name].sh`` in the ``scripts`` directory. Two database files named ``FV3LAM_wflow.db`` and ``FV3LAM_wflow_lock.db`` are generated and updated by the Rocoto calls. There is usually no need for users to modify these files. To relaunch the workflow from scratch, delete these two ``*.db`` files and then call the launch script repeatedly for each task. - - -.. _WorkflowTasksTable: - -.. table:: Baseline Workflow Tasks in the SRW App - - +----------------------+------------------------------------------------------------+ - | **Workflow Task** | **Task Description** | - +======================+============================================================+ - | make_grid | Pre-processing task to generate regional grid files. Only | - | | needs to be run once per experiment. | - +----------------------+------------------------------------------------------------+ - | make_orog | Pre-processing task to generate orography files. Only | - | | needs to be run once per experiment. | - +----------------------+------------------------------------------------------------+ - | make_sfc_climo | Pre-processing task to generate surface climatology files. | - | | Only needs to be run once per experiment. | - +----------------------+------------------------------------------------------------+ - | get_extrn_ics | Cycle-specific task to obtain external data for the | - | | initial conditions (ICs) | - +----------------------+------------------------------------------------------------+ - | get_extrn_lbcs | Cycle-specific task to obtain external data for the | - | | lateral boundary conditions (LBCs) | - +----------------------+------------------------------------------------------------+ - | make_ics | Generate initial conditions from the external data | - +----------------------+------------------------------------------------------------+ - | make_lbcs | Generate LBCs from the external data | - +----------------------+------------------------------------------------------------+ - | run_fcst | Run the forecast model (UFS Weather Model) | - +----------------------+------------------------------------------------------------+ - | run_post | Run the post-processing tool (UPP) | - +----------------------+------------------------------------------------------------+ - -In addition to the baseline tasks described in :numref:`Table %s ` above, users may choose to run some or all of the METplus verification tasks. These tasks are described in :numref:`Table %s ` below. - -.. _VXWorkflowTasksTable: - -.. table:: Verification (VX) Workflow Tasks in the SRW App - - +-----------------------+------------------------------------------------------------+ - | **Workflow Task** | **Task Description** | - +=======================+============================================================+ - | GET_OBS_CCPA | Retrieves and organizes hourly :term:`CCPA` data from NOAA | - | | HPSS. Can only be run if ``verify_pre.yaml`` is included | - | | in a ``tasksgroups`` list *and* user has access to NOAA | - | | :term:`HPSS` data. | - +-----------------------+------------------------------------------------------------+ - | GET_OBS_NOHRSC | Retrieves and organizes hourly :term:`NOHRSC` data from | - | | NOAA HPSS. Can only be run if ``verify_pre.yaml`` is | - | | included in a ``tasksgroups`` list *and* user has access | - | | to NOAA :term:`HPSS` data. ``ASNOW`` should also be added | - | | to the ``VX_FIELDS`` list. | - +-----------------------+------------------------------------------------------------+ - | GET_OBS_NDAS | Retrieves and organizes hourly :term:`NDAS` data from NOAA | - | | HPSS. Can only be run if ``verify_pre.yaml`` is included | - | | in a ``tasksgroups`` list *and* user has access to NOAA | - | | :term:`HPSS` data. | - +-----------------------+------------------------------------------------------------+ - | GET_OBS_MRMS | Retrieves and organizes hourly :term:`MRMS` composite | - | | reflectivity and :term:`echo top` data from NOAA HPSS. Can | - | | only be run if ``verify_pre.yaml`` is included in a | - | | ``tasksgroups`` list *and* user has access to NOAA | - | | :term:`HPSS` data. | - +-----------------------+------------------------------------------------------------+ - | VX_GRIDSTAT | Runs METplus grid-to-grid verification for 1-h accumulated | - | | precipitation | - +-----------------------+------------------------------------------------------------+ - | VX_GRIDSTAT_REFC | Runs METplus grid-to-grid verification for composite | - | | reflectivity | - +-----------------------+------------------------------------------------------------+ - | VX_GRIDSTAT_RETOP | Runs METplus grid-to-grid verification for :term:`echo top`| - +-----------------------+------------------------------------------------------------+ - | VX_GRIDSTAT_##h | Runs METplus grid-to-grid verification for 3-h, 6-h, and | - | | 24-h (i.e., daily) accumulated precipitation. Valid values | - | | for ``##`` are ``03``, ``06``, and ``24``. | - +-----------------------+------------------------------------------------------------+ - | VX_POINTSTAT | Runs METplus grid-to-point verification for surface and | - | | upper-air variables | - +-----------------------+------------------------------------------------------------+ - | VX_ENSGRID | Runs METplus grid-to-grid ensemble verification for 1-h | - | | accumulated precipitation. | - | | Can only be run if ``DO_ENSEMBLE: true`` and | - | | ``verify_ensgrid.yaml`` is included in a ``taskgroups`` | - | | list. | - +-----------------------+------------------------------------------------------------+ - | VX_ENSGRID_REFC | Runs METplus grid-to-grid ensemble verification for | - | | composite reflectivity. | - | | Can only be run if ``DO_ENSEMBLE: true`` and | - | | ``verify_ensgrid.yaml`` is included in a ``taskgroups`` | - | | list. | - +-----------------------+------------------------------------------------------------+ - | VX_ENSGRID_RETOP | Runs METplus grid-to-grid ensemble verification for | - | | :term:`echo top`. | - | | Can only be run if ``DO_ENSEMBLE: true`` and | - | | ``verify_ensgrid.yaml`` is included in a ``taskgroups`` | - | | list. | - +-----------------------+------------------------------------------------------------+ - | VX_ENSGRID_##h | Runs METplus grid-to-grid ensemble verification for 3-h, | - | | 6-h, and 24-h (i.e., daily) accumulated precipitation. | - | | Valid values for ``##`` are ``03``, ``06``, and ``24``. | - | | Can only be run if ``DO_ENSEMBLE: true`` and | - | | ``verify_ensgrid.yaml`` is included in a ``taskgroups`` | - | | list. | - +-----------------------+------------------------------------------------------------+ - | VX_ENSGRID_MEAN | Runs METplus grid-to-grid verification for ensemble mean | - | | 1-h accumulated precipitation. | - | | Can only be run if ``DO_ENSEMBLE: true`` and | - | | ``verify_ensgrid.yaml`` is included in a ``taskgroups`` | - | | list. | - +-----------------------+------------------------------------------------------------+ - | VX_ENSGRID_PROB | Runs METplus grid-to-grid verification for 1-h accumulated | - | | precipitation probabilistic output. | - | | Can only be run if ``DO_ENSEMBLE: true`` and | - | | ``verify_ensgrid.yaml`` is included in a ``taskgroups`` | - | | list. | - +-----------------------+------------------------------------------------------------+ - | VX_ENSGRID_MEAN_##h | Runs METplus grid-to-grid verification for ensemble mean | - | | 3-h, 6-h, and 24h (i.e., daily) accumulated precipitation. | - | | Valid values for ``##`` are ``03``, ``06``, and ``24``. | - | | Can only be run if ``DO_ENSEMBLE: true`` and | - | | ``verify_ensgrid.yaml`` is included in a ``taskgroups`` | - | | list. | - +-----------------------+------------------------------------------------------------+ - | VX_ENSGRID_PROB_##h | Runs METplus grid-to-grid verification for 3-h, 6-h, and | - | | 24h (i.e., daily) accumulated precipitation probabilistic | - | | output. Valid values for ``##`` are ``03``, ``06``, and | - | | ``24``. | - | | Can only be run if ``DO_ENSEMBLE: true`` and | - | | ``verify_ensgrid.yaml`` is included in a ``taskgroups`` | - | | list. | - +-----------------------+------------------------------------------------------------+ - | VX_ENSGRID_PROB_REFC | Runs METplus grid-to-grid verification for ensemble | - | | probabilities for composite reflectivity. | - | | Can only be run if ``DO_ENSEMBLE: true`` and | - | | ``verify_ensgrid.yaml`` is included in a ``taskgroups`` | - | | list. | - +-----------------------+------------------------------------------------------------+ - | VX_ENSGRID_PROB_RETOP | Runs METplus grid-to-grid verification for ensemble | - | | probabilities for :term:`echo top`. | - | | Can only be run if ``DO_ENSEMBLE: true`` and | - | | ``verify_ensgrid.yaml`` is included in a ``taskgroups`` | - | | list. | - +-----------------------+------------------------------------------------------------+ - | VX_ENSPOINT | Runs METplus grid-to-point ensemble verification for | - | | surface and upper-air variables. | - | | Can only be run if ``DO_ENSEMBLE: true`` and | - | | ``verify_ensgrid.yaml`` is included in a ``taskgroups`` | - | | list. | - +-----------------------+------------------------------------------------------------+ - | VX_ENSPOINT_MEAN | Runs METplus grid-to-point verification for ensemble mean | - | | surface and upper-air variables. | - | | Can only be run if ``DO_ENSEMBLE: true`` and | - | | ``verify_ensgrid.yaml`` is included in a ``taskgroups`` | - | | list. | - +-----------------------+------------------------------------------------------------+ - | VX_ENSPOINT_PROB | Runs METplus grid-to-point verification for ensemble | - | | probabilities for surface and upper-air variables. | - | | Can only be run if ``DO_ENSEMBLE: true`` and | - | | ``verify_ensgrid.yaml`` is included in a ``taskgroups`` | - | | list. | - +-----------------------+------------------------------------------------------------+ - - -.. _Run: - -Run the Workflow -======================= - -The workflow can be run using the Rocoto workflow manager (see :numref:`Section %s `) or using standalone wrapper scripts (see :numref:`Section %s `). - -.. attention:: - - If users are running the SRW App on a system that does not have Rocoto installed (e.g., `Level 3 & 4 `__ systems, such as MacOS or generic Linux systems), they should follow the process outlined in :numref:`Section %s ` instead of the instructions in this section. - - -.. _UseRocoto: - -Run the Workflow Using Rocoto --------------------------------- - -The information in this section assumes that Rocoto is available on the desired platform. All official HPC platforms for the UFS SRW App release make use of the Rocoto workflow management software for running experiments. However, if Rocoto is not available, it is still possible to run the workflow using stand-alone scripts according to the process outlined in :numref:`Section %s `. - -There are three ways to run the workflow with Rocoto: (1) automation via crontab (2) by calling the ``launch_FV3LAM_wflow.sh`` script, and (3) by manually issuing the ``rocotorun`` command. - -.. note:: - Users may find it helpful to review :numref:`Chapter %s ` to gain a better understanding of Rocoto commands and workflow management before continuing, but this is not required to run the experiment. - -Optionally, an environment variable can be set to navigate to the ``$EXPTDIR`` more easily. If the login shell is bash, it can be set as follows: - -.. code-block:: console - - export EXPTDIR=// - -If the login shell is csh/tcsh, it can be set using: - -.. code-block:: console - - setenv EXPTDIR // - - -.. _Automate: - -Automated Option -^^^^^^^^^^^^^^^^^^^ - -The simplest way to run the Rocoto workflow is to automate the process using a job scheduler such as :term:`Cron`. For automatic resubmission of the workflow at regular intervals (e.g., every 3 minutes), the user can add the following commands to their ``config.yaml`` file *before* generating the experiment: - -.. code-block:: console - - USE_CRON_TO_RELAUNCH: true - CRON_RELAUNCH_INTVL_MNTS: 3 - -This will automatically add an appropriate entry to the user's :term:`cron table` and launch the workflow. Alternatively, the user can add a crontab entry manually using the ``crontab -e`` command. As mentioned in :numref:`Section %s `, the last line of output from ``./generate_FV3LAM_wflow.py`` (starting with ``*/3 * * * *``), can be pasted into the crontab file. It can also be found in the ``$EXPTDIR/log.generate_FV3LAM_wflow`` file. The crontab entry should resemble the following: - -.. code-block:: console - - */3 * * * * cd && ./launch_FV3LAM_wflow.sh called_from_cron="TRUE" - -where ```` is changed to correspond to the user's ``$EXPTDIR``. The number ``3`` can be changed to a different positive integer and simply means that the workflow will be resubmitted every three minutes. - -.. hint:: - - * On NOAA Cloud instances, ``*/1 * * * *`` (or ``CRON_RELAUNCH_INTVL_MNTS: 1``) is the preferred option for cron jobs because compute nodes will shut down if they remain idle too long. If the compute node shuts down, it can take 15-20 minutes to start up a new one. - * On other NOAA HPC systems, admins discourage the ``*/1 * * * *`` due to load problems. ``*/3 * * * *`` (or ``CRON_RELAUNCH_INTVL_MNTS: 3``) is the preferred option for cron jobs on non-NOAA Cloud systems. - -To check the experiment progress: - -.. code-block:: console - - cd $EXPTDIR - rocotostat -w FV3LAM_wflow.xml -d FV3LAM_wflow.db -v 10 - -After finishing the experiment, open the crontab using ``crontab -e`` and delete the crontab entry. - -.. note:: - - On Orion, *cron* is only available on the orion-login-1 node, so users will need to work on that node when running *cron* jobs on Orion. - -.. _Success: - -The workflow run is complete when all tasks have "SUCCEEDED". If everything goes smoothly, users will eventually see a workflow status table similar to the following: - -.. code-block:: console - - CYCLE TASK JOBID STATE EXIT STATUS TRIES DURATION - ========================================================================================================== - 201906151800 make_grid 4953154 SUCCEEDED 0 1 5.0 - 201906151800 make_orog 4953176 SUCCEEDED 0 1 26.0 - 201906151800 make_sfc_climo 4953179 SUCCEEDED 0 1 33.0 - 201906151800 get_extrn_ics 4953155 SUCCEEDED 0 1 2.0 - 201906151800 get_extrn_lbcs 4953156 SUCCEEDED 0 1 2.0 - 201906151800 make_ics 4953184 SUCCEEDED 0 1 16.0 - 201906151800 make_lbcs 4953185 SUCCEEDED 0 1 71.0 - 201906151800 run_fcst 4953196 SUCCEEDED 0 1 1035.0 - 201906151800 run_post_f000 4953244 SUCCEEDED 0 1 5.0 - 201906151800 run_post_f001 4953245 SUCCEEDED 0 1 4.0 - ... - 201906151800 run_post_f012 4953381 SUCCEEDED 0 1 7.0 - -If users choose to run METplus verification tasks as part of their experiment, the output above will include additional lines after ``run_post_f012``. The output will resemble the following but may be significantly longer when using ensemble verification: - -.. code-block:: console - - CYCLE TASK JOBID STATE EXIT STATUS TRIES DURATION - ========================================================================================================== - 201906151800 make_grid 30466134 SUCCEEDED 0 1 5.0 - ... - 201906151800 run_post_f012 30468271 SUCCEEDED 0 1 7.0 - 201906151800 run_gridstatvx 30468420 SUCCEEDED 0 1 53.0 - 201906151800 run_gridstatvx_refc 30468421 SUCCEEDED 0 1 934.0 - 201906151800 run_gridstatvx_retop 30468422 SUCCEEDED 0 1 1002.0 - 201906151800 run_gridstatvx_03h 30468491 SUCCEEDED 0 1 43.0 - 201906151800 run_gridstatvx_06h 30468492 SUCCEEDED 0 1 29.0 - 201906151800 run_gridstatvx_24h 30468493 SUCCEEDED 0 1 20.0 - 201906151800 run_pointstatvx 30468423 SUCCEEDED 0 1 670.0 - - -Launch the Rocoto Workflow Using a Script -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -Users who prefer not to automate their experiments can run the Rocoto workflow using the ``launch_FV3LAM_wflow.sh`` script provided. Simply call it without any arguments from the experiment directory: - -.. code-block:: console - - cd $EXPTDIR - ./launch_FV3LAM_wflow.sh - -This script creates a log file named ``log.launch_FV3LAM_wflow`` in ``$EXPTDIR`` or appends information to the file if it already exists. The launch script also creates the ``log/FV3LAM_wflow.log`` file, which shows Rocoto task information. Check the end of the log file periodically to see how the experiment is progressing: - -.. code-block:: console - - tail -n 40 log.launch_FV3LAM_wflow - -In order to launch additional tasks in the workflow, call the launch script again; this action will need to be repeated until all tasks in the workflow have been launched. To (re)launch the workflow and check its progress on a single line, run: - -.. code-block:: console - - ./launch_FV3LAM_wflow.sh; tail -n 40 log.launch_FV3LAM_wflow - -This will output the last 40 lines of the log file, which list the status of the workflow tasks (e.g., SUCCEEDED, DEAD, RUNNING, SUBMITTING, QUEUED). The number 40 can be changed according to the user's preferences. The output will look similar to this: - -.. code-block:: console - - CYCLE TASK JOBID STATE EXIT STATUS TRIES DURATION - ====================================================================================================== - 201906151800 make_grid druby://hfe01:33728 SUBMITTING - 0 0.0 - 201906151800 make_orog - - - - - - 201906151800 make_sfc_climo - - - - - - 201906151800 get_extrn_ics druby://hfe01:33728 SUBMITTING - 0 0.0 - 201906151800 get_extrn_lbcs druby://hfe01:33728 SUBMITTING - 0 0.0 - 201906151800 make_ics - - - - - - 201906151800 make_lbcs - - - - - - 201906151800 run_fcst - - - - - - 201906151800 run_post_00 - - - - - - 201906151800 run_post_01 - - - - - - 201906151800 run_post_02 - - - - - - 201906151800 run_post_03 - - - - - - 201906151800 run_post_04 - - - - - - 201906151800 run_post_05 - - - - - - 201906151800 run_post_06 - - - - - - - Summary of workflow status: - ~~~~~~~~~~~~~~~~~~~~~~~~~~ - - 0 out of 1 cycles completed. - Workflow status: IN PROGRESS - -If all the tasks complete successfully, the "Workflow status" at the bottom of the log file will change from "IN PROGRESS" to "SUCCESS". If certain tasks could not complete, the "Workflow status" will instead change to "FAILURE". Error messages for each task can be found in the task log files located in ``$EXPTDIR/log``. - -The workflow run is complete when all tasks have "SUCCEEDED", and the ``rocotostat`` command outputs a table similar to the one :ref:`above `. - - -.. _RocotoManualRun: - -Launch the Rocoto Workflow Manually -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -**Load Rocoto** - -Instead of running the ``./launch_FV3LAM_wflow.sh`` script, users can load Rocoto and any other required modules manually. This gives the user more control over the process and allows them to view experiment progress more easily. On Level 1 systems, the Rocoto modules are loaded automatically in :numref:`Step %s `. For most other systems, users can load a modified ``wflow_`` modulefile, or they can use a variant on the following commands to load the Rocoto module: - -.. code-block:: console - - module use - module load rocoto - -Some systems may require a version number (e.g., ``module load rocoto/1.3.3``) - -**Run the Rocoto Workflow** - -After loading Rocoto, ``cd`` to the experiment directory and call ``rocotorun`` to launch the workflow tasks. This will start any tasks that do not have a dependency. As the workflow progresses through its stages, ``rocotostat`` will show the state of each task and allow users to monitor progress: - -.. code-block:: console - - cd $EXPTDIR - rocotorun -w FV3LAM_wflow.xml -d FV3LAM_wflow.db -v 10 - rocotostat -w FV3LAM_wflow.xml -d FV3LAM_wflow.db -v 10 - -The ``rocotorun`` and ``rocotostat`` commands above will need to be resubmitted regularly and repeatedly until the experiment is finished. In part, this is to avoid having the system time out. This also ensures that when one task ends, tasks dependent on it will run as soon as possible, and ``rocotostat`` will capture the new progress. - -If the experiment fails, the ``rocotostat`` command will indicate which task failed. Users can look at the log file in the ``log`` subdirectory for the failed task to determine what caused the failure. For example, if the ``make_grid`` task failed, users can open the ``make_grid.log`` file to see what caused the problem: - -.. code-block:: console - - cd $EXPTDIR/log - vi make_grid.log - -.. note:: - - If users have the `Slurm workload manager `__ on their system, they can run the ``squeue`` command in lieu of ``rocotostat`` to check what jobs are currently running. - - -.. _RunUsingStandaloneScripts: - -Run the Workflow Using Stand-Alone Scripts ---------------------------------------------- - -The regional workflow can be run using standalone shell scripts in cases where the Rocoto software is not available on a given platform. If Rocoto *is* available, see :numref:`Section %s ` to run the workflow using Rocoto. - -.. attention:: - - When working on an HPC system, users should allocate a compute node prior to running their experiment. The proper command will depend on the system's resource manager, but some guidance is offered in :numref:`Section %s `. It may be necessay to reload the regional workflow (see :numref:`Section %s `). It may also be necessary to load the ``build__`` scripts as described in :numref:`Section %s `. - -#. ``cd`` into the experiment directory. For example, from ``ush``, presuming default directory settings: - - .. code-block:: console - - cd ../../expt_dirs/test_community - -#. Set the environment variable ``$EXPTDIR`` for either bash or csh, respectively: - - .. code-block:: console - - export EXPTDIR=`pwd` - setenv EXPTDIR `pwd` - -#. Copy the wrapper scripts from the ``ush`` directory into the experiment directory. Each workflow task has a wrapper script that sets environment variables and runs the job script. - - .. code-block:: console - - cp /ufs-srweather-app/ush/wrappers/* . - -#. Set the ``OMP_NUM_THREADS`` variable. - - .. code-block:: console - - export OMP_NUM_THREADS=1 - -#. Run each of the listed scripts in order. Scripts with the same stage number (listed in :numref:`Table %s `) may be run simultaneously. - - .. code-block:: console - - ./run_make_grid.sh - ./run_get_ics.sh - ./run_get_lbcs.sh - ./run_make_orog.sh - ./run_make_sfc_climo.sh - ./run_make_ics.sh - ./run_make_lbcs.sh - ./run_fcst.sh - ./run_post.sh - -Each task should finish with error code 0. For example: - -.. code-block:: console - - End exregional_get_extrn_mdl_files.sh at Wed Nov 16 18:08:19 UTC 2022 with error code 0 (time elapsed: 00:00:01) - -Check the batch script output file in your experiment directory for a “SUCCESS” message near the end of the file. - -.. _RegionalWflowTasks: - -.. table:: List of tasks in the regional workflow in the order that they are executed. - Scripts with the same stage number may be run simultaneously. The number of - processors and wall clock time is a good starting point for Cheyenne or Hera - when running a 48-h forecast on the 25-km CONUS domain. For a brief description of tasks, see :numref:`Table %s `. - - +------------+------------------------+----------------+----------------------------+ - | **Stage/** | **Task Run Script** | **Number of** | **Wall Clock Time (H:mm)** | - | | | **Processors** | | - +============+========================+================+============================+ - | 1 | run_get_ics.sh | 1 | 0:20 (depends on HPSS vs | - | | | | FTP vs staged-on-disk) | - +------------+------------------------+----------------+----------------------------+ - | 1 | run_get_lbcs.sh | 1 | 0:20 (depends on HPSS vs | - | | | | FTP vs staged-on-disk) | - +------------+------------------------+----------------+----------------------------+ - | 1 | run_make_grid.sh | 24 | 0:20 | - +------------+------------------------+----------------+----------------------------+ - | 2 | run_make_orog.sh | 24 | 0:20 | - +------------+------------------------+----------------+----------------------------+ - | 3 | run_make_sfc_climo.sh | 48 | 0:20 | - +------------+------------------------+----------------+----------------------------+ - | 4 | run_make_ics.sh | 48 | 0:30 | - +------------+------------------------+----------------+----------------------------+ - | 4 | run_make_lbcs.sh | 48 | 0:30 | - +------------+------------------------+----------------+----------------------------+ - | 5 | run_fcst.sh | 48 | 0:30 | - +------------+------------------------+----------------+----------------------------+ - | 6 | run_post.sh | 48 | 0:25 (2 min per output | - | | | | forecast hour) | - +------------+------------------------+----------------+----------------------------+ - -Users can access log files for specific tasks in the ``$EXPTDIR/log`` directory. To see how the experiment is progressing, users can also check the end of the ``log.launch_FV3LAM_wflow`` file from the command line: - -.. code-block:: console - - tail -n 40 log.launch_FV3LAM_wflow - -.. hint:: - If any of the scripts return an error that "Primary job terminated normally, but one process returned a non-zero exit code," there may not be enough space on one node to run the process. On an HPC system, the user will need to allocate a(nother) compute node. The process for doing so is system-dependent, and users should check the documentation available for their HPC system. Instructions for allocating a compute node on NOAA HPC systems can be viewed in :numref:`Section %s ` as an example. - -.. note:: - On most HPC systems, users will need to submit a batch job to run multi-processor jobs. On some HPC systems, users may be able to run the first two jobs (serial) on a login node/command-line. Example scripts for Slurm (Hera) and PBS (Cheyenne) resource managers are provided (``sq_job.sh`` and ``qsub_job.sh``, respectively). These examples will need to be adapted to each user's system. Alternatively, some batch systems allow users to specify most of the settings on the command line (with the ``sbatch`` or ``qsub`` command, for example). diff --git a/docs/UsersGuide/source/_static/JenkinsCICD.png b/docs/UsersGuide/source/_static/JenkinsCICD.png deleted file mode 100644 index 48c30bde1a..0000000000 Binary files a/docs/UsersGuide/source/_static/JenkinsCICD.png and /dev/null differ diff --git a/docs/UsersGuide/source/_static/RRFS_CONUS_13km.sphr.native_wrtcmp.png b/docs/UsersGuide/source/_static/RRFS_CONUS_13km.sphr.native_wrtcmp.png deleted file mode 100644 index 2d1ed25b68..0000000000 Binary files a/docs/UsersGuide/source/_static/RRFS_CONUS_13km.sphr.native_wrtcmp.png and /dev/null differ diff --git a/docs/UsersGuide/source/_static/RRFS_CONUS_25km.sphr.native_wrtcmp.png b/docs/UsersGuide/source/_static/RRFS_CONUS_25km.sphr.native_wrtcmp.png deleted file mode 100644 index bc321d6d74..0000000000 Binary files a/docs/UsersGuide/source/_static/RRFS_CONUS_25km.sphr.native_wrtcmp.png and /dev/null differ diff --git a/docs/UsersGuide/source/_static/RRFS_CONUS_3km.sphr.native_wrtcmp.png b/docs/UsersGuide/source/_static/RRFS_CONUS_3km.sphr.native_wrtcmp.png deleted file mode 100644 index 0d4b87a4be..0000000000 Binary files a/docs/UsersGuide/source/_static/RRFS_CONUS_3km.sphr.native_wrtcmp.png and /dev/null differ diff --git a/docs/UsersGuide/source/_static/SRW_build_process.png b/docs/UsersGuide/source/_static/SRW_build_process.png deleted file mode 100644 index e36ffce706..0000000000 Binary files a/docs/UsersGuide/source/_static/SRW_build_process.png and /dev/null differ diff --git a/docs/UsersGuide/source/_static/SRW_regional_workflow_gen.png b/docs/UsersGuide/source/_static/SRW_regional_workflow_gen.png deleted file mode 100644 index 6f085589ed..0000000000 Binary files a/docs/UsersGuide/source/_static/SRW_regional_workflow_gen.png and /dev/null differ diff --git a/docs/UsersGuide/source/_static/SRW_run_process.png b/docs/UsersGuide/source/_static/SRW_run_process.png deleted file mode 100644 index ee0ff7ec76..0000000000 Binary files a/docs/UsersGuide/source/_static/SRW_run_process.png and /dev/null differ diff --git a/docs/UsersGuide/source/_static/SRW_wflow_flowchart.png b/docs/UsersGuide/source/_static/SRW_wflow_flowchart.png deleted file mode 100644 index 0cee4f5e65..0000000000 Binary files a/docs/UsersGuide/source/_static/SRW_wflow_flowchart.png and /dev/null differ diff --git a/docs/UsersGuide/source/_static/SRW_wflow_input_path.png b/docs/UsersGuide/source/_static/SRW_wflow_input_path.png deleted file mode 100644 index 7b2b8cd494..0000000000 Binary files a/docs/UsersGuide/source/_static/SRW_wflow_input_path.png and /dev/null differ diff --git a/docs/UsersGuide/source/_static/SUBCONUS_Ind_3km.png b/docs/UsersGuide/source/_static/SUBCONUS_Ind_3km.png deleted file mode 100644 index 8bbcbc7aa3..0000000000 Binary files a/docs/UsersGuide/source/_static/SUBCONUS_Ind_3km.png and /dev/null differ diff --git a/docs/UsersGuide/source/conf.py b/docs/UsersGuide/source/conf.py index 1490321e96..a6023d339d 100644 --- a/docs/UsersGuide/source/conf.py +++ b/docs/UsersGuide/source/conf.py @@ -211,6 +211,7 @@ def setup(app): 'hpc-stack': ('https://hpc-stack-epic.readthedocs.io/en/latest/', None), 'met': ('https://met.readthedocs.io/en/latest/', None), 'srw_v2.1.0': ('https://ufs-srweather-app.readthedocs.io/en/release-public-v2.1.0/', None), + 'ufs-wm': ('https://ufs-weather-model.readthedocs.io/en/latest/', None), } # -- Options for todo extension ---------------------------------------------- diff --git a/docs/UsersGuide/source/index.rst b/docs/UsersGuide/source/index.rst index 1a51b34552..52937c81ae 100644 --- a/docs/UsersGuide/source/index.rst +++ b/docs/UsersGuide/source/index.rst @@ -11,21 +11,7 @@ UFS Short-Range Weather App Users Guide :maxdepth: 3 - Introduction - Quickstart - ContainerQuickstart - BuildSRW - RunSRW - Tutorial - AQM - Components - InputOutputFiles - LAMGrids - ConfigWorkflow - RocotoInfo - DefineWorkflow - WE2Etests - TemplateVars - VXCases - FAQ - Glossary + BackgroundInfo/index + BuildingRunningTesting/index + CustomizingTheWorkflow/index + Reference/index diff --git a/docs/UsersGuide/source/references.bib b/docs/UsersGuide/source/references.bib index 0845ca3fa7..196233b372 100644 --- a/docs/UsersGuide/source/references.bib +++ b/docs/UsersGuide/source/references.bib @@ -12,3 +12,13 @@ @misc{Purser_2020 year = {2020}, howpublished = {Unified Forecast System (UFS) Users’ Workshop}, } + +@article{LinEtAl2021, + title={Harmonized Emissions Component (HEMCO) 3.0 as a versatile emissions component for atmospheric models: Application in the GEOS-Chem, NASA GEOS, WRF-GC, CESM2, NOAA GEFS-Aerosol, and NOAA UFS models}, + author={H. Lin and D.J. Jacob and E.W. Lundgren and M.P. Sulprizio and C.A. Keller and T.M. Fritz and S.D. Eastham and L.K. Emmons and P.C. Campbell and B. Baker and R.D. Saylor, and R. Montuoro}, + journal={Geosci. Model Dev.}, + year={2021}, + volume={14}, + pages={5487–5506}, + doi={10.5194/gmd-14-5487-2021}, + } \ No newline at end of file diff --git a/docs/UsersGuide/source/tables/Tests.csv b/docs/UsersGuide/source/tables/Tests.csv index dce4a03fc0..94ca55bb03 100644 --- a/docs/UsersGuide/source/tables/Tests.csv +++ b/docs/UsersGuide/source/tables/Tests.csv @@ -2,30 +2,33 @@ yes,yes,grid_RRFS_CONUScompact_25km_ics_HRRR_lbcs_RAP_suite_RRFS_v1beta,RRFS_CONUScompact_25km,FV3_RRFS_v1beta,HRRR,RAP,2020081000,3,8,22, yes,yes,nco_grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_timeoffset_suite_GFS_v16,RRFS_CONUS_25km,FV3_GFS_v16,FV3GFS,FV3GFS,2022081012,6,10,15, yes,yes,grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v15p2,RRFS_CONUS_25km,FV3_GFS_v15p2,FV3GFS,FV3GFS,2019070100,6,7,10, -yes,yes,grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v17_p8,RRFS_CONUS_25km,FV3_GFS_v17_p8,FV3GFS,FV3GFS,2019070100,6,11,15, +yes,yes,grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v17_p8_plot,RRFS_CONUS_25km,FV3_GFS_v17_p8,FV3GFS,FV3GFS,2019070100,6,11,15, yes,yes,grid_RRFS_CONUScompact_25km_ics_HRRR_lbcs_HRRR_suite_HRRR,RRFS_CONUScompact_25km,FV3_HRRR,HRRR,HRRR,2020081000,24,26,20 yes,yes,grid_SUBCONUS_Ind_3km_ics_HRRR_lbcs_RAP_suite_WoFS_v0,SUBCONUS_Ind_3km,FV3_WoFS_v0,HRRR,RAP,2020081000,6,13,20, yes,yes,grid_RRFS_CONUS_25km_ics_NAM_lbcs_NAM_suite_GFS_v16,RRFS_CONUS_25km,FV3_GFS_v16,NAM,NAM,2021051212,6,17,25 ,yes,community,RRFS_CONUS_25km,FV3_GFS_v16,FV3GFS,FV3GFS,2019061518,12,17,, -,yes,custom_ESGgrid,*custom*,FV3_GFS_2017_gfdlmp_regional,FV3GFS,FV3GFS,2019070100,6,11,, -,yes,custom_GFDLgrid,*custom*,FV3_GFS_2017_gfdlmp,FV3GFS,FV3GFS,2019070100,6,10,19, -,yes,custom_GFDLgrid__GFDLgrid_USE_NUM_CELLS_IN_FILENAMES_eq_FALSE,*custom*,FV3_GFS_2017_gfdlmp,FV3GFS,FV3GFS,2019070100,6,11,, +,yes,custom_ESGgrid,*custom*,FV3_HRRR,FV3GFS,FV3GFS,2019070100,6,11,, +,yes,custom_ESGgrid_Central_Asia_3km,*custom*,FV3_GFS_v15_thompson_mynn_lam3km,FV3GFS,FV3GFS,2019070100,6,30,, +,yes,custom_ESGgrid_IndianOcean_6km,*custom*,FV3_RRFS_v1beta,FV3GFS,FV3GFS,2019061518,12,15,, +,yes,custom_ESGgrid_NewZealand_3km,*custom*,FV3_HRRR,FV3GFS,FV3GFS,2019061518,6,58,, +,yes,custom_ESGgrid_Peru_12km,*custom*,FV3_RAP,FV3GFS,FV3GFS,2019061500,12,18,, +,yes,custom_ESGgrid_SF_1p1km,*custom*,FV3_WoFS_v0,FV3GFS,FV3GFS,2019061500,6,173,, +,yes,custom_GFDLgrid,*custom*,FV3_GFS_v15p2,FV3GFS,FV3GFS,2019070100,6,10,19, +,yes,custom_GFDLgrid__GFDLgrid_USE_NUM_CELLS_IN_FILENAMES_eq_FALSE,*custom*,FV3_GFS_v15p2,FV3GFS,FV3GFS,2019070100,6,11,, ,yes,deactivate_tasks,RRFS_CONUS_25km,FV3_GFS_v15p2,*pregen*,*pregen*,2019070100,6,1,19, ,yes,grid_CONUS_25km_GFDLgrid_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16,CONUS_25km_GFDLgrid,FV3_GFS_v16,FV3GFS,FV3GFS,2019070100,6,8,20, ,yes,grid_CONUS_3km_GFDLgrid_ics_FV3GFS_lbcs_FV3GFS_suite_RRFS_v1beta,CONUS_3km_GFDLgrid,FV3_RRFS_v1beta,FV3GFS,FV3GFS,2019070100,3,260,38, -,yes,grid_RRFS_AK_13km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16,RRFS_AK_13km,FV3_GFS_v16,FV3GFS,FV3GFS,2019070100,6,135,45, +,yes,grid_RRFS_AK_13km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16_plot,RRFS_AK_13km,FV3_GFS_v16,FV3GFS,FV3GFS,2019070100,6,135,45, ,yes,grid_RRFS_AK_3km_ics_FV3GFS_lbcs_FV3GFS_suite_HRRR,RRFS_AK_3km,FV3_HRRR,FV3GFS,FV3GFS,2019070100,3,450,65, -,yes,grid_RRFS_CONUS_13km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v15p2,RRFS_CONUS_13km,FV3_GFS_v15p2,FV3GFS,FV3GFS,2019070100,6,13,17, -,yes,grid_RRFS_CONUS_13km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16,RRFS_CONUS_13km,FV3_GFS_v16,FV3GFS,FV3GFS,2019070100,6,27,18, +,yes,grid_RRFS_CONUS_13km_ics_FV3GFS_lbcs_FV3GFS_suite_RAP,RRFS_CONUS_13km,FV3_RAP,FV3GFS,FV3GFS,2019070100,6,13,17, +,yes,grid_RRFS_CONUS_13km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16_plot,RRFS_CONUS_13km,FV3_GFS_v16,FV3GFS,FV3GFS,2019070100,6,27,18, ,yes,grid_RRFS_CONUS_13km_ics_FV3GFS_lbcs_FV3GFS_suite_HRRR,RRFS_CONUS_13km,FV3_HRRR,FV3GFS,FV3GFS,2019070100,6,27,20, ,yes,grid_RRFS_CONUS_13km_ics_FV3GFS_lbcs_FV3GFS_suite_RRFS_v1beta,RRFS_CONUS_13km,FV3_RRFS_v1beta,FV3GFS,FV3GFS,2019070100,6,27,20, -,yes,grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_2017_gfdlmp_regional_plot,RRFS_CONUS_25km,FV3_GFS_2017_gfdlmp_regional,FV3GFS,FV3GFS,2019070100,6,7,20, ,yes,grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_2017_gfdlmp,RRFS_CONUS_25km,FV3_GFS_2017_gfdlmp,FV3GFS,FV3GFS,2019070112 2019070200,6,39,24, ,yes,grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16,RRFS_CONUS_25km,FV3_GFS_v16,FV3GFS,FV3GFS,2019070100,6,5,23, -,yes,grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_HRRR,RRFS_CONUS_25km,FV3_HRRR,FV3GFS,FV3GFS,2019070100,6,10,37, +,yes,grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_HRRR,RRFS_CONUS_25km,FV3_HRRR,FV3GFS,FV3GFS,2019070100,6,15,30, ,yes,grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_RRFS_v1beta,RRFS_CONUS_25km,FV3_RRFS_v1beta,FV3GFS,FV3GFS,2019070100,6,12,34, -,yes,grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_RAP_suite_HRRR,RRFS_CONUS_25km,FV3_HRRR,FV3GFS,RAP,2019061518,6,18,45, -,yes,grid_RRFS_CONUS_25km_ics_GSMGFS_lbcs_GSMGFS_suite_GFS_2017_gfdlmp,RRFS_CONUS_25km,FV3_GFS_2017_gfdlmp,GSMGFS,GSMGFS,2019052000,6,9,23, +,yes,grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_RAP_suite_RAP,RRFS_CONUS_25km,FV3_RAP,FV3GFS,RAP,2019061518,6,18,45, ,yes,grid_RRFS_CONUS_25km_ics_GSMGFS_lbcs_GSMGFS_suite_GFS_v15p2,RRFS_CONUS_25km,FV3_GFS_v15p2,GSMGFS,GSMGFS,2019052000,6,7,19, ,yes,grid_RRFS_CONUS_25km_ics_NAM_lbcs_NAM_suite_RRFS_v1beta,RRFS_CONUS_25km,FV3_RRFS_v1beta,NAM,NAM,2021061500,6,15,31, ,yes,grid_RRFS_CONUS_3km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v15p2,RRFS_CONUS_3km,FV3_GFS_v15p2,FV3GFS,FV3GFS,2019070100,6,250,35, @@ -38,15 +41,16 @@ yes,yes,grid_RRFS_CONUS_25km_ics_NAM_lbcs_NAM_suite_GFS_v16,RRFS_CONUS_25km,FV3_ ,yes,grid_RRFS_CONUScompact_13km_ics_HRRR_lbcs_RAP_suite_RRFS_v1beta,RRFS_CONUScompact_13km,FV3_RRFS_v1beta,HRRR,RAP,2020080100,6,25,20, ,yes,grid_RRFS_CONUScompact_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16,RRFS_CONUScompact_25km,FV3_GFS_v16,FV3GFS,FV3GFS,2019070100,6,10,22, ,yes,grid_RRFS_CONUScompact_3km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16,RRFS_CONUScompact_3km,FV3_GFS_v16,FV3GFS,FV3GFS,2019070100,3,285,51, -,yes,grid_RRFS_CONUScompact_3km_ics_HRRR_lbcs_RAP_suite_HRRR,RRFS_CONUScompact_3km,FV3_HRRR,HRRR,RAP,2020081000,6,340,1:00, +,yes,grid_RRFS_CONUScompact_3km_ics_HRRR_lbcs_RAP_suite_HRRR,RRFS_CONUScompact_3km,FV3_HRRR,HRRR,RAP,2020081000,6,340,60, ,yes,grid_RRFS_CONUScompact_3km_ics_HRRR_lbcs_RAP_suite_RRFS_v1beta,RRFS_CONUScompact_3km,FV3_RRFS_v1beta,HRRR,RAP,2020080100,6,335,53, -,yes,grid_RRFS_NA_13km_ics_FV3GFS_lbcs_FV3GFS_suite_RRFS_v1beta,RRFS_NA_13km,FV3_RRFS_v1beta,FV3GFS,FV3GFS,2019070100,6,100,36, +,yes,grid_RRFS_NA_13km_ics_FV3GFS_lbcs_FV3GFS_suite_FV3_RAP,RRFS_NA_13km,FV3_RAP,FV3GFS,FV3GFS,2019070100,6,100,36, ,yes,grid_SUBCONUS_Ind_3km_ics_FV3GFS_lbcs_FV3GFS_suite_WoFS_v0,SUBCONUS_Ind_3km,FV3_WoFS_v0,FV3GFS,FV3GFS,2019061500,6,14,23, ,yes,grid_SUBCONUS_Ind_3km_ics_HRRR_lbcs_HRRR_suite_HRRR,SUBCONUS_Ind_3km,FV3_HRRR,HRRR,HRRR,2020081000,3,17,23, ,yes,grid_SUBCONUS_Ind_3km_ics_NAM_lbcs_NAM_suite_GFS_v16,SUBCONUS_Ind_3km,FV3_GFS_v16,NAM,NAM,2021061500,6,13,41, ,yes,grid_SUBCONUS_Ind_3km_ics_RAP_lbcs_RAP_suite_RRFS_v1beta,SUBCONUS_Ind_3km,FV3_RRFS_v1beta,RAP,RAP,2020080103,3,20,22, ,yes,GST_release_public_v1,RRFS_CONUS_25km,FV3_GFS_v15p2,FV3GFS,FV3GFS,2019061500,48,31,35, -,yes,MET_verification_only_vx,RRFS_CONUS_25km,*none*,*none*,*none*,2019061500,6,1,08,Runs verification tasks on staged data without running the rest of the workflow +,yes,MET_ensemble_verification_only_vx,RRFS_CONUS_25km,*none*,*none*,*none*,2019061500,6,1,15,Runs ensemble verification tasks on staged data without running the rest of the workflow +,yes,MET_verification_only_vx,RRFS_CONUS_25km,*none*,*none*,*none*,2019061500,6,1,8,Runs verification tasks on staged data without running the rest of the workflow ,yes,nco,RRFS_CONUS_25km,FV3_GFS_v16,FV3GFS,FV3GFS,2022040700,6,7,20, ,yes,nco_ensemble,RRFS_CONUS_25km,FV3_GFS_v15p2,FV3GFS,FV3GFS,2019070100 2019070112 2019070200 2019070212,6,55,20, ,yes,nco_grid_RRFS_CONUS_13km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16,RRFS_CONUS_13km,FV3_GFS_v16,FV3GFS,FV3GFS,2019061500,6,26,20, @@ -55,16 +59,20 @@ yes,yes,grid_RRFS_CONUS_25km_ics_NAM_lbcs_NAM_suite_GFS_v16,RRFS_CONUS_25km,FV3_ ,yes,pregen_grid_orog_sfc_climo,RRFS_CONUS_25km,FV3_GFS_v15p2,FV3GFS,FV3GFS,2019070100,6,6,17, ,yes,specify_EXTRN_MDL_SYSBASEDIR_ICS_LBCS,RRFS_CONUS_25km,FV3_GFS_v15p2,FV3GFS,FV3GFS,2021061500,6,6,19, ,yes,specify_template_filenames,RRFS_CONUS_25km,FV3_GFS_v15p2,FV3GFS,FV3GFS,2019070100,6,6,28, -,hera/jet/orion,get_from_AWS_ics_GEFS_lbcs_GEFS_fmt_grib2_2022040400_ensemble_2mems,RRFS_CONUS_3km,FV3_HRRR,GEFS,GEFS,2022040400,6,805,45, -,hera/jet/orion,get_from_NOMADS_ics_FV3GFS_lbcs_FV3GFS,RRFS_CONUS_25km,FV3_GFS_2017_gfdlmp,FV3GFS,FV3GFS,*2 days ago*,6,12,40,Retrieves online data and runs a forecast valid 00z 2 days previous to the test date +,hera/jet/orion/gaea,get_from_AWS_ics_GEFS_lbcs_GEFS_fmt_grib2_2022040400_ensemble_2mems,RRFS_CONUS_3km,FV3_HRRR,GEFS,GEFS,2022040400,6,805,45, +,hera/jet/orion/gaea,get_from_NOMADS_ics_FV3GFS_lbcs_FV3GFS,RRFS_CONUS_25km,FV3_GFS_v15p2,FV3GFS,FV3GFS,*2 days ago*,6,12,40,Retrieves online data and runs a forecast valid 00z 2 days previous to the test date +,hera/jet,custom_ESGgrid_Great_Lakes_snow_8km,*custom*,FV3_RAP,RAP,RAP,2023021700,6,,, ,hera/jet,get_from_HPSS_ics_FV3GFS_lbcs_FV3GFS_fmt_grib2_2019061200,RRFS_CONUS_25km,FV3_GFS_v15p2,FV3GFS,FV3GFS,2019061200,6,6,19, ,hera/jet,get_from_HPSS_ics_FV3GFS_lbcs_FV3GFS_fmt_nemsio_2019061200,RRFS_CONUS_25km,FV3_GFS_v15p2,FV3GFS,FV3GFS,2019061200,6,10,23, ,hera/jet,get_from_HPSS_ics_FV3GFS_lbcs_FV3GFS_fmt_nemsio_2021032018,RRFS_CONUS_25km,FV3_GFS_v15p2,FV3GFS,FV3GFS,2021032018,6,10,23, ,hera/jet,get_from_HPSS_ics_FV3GFS_lbcs_FV3GFS_fmt_netcdf_2022060112_48h,RRFS_CONUS_25km,FV3_GFS_v16,FV3GFS,FV3GFS,2022060112,48,55,43, ,hera/jet,get_from_HPSS_ics_GDAS_lbcs_GDAS_fmt_netcdf_2022040400_ensemble_2mems,RRFS_CONUS_3km,FV3_HRRR,GDAS,GDAS,2022040400,6,765,48, -,hera/jet,get_from_HPSS_ics_GSMGFS_lbcs_GSMGFS,RRFS_CONUS_25km,FV3_GFS_2017_gfdlmp,GSMGFS,GSMGFS,2019052000,6,10,25, +,hera/jet,get_from_HPSS_ics_GSMGFS_lbcs_GSMGFS,RRFS_CONUS_25km,FV3_GFS_v15p2,GSMGFS,GSMGFS,2019052000,6,10,25, ,hera/jet,get_from_HPSS_ics_HRRR_lbcs_RAP,RRFS_CONUScompact_25km,FV3_HRRR,HRRR,RAP,2020080100,6,14,30, ,hera/jet,get_from_HPSS_ics_RAP_lbcs_RAP,RRFS_CONUS_25km,FV3_HRRR,RAP,RAP,2019052000,6,17,35, +,hera/jet,long_fcst,RRFS_CONUScompact_25km,FV3_RAP,FV3GFS,FV3GFS,2023060112,108,50,, +,hera/jet,MET_ensemble_verification_only_vx_time_lag,RRFS_CONUS_3km,*none*,*none*,*none*,2021050500,6,3,30, +,,aqm_grid_AQM_NA13km_suite_GFS_v16,AQM_NA_13km,FV3_GFS_v16,FV3GFS,FV3GFS,2023021700 2023021706,6,,,This is an air-quality model test that requires special compilation to run; not supported in this release ,,grid_RRFS_NA_3km_ics_FV3GFS_lbcs_FV3GFS_suite_RRFS_v1beta,RRFS_NA_3km,FV3_RRFS_v1beta,FV3GFS,FV3GFS,2019070100,3,,,The RRFS_NA_3km domain currently has segfault problems--this test is not run ,,subhourly_post,RRFS_CONUScompact_25km,FV3_RRFS_v1beta,HRRR,RAP,2020081000,3,,,Subhourly post tasks are currently broken--these tests are not run ,,subhourly_post_ensemble_2mems,RRFS_CONUScompact_25km,FV3_RRFS_v1beta,HRRR,RAP,2020081000,3,,,Subhourly post tasks are currently broken--these tests are not run