From 2012e8dc3db7c1298989674c9d218a1063b38832 Mon Sep 17 00:00:00 2001 From: gspetro Date: Thu, 22 Dec 2022 14:05:14 -0500 Subject: [PATCH 01/17] update ConfigWorkflow.rst w/plotting info --- docs/UsersGuide/source/ConfigWorkflow.rst | 47 +++++++++++++++++++++++ 1 file changed, 47 insertions(+) diff --git a/docs/UsersGuide/source/ConfigWorkflow.rst b/docs/UsersGuide/source/ConfigWorkflow.rst index ecde1be5d3..1223524451 100644 --- a/docs/UsersGuide/source/ConfigWorkflow.rst +++ b/docs/UsersGuide/source/ConfigWorkflow.rst @@ -490,6 +490,9 @@ Verification Tasks .. COMMENT: COMMENT: Define "ensemble-stat verification for gridded data," "ensemble point verification," "ensemble-stat point verification," and "point verification of ensemble-stat output"? +``RUN_TASK_PLOT_ALLVARS:`` (Default: false) + Flag that determines whether to run python plotting scripts. + .. _make-grid: MAKE_GRID Configuration Parameters @@ -1134,6 +1137,9 @@ These parameters are associated with the fixed (i.e., static) files. On `Level 1 ``FIXlut``: (Default: "") System directory where the lookup tables for optics properties are located. +``FIXshp``: (Default: "") + System directory where the graphics shapefiles are located. + ``TOPO_DIR``: (Default: "") The location on disk of the static input files used by the ``make_orog`` task (i.e., ``orog.x`` and ``shave.x``). Can be the same as ``FIXgsm``. @@ -1726,6 +1732,47 @@ Non-default parameters for the ``run_enspointvx_prob`` task are set in the ``tas ``MAXTRIES_VX_ENSPOINT_PROB``: (Default: 1) Maximum number of times to attempt the task. +PLOT_ALLVARS Configuration Parameters +======================================== + +Non-default parameters for the ``plot_allvars`` task are set in the ``task_plot_allvars:`` section of the ``config.yaml`` file. + +Basic Task Parameters +-------------------------- + +For each workflow task, certain parameter values must be passed to the job scheduler (e.g., Slurm), which submits a job for the task. Typically, users do not need to adjust the default values. + +``PLOT_ALLVARS_TN``: (Default: "plot_allvars") + Set the name of this Rocoto workflow task. Users typically do not need to change this value. + +``NNODES_PLOT_ALLVARS``: (Default: 1) + Number of nodes to use for the job. + +``PPN_PLOT_ALLVARS``: (Default: 24) + Number of :term:`MPI` processes per node. + +``WTIME_PLOT_ALLVARS``: (Default: 01:00:00) + Maximum time for the task to complete. + +``MAXTRIES_PLOT_ALLVARS``: (Default: 1) + Maximum number of times to attempt the task. + +Additional Parameters +------------------------ + +Typially, the following parameters must be set explicitly by the user in the configuration file (``config.yaml``) when executing the plotting tasks. + +``COMOUT_REF``: (Default: "") + The directory where the GRIB2 files from post-processing are located. In *community* mode (i.e., when ``RUN_ENVIR: "community"``), this directory will correspond to the location in the experiment directory where the post-processed output can be found (e.g., ``$EXPTDIR/$DATE_FIRST_CYCL/postprd``). In *nco* mode, this directory should be set to the location of the COMOUT directory and end with ``$PDY/$cyc``. + +``PLOT_FCST_START``: (Default: 0) + The starting forecast hour for the plotting task. For example, if a forecast starts at 18h/18z, this is considered the 0th forecast hour, so "starting forecast hour" should be 0, not 18. If a forecast starts at 18h/18z, but the user only wants plots from the 6th forecast hour on, "starting forecast hour" should be 6. + +``PLOT_FCST_INC``: (Default: 3) + Forecast hour increment for the plotting task. This may be the same as ``INCR_CYCL_FREQ``, or it may be a multiple of ``INCR_CYCL_FREQ``. For example, if ``INCR_CYCL_FREQ`` is set to 3, there will be forecast output every three hours for the duration of the forecast. If the user wants plots of all of this output, they should set ``PLOT_FCST_INC: 3``. If the user only wants plots for some of the output (e.g., every 6 hours), they should set ``PLOT_FCST_INC: 6``. However, there must be forecast output available at the designated increments to produce the plots. In this example, setting ``PLOT_FCST_INC: 7`` would produce an error because there is only forecast output available for hours 3, 6, 9, ..., etc. + +``PLOT_FCST_END``: (Default: "") + The last forecast hour for the plotting task. For example, if a forecast run for 24 hours, and the user wants plots for each available hour of forecast output, they should set ``PLOT_FCST_END: 24``. If the user only wants plots from the first 12 hours of the forecast, the "last forecast hour" should be 12. Global Configuration Parameters =================================== From 8d5c7c45086f8611ff6f6d2c732c2d7fd1c19c9c Mon Sep 17 00:00:00 2001 From: gspetro Date: Thu, 22 Dec 2022 17:42:00 -0500 Subject: [PATCH 02/17] add plotting info to RunSRW --- docs/UsersGuide/source/ConfigWorkflow.rst | 7 +++- docs/UsersGuide/source/RunSRW.rst | 45 ++++++++++++++++++++++- 2 files changed, 50 insertions(+), 2 deletions(-) diff --git a/docs/UsersGuide/source/ConfigWorkflow.rst b/docs/UsersGuide/source/ConfigWorkflow.rst index 1223524451..709a54e063 100644 --- a/docs/UsersGuide/source/ConfigWorkflow.rst +++ b/docs/UsersGuide/source/ConfigWorkflow.rst @@ -490,6 +490,9 @@ Verification Tasks .. COMMENT: COMMENT: Define "ensemble-stat verification for gridded data," "ensemble point verification," "ensemble-stat point verification," and "point verification of ensemble-stat output"? +Plotting Task +---------------- + ``RUN_TASK_PLOT_ALLVARS:`` (Default: false) Flag that determines whether to run python plotting scripts. @@ -1138,7 +1141,7 @@ These parameters are associated with the fixed (i.e., static) files. On `Level 1 System directory where the lookup tables for optics properties are located. ``FIXshp``: (Default: "") - System directory where the graphics shapefiles are located. + System directory where the graphics shapefiles are located. On Level 1 systems, these are set within the machine files. Users on other systems will need to provide the path to the directory that contains the *Natural Earth* shapfiles. ``TOPO_DIR``: (Default: "") The location on disk of the static input files used by the ``make_orog`` task (i.e., ``orog.x`` and ``shave.x``). Can be the same as ``FIXgsm``. @@ -1732,6 +1735,8 @@ Non-default parameters for the ``run_enspointvx_prob`` task are set in the ``tas ``MAXTRIES_VX_ENSPOINT_PROB``: (Default: 1) Maximum number of times to attempt the task. +.. _PlotVars: + PLOT_ALLVARS Configuration Parameters ======================================== diff --git a/docs/UsersGuide/source/RunSRW.rst b/docs/UsersGuide/source/RunSRW.rst index 58eaa4e2f7..838f49ff23 100644 --- a/docs/UsersGuide/source/RunSRW.rst +++ b/docs/UsersGuide/source/RunSRW.rst @@ -1267,4 +1267,47 @@ Users can access log files for specific tasks in the ``$EXPTDIR/log`` directory. Plot the Output =============== -Two python scripts are provided to generate plots from the :term:`FV3`-LAM post-processed :term:`GRIB2` output. Information on how to generate the graphics can be found in :numref:`Chapter %s `. + +.. Add this section above to the Config section before the METplus info! + +An optional Python plotting task can be activated in the workflow to generate plots for the :term:`FV3`-:term:`LAM` post-processed :term:`GRIB2` +output over the :term:`CONUS`. It generates plots for a number of variables, including: + +* 2-m temperature +* 2-m dew point temperature +* 10-m winds +* 500 hPa heights, winds, and vorticity +* 250 hPa winds +* Accumulated precipitation +* Composite reflectivity +* Surface-based :term:`CAPE`/:term:`CIN` +* Max/Min 2-5 km updraft helicity +* Sea level pressure (SLP) + +This workflow task can produce both plots from a single experiment and difference plots that compare the same cycle from two different experiments. When plotting the difference, the two experiments must be on the same domain and available for +the same cycle starting date/time and forecast hours. Other parameters may differ (e.g., the experiments may use different physics suites). + +.. _Cartopy: + +Cartopy Shapefiles +--------------------- + +The Python plotting tasks require a path to the directory where the Cartopy Natural Earth shapefiles are located. The medium scale (1:50m) cultural and physical shapefiles are used to create coastlines and other geopolitical borders on the map. On `Level 1 `__ systems, this path is already set in the machine file using the variable ``FIXshp``. Users on other systems will need to download the shapefiles and update the path of ``$FIXshp`` in the machine file they are using (e.g., ``$HOME/ush/machine/macos.yaml`` for a generic MacOS system). The subset of shapefiles required for the plotting task can be obtained from the `SRW Data Bucket `__. The full set of Cartopy shapefiles can be downloaded `here `__. + +Additionally, users may need to add or modify certain variables in ``config.yaml``. For example: + +.. code-block:: console + + workflow_switches: + RUN_TASK_PLOT_ALLVARS: true + task_plot_allvars: + COMOUT_REF: + PLOT_FCST_START: 0 + PLOT_FCST_INC: 6 + PLOT_FCST_END: 12 + +where ``$EXPTDIR`` is the path to the experiment directory, and ``$CDATE`` is the cycle date. In *community* mode, using default directory names, ``$COMOUT_REF`` will resemble ``/path/to/expt_dirs/test_community/2019061518/postprd``. Additional details on the plotting variables is provided in :numref:`Section %s `. + + +.. NOTES: + plot_allvars and plot_allvars_diff are done within the same task. The user would need to provide the baseline experiment directory EXPTDIR_REF if diff plots are needed. \ No newline at end of file From 4c6eaeabf13c44a05357c0b56d151245f5dfc11f Mon Sep 17 00:00:00 2001 From: gspetro Date: Wed, 4 Jan 2023 10:44:04 -0500 Subject: [PATCH 03/17] add Template Vars chapter --- docs/UsersGuide/source/TemplateVars.rst | 72 +++++++++++++++++++++++++ docs/UsersGuide/source/index.rst | 1 + 2 files changed, 73 insertions(+) create mode 100644 docs/UsersGuide/source/TemplateVars.rst diff --git a/docs/UsersGuide/source/TemplateVars.rst b/docs/UsersGuide/source/TemplateVars.rst new file mode 100644 index 0000000000..8950de80c8 --- /dev/null +++ b/docs/UsersGuide/source/TemplateVars.rst @@ -0,0 +1,72 @@ +.. _TemplateVars: + +=============================================================== +Using Template Variables in the Experiment Configuration Files +=============================================================== + +The SRW App's experiment configuration system supports the use of template variables +in ``config_defaults.yaml`` and ``config.yaml``. A template variable --- or "template" --- is an experiment configuration variable that contains references to values of other variables. +These references are **not** set to the values of the referenced variables (or "expanded") when the experiment's variable definitions file (``var_defns.sh``) is generated or sourced. +Instead, they are expanded and evaluated **at run time** when bash's +``eval`` command is used on the template. + +Generic Example +================== + +As an example, consider a hypothetical template variable named ``MY_CMD`` that is defined in ``config_defaults.yaml`` +(or redefined by the user in ``config.yaml``) as follows: + + .. code-block:: console + + MY_CMD: 'cd ${some_dir}' + +Here, ``some_dir`` may be another experiment variable defined in ``var_defns.sh`` or a +local variable defined in a script or function that will evaluate the template. +It is important to use single quotes on the right-hand side of the definition above; +otherwise, bash will try to evaluate ``${some_dir}`` when constructing ``var_defns.sh``, +which may result in an error and/or unexpected behavior (e.g., if ``${some_dir}`` +is not yet defined). The experiment generation system will define ``MY_CMD`` in +``var_defns.sh`` in exactly the same way as in ``config_defaults.yaml`` and/or +``config.yaml``, e.g., ``MY_CMD: 'cd ${some_dir}'``. Then the following code snippet +in a script or function will evaluate the contents of ``MY_CMD`` using a locally-set +value of ``some_dir``: + + .. code-block:: none + + ... + . var_defns.sh # Source the experiment's variable definition file (assuming + # it is in the current directory). This defines the MY_CMD + # template variable (in addition to other variables). + ... + some_dir="20200715" # Set the local variable some_dir. + ... + eval ${MY_CMD} # Use eval to evaluate the contents of MY_CMD. The value of + # some_dir specified in this file a few lines above is substituted + # for ${some_dir} in MY_CMD before MY_CMD is evaluated. + +Graphics Plotting Example +============================ + +When attempting to generate graphics plots from a forecast, users have the option to +produce difference plots from two experiments that are on the same domain and +available for the same cycle starting date/time and forecast hours. +To generate difference plots, users must use the template variable ``COMOUT_REF`` +to indicate where the :term:`GRIB2` files from post-processing are located. + +In *community* mode (i.e., when ``RUN_ENVIR: "community"``), this directory will +take the form ``/path/to/exptdir/$PDY$cyc/postprd``, where ``$PDY`` refers to the +cycle date in YYYYMMDD format, and ``$cyc`` refers to the starting hour of the cycle. +(These variables are set in previous tasks based on the value of ``DATE_FIRST_CYCL``.) +Concretely, users can set ``COMOUT_REF`` as follows: + +.. code-block:: console + + COMOUT_REF: '${EXPT_BASEDIR}/${EXPT_SUBDIR}/${PDY}${cyc}/postprd' + +In *nco* mode, this directory should be set to the location of the ``COMOUT`` directory +(``${COMOUT}`` in the example below) and end with ``${PDY}/${cyc}``. For example: + +.. code-block:: console + + COMOUT_REF: '${COMOUT}/${PDY}/${cyc}/' + diff --git a/docs/UsersGuide/source/index.rst b/docs/UsersGuide/source/index.rst index 93595b3063..c1d6c844b5 100644 --- a/docs/UsersGuide/source/index.rst +++ b/docs/UsersGuide/source/index.rst @@ -22,6 +22,7 @@ UFS Short-Range Weather App Users Guide ConfigWorkflow RocotoInfo WE2Etests + TemplateVars VXCases Graphics FAQ From fc066a0dd67d4d17c3f6afb985d8ade197f0597e Mon Sep 17 00:00:00 2001 From: gspetro Date: Wed, 4 Jan 2023 11:45:03 -0500 Subject: [PATCH 04/17] add plotting info to RunSRW; remove Graphics ch --- docs/UsersGuide/source/RunSRW.rst | 126 ++++++++++++++++++------------ docs/UsersGuide/source/index.rst | 1 - 2 files changed, 75 insertions(+), 52 deletions(-) diff --git a/docs/UsersGuide/source/RunSRW.rst b/docs/UsersGuide/source/RunSRW.rst index 838f49ff23..e93c89dd08 100644 --- a/docs/UsersGuide/source/RunSRW.rst +++ b/docs/UsersGuide/source/RunSRW.rst @@ -534,6 +534,81 @@ Valid values for configuration variables should be consistent with those in the To configure an experiment and python environment for a general Linux or Mac system, see the :ref:`next section `. To configure an experiment to run METplus verification tasks, see :numref:`Section %s `. Otherwise, skip to :numref:`Section %s ` to generate the workflow. +.. _PlotOutput: + +Plot the Output +=============== + +.. Add this section above to the Config section before the METplus info! + +An optional Python plotting task (PLOT_ALLVARS) can be activated in the workflow to generate plots for the :term:`FV3`-:term:`LAM` post-processed :term:`GRIB2` +output over the :term:`CONUS`. It generates plots for a number of variables, including: + +* 2-m temperature +* 2-m dew point temperature +* 10-m winds +.. * 500 hPa heights, winds, and vorticity --> seems to be omitted? +* 250 hPa winds +* Accumulated precipitation +* Composite reflectivity +* Surface-based :term:`CAPE`/:term:`CIN` +* Max/Min 2-5 km updraft helicity +* Sea level pressure (SLP) + +This workflow task can produce both plots from a single experiment and difference plots that compare the same cycle from two different experiments. When plotting the difference, the two experiments must be on the same domain and available for +the same cycle starting date/time and forecast hours. Other parameters may differ (e.g., the experiments may use different physics suites). + +.. _Cartopy: + +Cartopy Shapefiles +--------------------- + +The Python plotting tasks require a path to the directory where the Cartopy Natural Earth shapefiles are located. The medium scale (1:50m) cultural and physical shapefiles are used to create coastlines and other geopolitical borders on the map. On `Level 1 `__ systems, this path is already set in the machine file using the variable ``FIXshp``. Users on other systems will need to download the shapefiles and update the path of ``$FIXshp`` in the machine file they are using (e.g., ``$HOME/ush/machine/macos.yaml`` for a generic MacOS system, where ``$HOME`` is the path to the ``ufs-srweather-app`` directory). The subset of shapefiles required for the plotting task can be obtained from the `SRW Data Bucket `__. The full set of Cartopy shapefiles can be downloaded `here `__. + +Task Configuration +--------------------- + +Users will need to add or modify certain variables in ``config.yaml`` to run the plotting task(s). At a minimum, users must set ``RUN_TASK_PLOT_ALLVARS`` to true: + +.. code-block:: console + + workflow_switches: + RUN_TASK_PLOT_ALLVARS: true + +Plotting Output From a Single Experiment +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Users may also wish to adjust the start, end, and increment value for the plotting task. For example: + +.. code-block:: console + + task_plot_allvars: + PLOT_FCST_START: 0 + PLOT_FCST_INC: 6 + PLOT_FCST_END: 12 + +If the user chooses not to set these values, the default values will be used (see :numref:`Section %s `). + +.. note:: + If a forecast starts at 18h, this is considered the 0th forecast hour, so "starting forecast hour" should be 0, not 18. + +The output files (in ``.png`` format) will be located in the experiment directory under the ``$CDATE/postprd`` subdirectory where ``$CDATE`` +corresponds to the cycle date and hour in YYYYMMDDHH format (e.g., ``2019061518``). + +Plotting the Difference Between Two Experiments +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +When plotting the difference between two experiments, users must set the baseline experiment directory using the ``COMOUT_REF`` template variable. For example, in *community* mode, users can set: + +.. code-block:: console + + task_plot_allvars: + COMOUT_REF: '${EXPT_BASEDIR}/${EXPT_SUBDIR}/${PDY}${cyc}/postprd' + +In *community* mode, using default directory names and settings, ``$COMOUT_REF`` will resemble ``/path/to/expt_dirs/test_community/2019061518/postprd``. Additional details on the plotting variables are provided in :numref:`Section %s `. + +The output files (in ``.png`` format) will be located in the ``postprd`` directory for the experiment. + .. _LinuxMacEnvConfig: User-Specific Configuration on a Generic Linux/MacOS System @@ -1260,54 +1335,3 @@ Users can access log files for specific tasks in the ``$EXPTDIR/log`` directory. .. note:: On most HPC systems, users will need to submit a batch job to run multi-processor jobs. On some HPC systems, users may be able to run the first two jobs (serial) on a login node/command-line. Example scripts for Slurm (Hera) and PBS (Cheyenne) resource managers are provided (``sq_job.sh`` and ``qsub_job.sh``, respectively). These examples will need to be adapted to each user's system. Alternatively, some batch systems allow users to specify most of the settings on the command line (with the ``sbatch`` or ``qsub`` command, for example). - - - -.. _PlotOutput: - -Plot the Output -=============== - -.. Add this section above to the Config section before the METplus info! - -An optional Python plotting task can be activated in the workflow to generate plots for the :term:`FV3`-:term:`LAM` post-processed :term:`GRIB2` -output over the :term:`CONUS`. It generates plots for a number of variables, including: - -* 2-m temperature -* 2-m dew point temperature -* 10-m winds -* 500 hPa heights, winds, and vorticity -* 250 hPa winds -* Accumulated precipitation -* Composite reflectivity -* Surface-based :term:`CAPE`/:term:`CIN` -* Max/Min 2-5 km updraft helicity -* Sea level pressure (SLP) - -This workflow task can produce both plots from a single experiment and difference plots that compare the same cycle from two different experiments. When plotting the difference, the two experiments must be on the same domain and available for -the same cycle starting date/time and forecast hours. Other parameters may differ (e.g., the experiments may use different physics suites). - -.. _Cartopy: - -Cartopy Shapefiles ---------------------- - -The Python plotting tasks require a path to the directory where the Cartopy Natural Earth shapefiles are located. The medium scale (1:50m) cultural and physical shapefiles are used to create coastlines and other geopolitical borders on the map. On `Level 1 `__ systems, this path is already set in the machine file using the variable ``FIXshp``. Users on other systems will need to download the shapefiles and update the path of ``$FIXshp`` in the machine file they are using (e.g., ``$HOME/ush/machine/macos.yaml`` for a generic MacOS system). The subset of shapefiles required for the plotting task can be obtained from the `SRW Data Bucket `__. The full set of Cartopy shapefiles can be downloaded `here `__. - -Additionally, users may need to add or modify certain variables in ``config.yaml``. For example: - -.. code-block:: console - - workflow_switches: - RUN_TASK_PLOT_ALLVARS: true - task_plot_allvars: - COMOUT_REF: - PLOT_FCST_START: 0 - PLOT_FCST_INC: 6 - PLOT_FCST_END: 12 - -where ``$EXPTDIR`` is the path to the experiment directory, and ``$CDATE`` is the cycle date. In *community* mode, using default directory names, ``$COMOUT_REF`` will resemble ``/path/to/expt_dirs/test_community/2019061518/postprd``. Additional details on the plotting variables is provided in :numref:`Section %s `. - - -.. NOTES: - plot_allvars and plot_allvars_diff are done within the same task. The user would need to provide the baseline experiment directory EXPTDIR_REF if diff plots are needed. \ No newline at end of file diff --git a/docs/UsersGuide/source/index.rst b/docs/UsersGuide/source/index.rst index c1d6c844b5..9fc9c9acbb 100644 --- a/docs/UsersGuide/source/index.rst +++ b/docs/UsersGuide/source/index.rst @@ -24,6 +24,5 @@ UFS Short-Range Weather App Users Guide WE2Etests TemplateVars VXCases - Graphics FAQ Glossary From 7b04b9868b2789aeac253ad808b487eac839a2e8 Mon Sep 17 00:00:00 2001 From: gspetro Date: Wed, 4 Jan 2023 12:57:44 -0500 Subject: [PATCH 05/17] add sphinx_rtd extension --- docs/UsersGuide/source/conf.py | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/UsersGuide/source/conf.py b/docs/UsersGuide/source/conf.py index 1d2ca24325..1490321e96 100644 --- a/docs/UsersGuide/source/conf.py +++ b/docs/UsersGuide/source/conf.py @@ -41,6 +41,7 @@ # extensions coming with Sphinx (named 'sphinx.ext.*') or your custom # ones. extensions = [ + 'sphinx_rtd_theme', 'sphinx.ext.autodoc', 'sphinx.ext.doctest', 'sphinx.ext.intersphinx', From 4009caeffc0b4676e030275f6a91a7d83a9e8d54 Mon Sep 17 00:00:00 2001 From: gspetro Date: Wed, 4 Jan 2023 13:36:46 -0500 Subject: [PATCH 06/17] add docutils requirement to render bullet points correctly --- docs/UsersGuide/requirements.txt | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/UsersGuide/requirements.txt b/docs/UsersGuide/requirements.txt index 9c7258463b..e667ae45c1 100644 --- a/docs/UsersGuide/requirements.txt +++ b/docs/UsersGuide/requirements.txt @@ -1,2 +1,3 @@ sphinxcontrib-bibtex sphinx_rtd_theme +docutils=0.17 \ No newline at end of file From 7ff6eee427845aa96a60f2c20993de2cfbda4165 Mon Sep 17 00:00:00 2001 From: gspetro Date: Wed, 4 Jan 2023 14:13:03 -0500 Subject: [PATCH 07/17] reformat plotting section --- docs/UsersGuide/source/RunSRW.rst | 12 +++++------- 1 file changed, 5 insertions(+), 7 deletions(-) diff --git a/docs/UsersGuide/source/RunSRW.rst b/docs/UsersGuide/source/RunSRW.rst index e93c89dd08..41457005f7 100644 --- a/docs/UsersGuide/source/RunSRW.rst +++ b/docs/UsersGuide/source/RunSRW.rst @@ -537,9 +537,7 @@ To configure an experiment and python environment for a general Linux or Mac sys .. _PlotOutput: Plot the Output -=============== - -.. Add this section above to the Config section before the METplus info! +----------------- An optional Python plotting task (PLOT_ALLVARS) can be activated in the workflow to generate plots for the :term:`FV3`-:term:`LAM` post-processed :term:`GRIB2` output over the :term:`CONUS`. It generates plots for a number of variables, including: @@ -561,12 +559,12 @@ the same cycle starting date/time and forecast hours. Other parameters may diffe .. _Cartopy: Cartopy Shapefiles ---------------------- +^^^^^^^^^^^^^^^^^^^^^ The Python plotting tasks require a path to the directory where the Cartopy Natural Earth shapefiles are located. The medium scale (1:50m) cultural and physical shapefiles are used to create coastlines and other geopolitical borders on the map. On `Level 1 `__ systems, this path is already set in the machine file using the variable ``FIXshp``. Users on other systems will need to download the shapefiles and update the path of ``$FIXshp`` in the machine file they are using (e.g., ``$HOME/ush/machine/macos.yaml`` for a generic MacOS system, where ``$HOME`` is the path to the ``ufs-srweather-app`` directory). The subset of shapefiles required for the plotting task can be obtained from the `SRW Data Bucket `__. The full set of Cartopy shapefiles can be downloaded `here `__. Task Configuration ---------------------- +^^^^^^^^^^^^^^^^^^^^^ Users will need to add or modify certain variables in ``config.yaml`` to run the plotting task(s). At a minimum, users must set ``RUN_TASK_PLOT_ALLVARS`` to true: @@ -576,7 +574,7 @@ Users will need to add or modify certain variables in ``config.yaml`` to run the RUN_TASK_PLOT_ALLVARS: true Plotting Output From a Single Experiment -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +```````````````````````````````````````````` Users may also wish to adjust the start, end, and increment value for the plotting task. For example: @@ -596,7 +594,7 @@ The output files (in ``.png`` format) will be located in the experiment director corresponds to the cycle date and hour in YYYYMMDDHH format (e.g., ``2019061518``). Plotting the Difference Between Two Experiments -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +``````````````````````````````````````````````````` When plotting the difference between two experiments, users must set the baseline experiment directory using the ``COMOUT_REF`` template variable. For example, in *community* mode, users can set: From 446f8757df41bbb18f0d103cd509357ec4eacfba Mon Sep 17 00:00:00 2001 From: gspetro Date: Wed, 4 Jan 2023 14:14:24 -0500 Subject: [PATCH 08/17] update requirements to render bullets --- docs/UsersGuide/requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/UsersGuide/requirements.txt b/docs/UsersGuide/requirements.txt index e667ae45c1..5158b75c29 100644 --- a/docs/UsersGuide/requirements.txt +++ b/docs/UsersGuide/requirements.txt @@ -1,3 +1,3 @@ sphinxcontrib-bibtex -sphinx_rtd_theme +sphinx_rtd_theme=1.1.1 docutils=0.17 \ No newline at end of file From 4cffc0de9ff62451a2aea50e15892168e3f50b71 Mon Sep 17 00:00:00 2001 From: gspetro Date: Wed, 4 Jan 2023 14:15:51 -0500 Subject: [PATCH 09/17] fix requirements syntax --- docs/UsersGuide/requirements.txt | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/UsersGuide/requirements.txt b/docs/UsersGuide/requirements.txt index 5158b75c29..1e7a0ab1f8 100644 --- a/docs/UsersGuide/requirements.txt +++ b/docs/UsersGuide/requirements.txt @@ -1,3 +1,3 @@ sphinxcontrib-bibtex -sphinx_rtd_theme=1.1.1 -docutils=0.17 \ No newline at end of file +sphinx_rtd_theme==1.1.1 +docutils==0.17 \ No newline at end of file From 7ff59870539eef7afd5307870523dcf63991eab0 Mon Sep 17 00:00:00 2001 From: gspetro Date: Wed, 4 Jan 2023 14:55:05 -0500 Subject: [PATCH 10/17] update sphinx requirements --- docs/UsersGuide/requirements.txt | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/docs/UsersGuide/requirements.txt b/docs/UsersGuide/requirements.txt index 1e7a0ab1f8..457a26ebaa 100644 --- a/docs/UsersGuide/requirements.txt +++ b/docs/UsersGuide/requirements.txt @@ -1,3 +1,4 @@ sphinxcontrib-bibtex sphinx_rtd_theme==1.1.1 -docutils==0.17 \ No newline at end of file +docutils==0.17 +sphinx==5.3.0 \ No newline at end of file From 23ec925610ee217d37ec093835d602c12d83c2a0 Mon Sep 17 00:00:00 2001 From: gspetro Date: Wed, 4 Jan 2023 15:00:30 -0500 Subject: [PATCH 11/17] asterisk to dash bullets --- docs/UsersGuide/source/RunSRW.rst | 21 +++++++++++---------- 1 file changed, 11 insertions(+), 10 deletions(-) diff --git a/docs/UsersGuide/source/RunSRW.rst b/docs/UsersGuide/source/RunSRW.rst index 41457005f7..9a60b64207 100644 --- a/docs/UsersGuide/source/RunSRW.rst +++ b/docs/UsersGuide/source/RunSRW.rst @@ -540,18 +540,19 @@ Plot the Output ----------------- An optional Python plotting task (PLOT_ALLVARS) can be activated in the workflow to generate plots for the :term:`FV3`-:term:`LAM` post-processed :term:`GRIB2` -output over the :term:`CONUS`. It generates plots for a number of variables, including: +output over the :term:`CONUS`. It generates graphics plots for a number of variables, including: + +- 2-m temperature +- 2-m dew point temperature +- 10-m winds +- 250 hPa winds +- Accumulated precipitation +- Composite reflectivity +- Surface-based :term:`CAPE`/:term:`CIN` +- Max/Min 2-5 km updraft helicity +- Sea level pressure (SLP) -* 2-m temperature -* 2-m dew point temperature -* 10-m winds .. * 500 hPa heights, winds, and vorticity --> seems to be omitted? -* 250 hPa winds -* Accumulated precipitation -* Composite reflectivity -* Surface-based :term:`CAPE`/:term:`CIN` -* Max/Min 2-5 km updraft helicity -* Sea level pressure (SLP) This workflow task can produce both plots from a single experiment and difference plots that compare the same cycle from two different experiments. When plotting the difference, the two experiments must be on the same domain and available for the same cycle starting date/time and forecast hours. Other parameters may differ (e.g., the experiments may use different physics suites). From 1b35d2c2e67d1a7893f4be3e578bc0259856528c Mon Sep 17 00:00:00 2001 From: gspetro Date: Wed, 4 Jan 2023 15:51:05 -0500 Subject: [PATCH 12/17] revert changes to fix bullet points --- docs/UsersGuide/requirements.txt | 4 +--- docs/UsersGuide/source/RunSRW.rst | 18 +++++++++--------- docs/UsersGuide/source/conf.py | 1 - 3 files changed, 10 insertions(+), 13 deletions(-) diff --git a/docs/UsersGuide/requirements.txt b/docs/UsersGuide/requirements.txt index 457a26ebaa..edcb382114 100644 --- a/docs/UsersGuide/requirements.txt +++ b/docs/UsersGuide/requirements.txt @@ -1,4 +1,2 @@ sphinxcontrib-bibtex -sphinx_rtd_theme==1.1.1 -docutils==0.17 -sphinx==5.3.0 \ No newline at end of file +sphinx_rtd_theme \ No newline at end of file diff --git a/docs/UsersGuide/source/RunSRW.rst b/docs/UsersGuide/source/RunSRW.rst index 9a60b64207..4a02925e8f 100644 --- a/docs/UsersGuide/source/RunSRW.rst +++ b/docs/UsersGuide/source/RunSRW.rst @@ -542,15 +542,15 @@ Plot the Output An optional Python plotting task (PLOT_ALLVARS) can be activated in the workflow to generate plots for the :term:`FV3`-:term:`LAM` post-processed :term:`GRIB2` output over the :term:`CONUS`. It generates graphics plots for a number of variables, including: -- 2-m temperature -- 2-m dew point temperature -- 10-m winds -- 250 hPa winds -- Accumulated precipitation -- Composite reflectivity -- Surface-based :term:`CAPE`/:term:`CIN` -- Max/Min 2-5 km updraft helicity -- Sea level pressure (SLP) +* 2-m temperature +* 2-m dew point temperature +* 10-m winds +* 250 hPa winds +* Accumulated precipitation +* Composite reflectivity +* Surface-based :term:`CAPE`/:term:`CIN` +* Max/Min 2-5 km updraft helicity +* Sea level pressure (SLP) .. * 500 hPa heights, winds, and vorticity --> seems to be omitted? diff --git a/docs/UsersGuide/source/conf.py b/docs/UsersGuide/source/conf.py index 1490321e96..1d2ca24325 100644 --- a/docs/UsersGuide/source/conf.py +++ b/docs/UsersGuide/source/conf.py @@ -41,7 +41,6 @@ # extensions coming with Sphinx (named 'sphinx.ext.*') or your custom # ones. extensions = [ - 'sphinx_rtd_theme', 'sphinx.ext.autodoc', 'sphinx.ext.doctest', 'sphinx.ext.intersphinx', From 75f9db1eb84c4f9bd7b4b610163b4ea7c26c5e0b Mon Sep 17 00:00:00 2001 From: gspetro Date: Wed, 4 Jan 2023 16:38:17 -0500 Subject: [PATCH 13/17] minor edits to RunSRW plotting section --- docs/UsersGuide/source/RunSRW.rst | 36 +++++++++++++++---------------- 1 file changed, 17 insertions(+), 19 deletions(-) diff --git a/docs/UsersGuide/source/RunSRW.rst b/docs/UsersGuide/source/RunSRW.rst index 4a02925e8f..6496bd90b9 100644 --- a/docs/UsersGuide/source/RunSRW.rst +++ b/docs/UsersGuide/source/RunSRW.rst @@ -21,9 +21,10 @@ The overall procedure for generating an experiment is shown in :numref:`Figure % * :ref:`Load the python environment for the regional workflow ` * :ref:`Set the experiment configuration parameters ` + * :ref:`Optional: Plot the output ` + * :ref:`Optional: Configure METplus Verification Suite ` - #. :ref:`Run the regional workflow ` - #. :ref:`Optional: Plot the output ` + #. :ref:`Run the regional workflow ` .. _AppOverallProc: @@ -542,19 +543,19 @@ Plot the Output An optional Python plotting task (PLOT_ALLVARS) can be activated in the workflow to generate plots for the :term:`FV3`-:term:`LAM` post-processed :term:`GRIB2` output over the :term:`CONUS`. It generates graphics plots for a number of variables, including: -* 2-m temperature -* 2-m dew point temperature -* 10-m winds -* 250 hPa winds -* Accumulated precipitation -* Composite reflectivity -* Surface-based :term:`CAPE`/:term:`CIN` -* Max/Min 2-5 km updraft helicity -* Sea level pressure (SLP) + * 2-m temperature + * 2-m dew point temperature + * 10-m winds + * 250 hPa winds + * Accumulated precipitation + * Composite reflectivity + * Surface-based :term:`CAPE`/:term:`CIN` + * Max/Min 2-5 km updraft helicity + * Sea level pressure (SLP) -.. * 500 hPa heights, winds, and vorticity --> seems to be omitted? +.. COMMENT: * 500 hPa heights, winds, and vorticity --> seems to be omitted? Why? -This workflow task can produce both plots from a single experiment and difference plots that compare the same cycle from two different experiments. When plotting the difference, the two experiments must be on the same domain and available for +This workflow task can produce both plots from a single experiment and difference plots that compare the same cycle from two experiments. When plotting the difference, the two experiments must be on the same domain and available for the same cycle starting date/time and forecast hours. Other parameters may differ (e.g., the experiments may use different physics suites). .. _Cartopy: @@ -562,20 +563,17 @@ the same cycle starting date/time and forecast hours. Other parameters may diffe Cartopy Shapefiles ^^^^^^^^^^^^^^^^^^^^^ -The Python plotting tasks require a path to the directory where the Cartopy Natural Earth shapefiles are located. The medium scale (1:50m) cultural and physical shapefiles are used to create coastlines and other geopolitical borders on the map. On `Level 1 `__ systems, this path is already set in the machine file using the variable ``FIXshp``. Users on other systems will need to download the shapefiles and update the path of ``$FIXshp`` in the machine file they are using (e.g., ``$HOME/ush/machine/macos.yaml`` for a generic MacOS system, where ``$HOME`` is the path to the ``ufs-srweather-app`` directory). The subset of shapefiles required for the plotting task can be obtained from the `SRW Data Bucket `__. The full set of Cartopy shapefiles can be downloaded `here `__. +The Python plotting tasks require a path to the directory where the Cartopy Natural Earth shapefiles are located. The medium scale (1:50m) cultural and physical shapefiles are used to create coastlines and other geopolitical borders on the map. On `Level 1 `__ systems, this path is already set in the system's machine file using the variable ``FIXshp``. Users on other systems will need to download the shapefiles and update the path of ``$FIXshp`` in the machine file they are using (e.g., ``$HOME/ush/machine/macos.yaml`` for a generic MacOS system, where ``$HOME`` is the path to the ``ufs-srweather-app`` directory). The subset of shapefiles required for the plotting task can be obtained from the `SRW Data Bucket `__. The full set of medium-scale (1:50m) Cartopy shapefiles can be downloaded `here `__. Task Configuration ^^^^^^^^^^^^^^^^^^^^^ -Users will need to add or modify certain variables in ``config.yaml`` to run the plotting task(s). At a minimum, users must set ``RUN_TASK_PLOT_ALLVARS`` to true: +Users will need to add or modify certain variables in ``config.yaml`` to run the plotting task(s). At a minimum, users must set ``RUN_TASK_PLOT_ALLVARS`` to true in the ``workflow_switches:`` section: .. code-block:: console workflow_switches: RUN_TASK_PLOT_ALLVARS: true - -Plotting Output From a Single Experiment -```````````````````````````````````````````` Users may also wish to adjust the start, end, and increment value for the plotting task. For example: @@ -591,7 +589,7 @@ If the user chooses not to set these values, the default values will be used (se .. note:: If a forecast starts at 18h, this is considered the 0th forecast hour, so "starting forecast hour" should be 0, not 18. -The output files (in ``.png`` format) will be located in the experiment directory under the ``$CDATE/postprd`` subdirectory where ``$CDATE`` +When plotting output from a single experiment, no further adjustments are necessary. The output files (in ``.png`` format) will be located in the experiment directory under the ``$CDATE/postprd`` subdirectory where ``$CDATE`` corresponds to the cycle date and hour in YYYYMMDDHH format (e.g., ``2019061518``). Plotting the Difference Between Two Experiments From 32b9646797abd657f37e5ddb6a6fefcbd900a0bf Mon Sep 17 00:00:00 2001 From: gspetro Date: Wed, 4 Jan 2023 16:39:15 -0500 Subject: [PATCH 14/17] remove Graphics chapter --- docs/UsersGuide/source/Graphics.rst | 211 ---------------------------- 1 file changed, 211 deletions(-) delete mode 100644 docs/UsersGuide/source/Graphics.rst diff --git a/docs/UsersGuide/source/Graphics.rst b/docs/UsersGuide/source/Graphics.rst deleted file mode 100644 index 0614dd4267..0000000000 --- a/docs/UsersGuide/source/Graphics.rst +++ /dev/null @@ -1,211 +0,0 @@ -.. _Graphics: - -=================== -Graphics Generation -=================== -Two Python plotting scripts are provided to generate plots from the :term:`FV3`-:term:`LAM` post-processed :term:`GRIB2` -output over the :term:`CONUS` for a number of variables, including: - -* 2-m temperature -* 2-m dew point temperature -* 10-m winds -* 500 hPa heights, winds, and vorticity -* 250 hPa winds -* Accumulated precipitation -* Composite reflectivity -* Surface-based :term:`CAPE`/:term:`CIN` -* Max/Min 2-5 km updraft helicity -* Sea level pressure (SLP) - -The Python scripts are located under ``ufs-srweather-app/ush/Python``. -The script ``plot_allvars.py`` plots the output from a single cycle within an experiment, while -the script ``plot_allvars_diff.py`` plots the difference between the same cycle from two different experiments. When -plotting the difference, the two experiments must be on the same domain and available for -the same cycle starting date/time and forecast hours. Other parameters may differ (e.g., the experiments may use different physics suites). - -Loading the Environment -========================== - -To use the plotting scripts, the regional workflow environment, which includes the required ``scipy``, ``matplotlib``, ``pygrib``, ``cartopy``, and ``pillow`` packages, must be loaded. To activate the regional workflow, see :numref:`Section %s `, or use the following summary: - -.. code-block:: console - - cd - source ../../etc/lmod-setup.sh - module use ../../modulefiles - module load wflow_ - -where ```` refers to a valid machine name (see :numref:`Section %s `). Then users should follow the instructions output by the console (e.g., ``conda activate regional_workflow``). - -.. _Cartopy: - -Cartopy Shapefiles -======================= - -The Python plotting scripts also require a path to the directory where the Cartopy Natural Earth shapefiles are located. The medium scale (1:50m) cultural and physical shapefiles are used to create coastlines and other geopolitical borders on the map. Cartopy provides the 'background_img()' method to add background images in a convenient way. The default scale (resolution) of background attributes in the Python scripts is 1:50m Natural Earth I with Shaded Relief and Water, which should be sufficient for most regional applications. - -The full set of Cartopy shapefiles can be downloaded `here `__. For convenience, the small subset of files required for these Python scripts can be obtained from the `SRW Data Bucket `__. They are also available on all `Level 1 `__ platforms in the following locations: - -.. _CartopyData: - -.. table:: Cartopy Shapefile Locations for Level 1 Systems - - +--------------+-----------------------------------------------------------------+ - | Machine | File location | - +==============+=================================================================+ - | Cheyenne | /glade/p/ral/jntp/UFS_SRW_App/develop/NaturalEarth | - +--------------+-----------------------------------------------------------------+ - | Gaea | /lustre/f2/pdata/ncep/UFS_SRW_App/develop/NaturalEarth | - +--------------+-----------------------------------------------------------------+ - | Hera | /scratch2/BMC/det/UFS_SRW_App/develop/NaturalEarth | - +--------------+-----------------------------------------------------------------+ - | Jet | /mnt/lfs4/BMC/wrfruc/UFS_SRW_App/develop/NaturalEarth | - +--------------+-----------------------------------------------------------------+ - | NOAA Cloud | /contrib/EPIC/UFS_SRW_App/develop/NaturalEarth | - +--------------+-----------------------------------------------------------------+ - | Orion | /work/noaa/fv3-cam/UFS_SRW_App/develop/NaturalEarth | - +--------------+-----------------------------------------------------------------+ - -Running the Plotting Scripts -====================================== - -Before generating plots, it is helpful to change location to the directory containing the plotting -scripts: - -.. code-block:: console - - cd ufs-srweather-app/ush/Python - -Plotting Output from One Experiment --------------------------------------- - -To generate plots for a single cycle, the ``plot_allvars.py`` script must be called with the -following command line arguments: - -#. Cycle date/time (``CDATE``) in YYYYMMDDHH format -#. Starting forecast hour -#. Ending forecast hour -#. Forecast hour increment -#. The top level of the experiment directory ``$EXPTDIR`` containing the post-processed data. The script will look for the data files in the directory ``$EXPTDIR/CDATE/postprd``. -#. The base directory ``CARTOPY_DIR`` of the cartopy shapefiles. The script will look for the shapefiles (``*.shp``) in the directory ``$CARTOPY_DIR/shapefiles/natural_earth/cultural``. See :numref:`Table %s ` for the correct ``$CARTOPY_DIR`` locations on Level 1 systems. -#. The name ``POST_OUTPUT_DOMAIN_NAME`` of the native grid used in the forecast - -.. note:: - If a forecast starts at 18h, this is considered the 0th forecast hour, so "starting forecast hour" should be 0, not 18. - -An example of plotting output from a cycle generated using the sample experiment/workflow -configuration in the ``config.community.yaml`` script (which uses the GFSv16 suite definition file) -is as follows: - -.. code-block:: console - - python plot_allvars.py 2019061518 0 12 6 /path-to/expt_dirs/test_community /path-to/NaturalEarth RRFS_CONUS_25km - -The output files (in ``.png`` format) will be located in the directory ``$EXPTDIR/CDATE/postprd``, -where in this case ``$EXPTDIR`` is ``/path-to/expt_dirs/test_CONUS_25km_GFSv16`` and ``$CDATE`` -is ``2019061518``. - -Plotting Differences from Two Experiments --------------------------------------------- - -To generate difference plots, the ``plot_allvars_diff.py`` script must be called with the following -command line arguments: - -#. Cycle date/time (``CDATE``) in YYYYMMDDHH format -#. Starting forecast hour -#. Ending forecast hour -#. Forecast hour increment -#. The top level of the first experiment directory ``$EXPTDIR1`` containing the first set of post-processed data. The script will look for the data files in the directory ``$EXPTDIR1/CDATE/postprd``. -#. The top level of the second experiment directory ``$EXPTDIR2`` containing the second set of post-processed data. The script will look for the data files in the directory ``$EXPTDIR2/CDATE/postprd``. -#. The base directory ``CARTOPY_DIR`` of the cartopy shapefiles. The script will look for the shapefiles (``*.shp``) in the directory ``$CARTOPY_DIR/shapefiles/natural_earth/cultural``. -#. The name ``POST_OUTPUT_DOMAIN_NAME`` of the native grid used in the forecasts (this must be the same for the two forecasts) - -An example of plotting differences from two experiments for the same date and predefined domain where one uses the ``FV3_GFS_v16`` suite definition file (SDF) and one uses the ``FV3_RRFS_v1beta`` SDF is as follows: - -.. code-block:: console - - python plot_allvars_diff.py 2019061518 0 12 6 /path-to/expt_dirs1/test_CONUS_3km_GFSv16 /path-to/expt_dirs2/test_CONUS_3km_RRFSv1beta /path-to/NaturalEarth RRFS_CONUS_25km - -In this case, the output ``.png`` files will be located in the directory ``$EXPTDIR1/CDATE/postprd``. - -.. _Batch: - -Submitting Plotting Scripts Through a Batch System -====================================================== - -If users plan to create plots of multiple forecast lead times and forecast variables, then they may need to submit the Python scripts to the batch system. Sample scripts are provided for use on a platform such as Hera that uses the Slurm job scheduler: ``sq_job.sh`` and ``sq_job_diff.sh``. Equivalent sample scripts are provided for use on a platform such as Cheyenne that uses PBS as the job scheduler: ``qsub_job.sh`` and ``qsub_job_diff.sh``. Examples of these scripts are located under ``ufs-srweather-app/ush/Python`` and can be used as a starting point to create a batch script for the user's specific platform/job scheduler. - -At a minimum, the account should be set appropriately prior to job submission: - -.. code-block:: console - - #SBATCH --account= - -Depending on the platform, users may also need to adjust the settings to use the correct Python environment and path to the shapefiles. - -When working with these batch scripts, several environment variables must be set prior to submission. -If plotting output from a single cycle, the variables to set are ``$HOMEdir`` and ``$EXPTDIR``. -If the user's login shell is bash, these variables can be set as follows: - -.. code-block:: console - - export HOMEdir=/path-to/ufs-srweather-app - export EXPTDIR=/path-to/experiment/directory - -If the user's login shell is csh/tcsh, they can be set as follows: - -.. code-block:: console - - setenv HOMEdir /path-to/ufs-srweather-app - setenv EXPTDIR /path-to/experiment/directory - -If plotting the difference between the same cycle from two different experiments, the variables -to set are ``$HOMEdir``, ``$EXPTDIR1``, and ``$EXPTDIR2``. If the user's login shell -is bash, these variables can be set as follows: - -.. code-block:: console - - export HOMEdir=/path-to/ufs-srweather-app - export EXPTDIR1=/path-to/experiment/directory1 - export EXPTDIR2=/path-to/experiment/directory2 - -If the user's login shell is csh/tcsh, they can be set as follows: - -.. code-block:: console - - setenv HOMEdir /path-to/ufs-srweather-app - setenv EXPTDIR1 /path-to/experiment/directory1 - setenv EXPTDIR2 /path-to/experiment/directory2 - -In addition, the variables ``CDATE``, ``FCST_START``, ``FCST_END``, and ``FCST_INC`` in the batch -scripts can be modified depending on the user's needs. By default, ``CDATE`` is set as follows -in the batch scripts: - -.. code-block:: console - - export CDATE=${DATE_FIRST_CYCL} - -This sets ``CDATE`` to the first cycle in the set of cycles that the experiment has run. If the -experiment contains multiple cycles and the user wants to plot output from a cycle other than -the very first one, ``CDATE`` in the batch scripts will have to be set to the specific YYYYMMDDHH -value for that cycle. Also, to plot hourly forecast output, ``FCST_INC`` should be set to 1; to -plot only a subset of the output hours, ``FCST_START``, ``FCST_END``, and ``FCST_INC`` must be -set accordingly. For example, to generate plots for every 3rd forecast hour starting with forecast hour 6 and ending with the last forecast hour, use: - -.. code-block:: console - - export FCST_START=6 - export FCST_END=${FCST_LEN_HRS} - export FCST_INC=3 - -The scripts must be submitted using the command appropriate for the job scheduler on the user's platform. For example, on Hera, ``sq_job.sh`` can be submitted as follows: - -.. code-block:: console - - sbatch sq_job.sh - -On Cheyenne, ``qsub_job.sh`` can be submitted as follows: - -.. code-block:: console - - qsub qsub_job.sh From 9067c550d03a86526f33aede0c0f8a5dcfe26cb6 Mon Sep 17 00:00:00 2001 From: gspetro Date: Wed, 4 Jan 2023 16:49:46 -0500 Subject: [PATCH 15/17] remove/update references to Graphics chapter --- docs/UsersGuide/source/Components.rst | 12 +++--------- docs/UsersGuide/source/ContainerQuickstart.rst | 4 ---- docs/UsersGuide/source/Introduction.rst | 5 ----- 3 files changed, 3 insertions(+), 18 deletions(-) diff --git a/docs/UsersGuide/source/Components.rst b/docs/UsersGuide/source/Components.rst index 2b6ff1ddd8..85b72a5289 100644 --- a/docs/UsersGuide/source/Components.rst +++ b/docs/UsersGuide/source/Components.rst @@ -65,14 +65,6 @@ Among other techniques, MET provides the capability to compute standard verifica METplus is being actively developed by :term:`NCAR`/Research Applications Laboratory (RAL), NOAA/Earth Systems Research Laboratories (ESRL), and NOAA/Environmental Modeling Center (:term:`EMC`), and it is open to community contributions. - -Visualization Example -===================== -A Python script is provided to create basic visualizations of the model output. The script is designed to output graphics in PNG format for several standard meteorological variables -when using the pre-defined :term:`CONUS` domain. A difference plotting script is also included to visually compare two runs for the same domain and resolution. These scripts are provided only as an example for users familiar with Python. They may be used to perform a visual check to verify that the application is producing reasonable results. - -After running ``manage_externals/checkout_externals``, the visualization scripts will be available in the ``ufs-srweather-app/ush/Python`` directory. Usage information and instructions are described in :numref:`Chapter %s ` and are also included at the top of each script. - Build System and Workflow ========================= @@ -81,7 +73,9 @@ The SRW Application has a portable build system and a user-friendly, modular, an An umbrella CMake-based build system is used for building the components necessary for running the end-to-end SRW Application, including the UFS Weather Model and the pre- and post-processing software. Additional libraries necessary for the application (e.g., :term:`NCEPLIBS-external` and :term:`NCEPLIBS`) are not included in the SRW Application build system but are available pre-built on pre-configured platforms. On other systems, they can be installed via the HPC-Stack (see :doc:`HPC-Stack Documentation `). There is a small set of system libraries and utilities that are assumed to be present on the target computer: the CMake build software; a Fortran, C, and C++ compiler; and an :term:`MPI` library. Once built, the provided experiment generator script can be used to create a Rocoto-based -workflow file that will run each task in the system in the proper sequence (see :numref:`Chapter %s ` or the `Rocoto documentation `__ for more information on Rocoto). If Rocoto and/or a batch system is not present on the available platform, the individual components can be run in a stand-alone, command line fashion with provided run scripts. The generated namelist for the atmospheric model can be modified in order to vary settings such as forecast starting and ending dates, forecast length hours, the :term:`CCPP` physics suite, integration time step, history file output frequency, and more. It also allows for configuration of other elements of the workflow; for example, users can choose whether to run some or all of the pre-processing, forecast model, and post-processing steps. +workflow file that will run each task in the system in the proper sequence (see :numref:`Chapter %s ` or the `Rocoto documentation `__ for more information on Rocoto). If Rocoto and/or a batch system is not present on the available platform, the individual components can be run in a stand-alone, command line fashion with provided run scripts. The generated namelist for the atmospheric model can be modified in order to vary settings such as forecast starting and ending dates, forecast length hours, the :term:`CCPP` physics suite, integration time step, history file output frequency, and more. It also allows for configuration of other elements of the workflow; for example, users can choose whether to run some or all of the pre-processing, forecast model, and post-processing steps. + +An optional Python plotting task is also included to create basic visualizations of the model output. The task outputs graphics in PNG format for several standard meteorological variables on the pre-defined :term:`CONUS` domain. A difference plotting option is also included to visually compare two runs for the same domain and resolution. These plots may be used to perform a visual check to verify that the application is producing reasonable results. Configuration instructions are provided in :numref:`Section %s `. The latest SRW Application release has been tested on a variety of platforms widely used by researchers, such as the NOAA Research and Development High-Performance Computing Systems (RDHPCS), including Hera, Orion, and Jet; the National Center for Atmospheric Research (:term:`NCAR`) Cheyenne system; and generic Linux and MacOS systems using Intel and GNU compilers. Four `levels of support `__ have been defined for the SRW Application, including pre-configured (Level 1), configurable (Level 2), limited-test (Level 3), and build-only (Level 4) platforms. Each level is further described below. diff --git a/docs/UsersGuide/source/ContainerQuickstart.rst b/docs/UsersGuide/source/ContainerQuickstart.rst index 025893a840..a83d87d631 100644 --- a/docs/UsersGuide/source/ContainerQuickstart.rst +++ b/docs/UsersGuide/source/ContainerQuickstart.rst @@ -388,7 +388,3 @@ New Experiment =============== To run a new experiment in the container at a later time, users will need to rerun the commands in :numref:`Section %s ` to reactivate the regional workflow. Then, users can configure a new experiment by updating the environment variables in ``config.yaml`` to reflect the desired experiment configuration. Basic instructions appear in :numref:`Section %s ` above, and detailed instructions can be viewed in :numref:`Section %s `. After adjusting the configuration file, regenerate the experiment by running ``./generate_FV3LAM_wflow.py``. - -Plot the Output -=============== -Two python scripts are provided to generate plots from the FV3-LAM post-processed GRIB2 output. Information on how to generate the graphics can be found in :numref:`Chapter %s `. diff --git a/docs/UsersGuide/source/Introduction.rst b/docs/UsersGuide/source/Introduction.rst index d2d2b93096..d8dd8e68a8 100644 --- a/docs/UsersGuide/source/Introduction.rst +++ b/docs/UsersGuide/source/Introduction.rst @@ -179,11 +179,6 @@ METplus Verification Suite The Model Evaluation Tools (MET) package is a set of statistical verification tools developed by the `Developmental Testbed Center `__ (DTC) for use by the :term:`NWP` community to help them assess and evaluate the performance of numerical weather predictions. MET is the core component of the enhanced METplus verification framework. The suite also includes the associated database and display systems called METviewer and METexpress. METplus spans a wide range of temporal and spatial scales. It is intended to be extensible through additional capabilities developed by the community. More details about METplus can be found in :numref:`Chapter %s ` and on the `METplus website `__. -Visualization Example -------------------------- - -The SRW Application includes Python scripts to create basic visualizations of the model output. :numref:`Chapter %s ` contains usage information and instructions; instructions also appear at the top of the scripts. - Build System and Workflow ---------------------------- From 1a7079c2b1e41e64b9c913bb97b21fe3770bce45 Mon Sep 17 00:00:00 2001 From: gspetro Date: Wed, 4 Jan 2023 17:04:43 -0500 Subject: [PATCH 16/17] fix typos --- docs/UsersGuide/source/ConfigWorkflow.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/UsersGuide/source/ConfigWorkflow.rst b/docs/UsersGuide/source/ConfigWorkflow.rst index 709a54e063..57d8151cbb 100644 --- a/docs/UsersGuide/source/ConfigWorkflow.rst +++ b/docs/UsersGuide/source/ConfigWorkflow.rst @@ -1141,7 +1141,7 @@ These parameters are associated with the fixed (i.e., static) files. On `Level 1 System directory where the lookup tables for optics properties are located. ``FIXshp``: (Default: "") - System directory where the graphics shapefiles are located. On Level 1 systems, these are set within the machine files. Users on other systems will need to provide the path to the directory that contains the *Natural Earth* shapfiles. + System directory where the graphics shapefiles are located. On Level 1 systems, these are set within the machine files. Users on other systems will need to provide the path to the directory that contains the *Natural Earth* shapefiles. ``TOPO_DIR``: (Default: "") The location on disk of the static input files used by the ``make_orog`` task (i.e., ``orog.x`` and ``shave.x``). Can be the same as ``FIXgsm``. @@ -1765,7 +1765,7 @@ For each workflow task, certain parameter values must be passed to the job sched Additional Parameters ------------------------ -Typially, the following parameters must be set explicitly by the user in the configuration file (``config.yaml``) when executing the plotting tasks. +Typically, the following parameters must be set explicitly by the user in the configuration file (``config.yaml``) when executing the plotting tasks. ``COMOUT_REF``: (Default: "") The directory where the GRIB2 files from post-processing are located. In *community* mode (i.e., when ``RUN_ENVIR: "community"``), this directory will correspond to the location in the experiment directory where the post-processed output can be found (e.g., ``$EXPTDIR/$DATE_FIRST_CYCL/postprd``). In *nco* mode, this directory should be set to the location of the COMOUT directory and end with ``$PDY/$cyc``. From 1838d4343d31b99d8ce461409e45d62e6cd520e3 Mon Sep 17 00:00:00 2001 From: gspetro Date: Thu, 5 Jan 2023 10:16:02 -0500 Subject: [PATCH 17/17] change /Users/gillianpetro to --- docs/UsersGuide/source/RunSRW.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/UsersGuide/source/RunSRW.rst b/docs/UsersGuide/source/RunSRW.rst index 6496bd90b9..1d74541837 100644 --- a/docs/UsersGuide/source/RunSRW.rst +++ b/docs/UsersGuide/source/RunSRW.rst @@ -563,7 +563,7 @@ the same cycle starting date/time and forecast hours. Other parameters may diffe Cartopy Shapefiles ^^^^^^^^^^^^^^^^^^^^^ -The Python plotting tasks require a path to the directory where the Cartopy Natural Earth shapefiles are located. The medium scale (1:50m) cultural and physical shapefiles are used to create coastlines and other geopolitical borders on the map. On `Level 1 `__ systems, this path is already set in the system's machine file using the variable ``FIXshp``. Users on other systems will need to download the shapefiles and update the path of ``$FIXshp`` in the machine file they are using (e.g., ``$HOME/ush/machine/macos.yaml`` for a generic MacOS system, where ``$HOME`` is the path to the ``ufs-srweather-app`` directory). The subset of shapefiles required for the plotting task can be obtained from the `SRW Data Bucket `__. The full set of medium-scale (1:50m) Cartopy shapefiles can be downloaded `here `__. +The Python plotting tasks require a path to the directory where the Cartopy Natural Earth shapefiles are located. The medium scale (1:50m) cultural and physical shapefiles are used to create coastlines and other geopolitical borders on the map. On `Level 1 `__ systems, this path is already set in the system's machine file using the variable ``FIXshp``. Users on other systems will need to download the shapefiles and update the path of ``$FIXshp`` in the machine file they are using (e.g., ``$SRW/ush/machine/macos.yaml`` for a generic MacOS system, where ``$SRW`` is the path to the ``ufs-srweather-app`` directory). The subset of shapefiles required for the plotting task can be obtained from the `SRW Data Bucket `__. The full set of medium-scale (1:50m) Cartopy shapefiles can be downloaded `here `__. Task Configuration ^^^^^^^^^^^^^^^^^^^^^