Update Develop-ref after #1841 (#1851)

Co-authored-by: jprestop <jpresto@ucar.edu> Co-authored-by: johnhg <johnhg@ucar.edu> Co-authored-by: Mrinal Biswas <biswas@ucar.edu> Co-authored-by: Julie Prestopnik <jpresto@seneca.rap.ucar.edu> Co-authored-by: j-opatz <jopatz@ucar.edu> Co-authored-by: Kathryn Newman <knewman@seneca.rap.ucar.edu> Co-authored-by: j-opatz <59586397+j-opatz@users.noreply.github.com> Co-authored-by: Lisa Goodrich <lisag@ucar.edu> Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> Co-authored-by: Christina Kalb <kalb@mohawk.rap.ucar.edu> Co-authored-by: Hank Fisher <fisherh@seneca.rap.ucar.edu> Co-authored-by: cristianastan2 <cstan@gmu.edu> Co-authored-by: John Halley Gotway <johnhg@ucar.edu> Co-authored-by: bikegeek <minnawin@ucar.edu> Co-authored-by: bikegeek <3753118+bikegeek@users.noreply.github.com> Co-authored-by: lisagoodrich <33230218+lisagoodrich@users.noreply.github.com> Co-authored-by: Christina Kalb <kalb@ucar.edu> Co-authored-by: Howard Soh <hsoh@seneca.rap.ucar.edu> Co-authored-by: Molly Smith <molly.b.smith@noaa.gov> Co-authored-by: hsoh-u <hsoh@ucar.edu> Co-authored-by: Hank Fisher <fisherh@ucar.edu>
dtcenter · Oct 5, 2022 · 1847bc0 · 1847bc0
1 parent 0c52df6
commit 1847bc0
Show file tree

Hide file tree

Showing 34 changed files with 850 additions and 617 deletions.
diff --git a/.github/parm/use_case_groups.json b/.github/parm/use_case_groups.json
@@ -99,6 +99,11 @@
     "index_list": "8",
     "run": false
   },
+  {
+    "category": "medium_range",
+    "index_list": "9",
+    "run": false
+  },
   {
     "category": "precipitation",
     "index_list": "0",
@@ -117,7 +122,7 @@
   {
     "category": "precipitation",
     "index_list": "3-7",
-    "run": true
+    "run": false
   },
   {
     "category": "precipitation",

diff --git a/docs/Contributors_Guide/create_wrapper.rst b/docs/Contributors_Guide/create_wrapper.rst
@@ -22,7 +22,7 @@ For example, the new_tool wrapper would be named **NewToolWrapper**.
 Add Entry to LOWER_TO_WRAPPER_NAME Dictionary
 ---------------------------------------------
 
-In *metplus/util/doc_util.py*, add entries to the LOWER_TO_WRAPPER_NAME
+In *metplus/util/constants.py*, add entries to the LOWER_TO_WRAPPER_NAME
 dictionary so that the wrapper can be found in the PROCESS_LIST even if
 it is formatted differently. The key should be the wrapper name in all
 lower-case letters without any underscores. The value should be the class name

diff --git a/docs/Release_Guide/release_steps/met/update_version_development.rst b/docs/Release_Guide/release_steps/met/update_version_development.rst
@@ -3,7 +3,7 @@ Update Version Number
 
 * Prior to creating the first development release (e.g. beta1) for a new version, the official release version number should have already been updated.
 
-* If the official release version has not yet been updated (e.g. beta1), follow the official release instructions to do so.
+* If the official release version has not yet been updated (e.g. beta1), follow the :ref:`official release instructions <update_version_official>` to do so.
 
 * If the official release version has already been updated (e.g. beta2 and beyond), do the following:
 

diff --git a/docs/Release_Guide/release_steps/met/update_version_official.rst b/docs/Release_Guide/release_steps/met/update_version_official.rst
@@ -1,5 +1,6 @@
 Update Version Number
 ---------------------
+.. _update_version_official:
 
 * Update the version in the code and documentation:
 

diff --git a/docs/Users_Guide/glossary.rst b/docs/Users_Guide/glossary.rst
@@ -1620,12 +1620,12 @@ METplus Configuration Glossary
      .. warning:: **DEPRECATED:** Please use :term:`FCST_THRESH_LIST` instead.
 
    FCST_THRESH_LIST
-     Specify the values of the FCST_THRESH column in the MET .stat file to use. This is optional in the METplus configuration file for running with :term:`LOOP_ORDER` = times.
+     Specify the values of the FCST_THRESH column in the MET .stat file to use.
 
      | *Used by:*  StatAnalysis
 
    OBS_THRESH_LIST
-     Specify the values of the OBS_THRESH column in the MET .stat file to use. This is optional in the METplus configuration file for running with :term:`LOOP_ORDER` = times.
+     Specify the values of the OBS_THRESH column in the MET .stat file to use.
 
      | *Used by:*  StatAnalysis
 
@@ -1647,20 +1647,20 @@ METplus Configuration Glossary
      .. warning:: **DEPRECATED:** Please use :term:`FCST_LEVEL_LIST` instead.
 
    FCST_LEVEL_LIST
-     Specify the values of the FCST_LEV column in the MET .stat file to use. This is optional in the METplus configuration file for running with :term:`LOOP_ORDER` = times.
+     Specify the values of the FCST_LEV column in the MET .stat file to use.
 
      | *Used by:*  StatAnalysis
 
    FCST_VAR_NAME
      .. warning:: **DEPRECATED:** Please use :term:`FCST_VAR_LIST` instead.
 
    FCST_VAR_LIST
-     Specify the values of the FCST_VAR column in the MET .stat file to use. This is optional in the METplus configuration file for running with :term:`LOOP_ORDER` = times.
+     Specify the values of the FCST_VAR column in the MET .stat file to use.
 
      | *Used by:*  StatAnalysis
 
    FCST_UNITS_LIST
-     Specify the values of the FCST_UNITS column in the MET .stat file to use. This is optional in the METplus configuration file for running with :term:`LOOP_ORDER` = times.
+     Specify the values of the FCST_UNITS column in the MET .stat file to use.
 
      | *Used by:*  StatAnalysis
 
@@ -2319,7 +2319,7 @@ METplus Configuration Glossary
 
 
    LINE_TYPE_LIST
-     Specify the MET STAT line types to be considered. For TCMPRPlotter, this is optional in the METplus configuration file for running with :term:`LOOP_ORDER` = times.
+     Specify the MET STAT line types to be considered.
 
      | *Used by:*  MakePlots, StatAnalysis, TCMPRPlotter
 
@@ -2395,7 +2395,7 @@ METplus Configuration Glossary
      .. warning:: **DEPRECATED:** Please use :term:`LOOP_BY` instead.
 
    LOOP_ORDER
-     Control the looping order for METplus. Valid options are "times" or "processes". "times" runs all items in the :term:`PROCESS_LIST` for a single run time, then repeat until all times have been evaluated. "processes" runs each item in the :term:`PROCESS_LIST` for all times specified, then repeat for the next item in the :term:`PROCESS_LIST`.
+     .. warning:: **DEPRECATED:** This previously controlled the looping order for METplus. This was removed in v5.0.0. The wrappers will always execute the logic that was previously run when LOOP_ORDER = processes, which runs each item in the :term:`PROCESS_LIST` for all times specified, then repeat for the next item in the :term:`PROCESS_LIST`.
 
      | *Used by:*  All
 
@@ -3207,20 +3207,20 @@ METplus Configuration Glossary
      .. warning:: **DEPRECATED:** Please use :term:`OBS_LEVEL_LIST` instead.
 
    OBS_LEVEL_LIST
-     Specify the values of the OBS_LEV column in the MET .stat file to use. This is optional in the METplus configuration file for running with :term:`LOOP_ORDER` = times.
+     Specify the values of the OBS_LEV column in the MET .stat file to use.
 
      | *Used by:*  StatAnalysis
 
    OBS_VAR_NAME
      .. warning:: **DEPRECATED:** Please use :term:`OBS_VAR_LIST` instead.
 
    OBS_VAR_LIST
-     Specify the values of the OBS_VAR column in the MET .stat file to use. This is optional in the METplus configuration file for running with :term:`LOOP_ORDER` = times.
+     Specify the values of the OBS_VAR column in the MET .stat file to use.
 
      | *Used by:*  StatAnalysis
 
    OBS_UNITS_LIST
-     Specify the values of the OBS_UNITS column in the MET .stat file to use. This is optional in the METplus configuration file for running with :term:`LOOP_ORDER` = times.
+     Specify the values of the OBS_UNITS column in the MET .stat file to use.
 
      | *Used by:*  StatAnalysis
 
@@ -3837,7 +3837,7 @@ METplus Configuration Glossary
      .. warning:: **DEPRECATED:** Please use :term:`MODEL<n>_STAT_ANALYSIS_DUMP_ROW_TEMPLATE` instead.
 
    MODEL<n>_STAT_ANALYSIS_DUMP_ROW_TEMPLATE
-     Specify the template to use for the stat_analysis dump_row file. A user customized template to use for the dump_row file. If left blank and a dump_row file is requested, a default version will be used. This is optional in the METplus configuration file for running with :term:`LOOP_ORDER` = times.
+     Specify the template to use for the stat_analysis dump_row file. A user customized template to use for the dump_row file. If left blank and a dump_row file is requested, a default version will be used.
 
      | *Used by:*  StatAnalysis
 
@@ -3853,7 +3853,7 @@ METplus Configuration Glossary
      .. warning:: **DEPRECATED:** Please use :term:`MODEL<n>_STAT_ANALYSIS_OUT_STAT_TEMPLATE` instead.
 
    MODEL<n>_STAT_ANALYSIS_OUT_STAT_TEMPLATE
-     Specify the template to use for the stat_analysis out_stat file. A user customized template to use for the out_stat file. If left blank and a out_stat file is requested, a default version will be used. This is optional in the METplus configuration file for running with :term:`LOOP_ORDER` = times.
+     Specify the template to use for the stat_analysis out_stat file. A user customized template to use for the out_stat file. If left blank and a out_stat file is requested, a default version will be used.
 
      | *Used by:*  StatAnalysis
 
@@ -7636,12 +7636,11 @@ METplus Configuration Glossary
      | *Used by:*  UserScript
 
    TC_PAIRS_RUN_ONCE
-     If True and LOOP_ORDER = processes, TCPairs will be run once using the
+     If True, TCPairs will be run once using the
      INIT_BEG or VALID_BEG value (depending on the value of LOOP_BY).
      This is the default setting and preserves the original logic of the
      wrapper. If this variable is set to False, then TCPairs will run once
-     for each run time iteration. If LOOP_ORDER = times, then TCPairs will
-     still run for each run time. The preferred configuration settings to
+     for each run time iteration. The preferred configuration settings to
      run TCPairs once for a range of init or valid times is to set INIT_BEG
      to INIT_END (if LOOP_BY = INIT) and define the range of init times to
      filter the data inside TCPairs with TC_PAIRS_INIT_BEG and
@@ -9901,6 +9900,19 @@ METplus Configuration Glossary
 
      | *Used by:* PlotPointObs
 
+   RUN_ID
+     Eight character hash string unique to a given run of METplus.
+     Automatically set by METplus at the beginning of a run.
+     Can be referenced in other METplus config variables to distinguish
+     multiple METplus runs that may have started within the same second.
+     For example, it can be added to :term:`LOG_TIMESTAMP_TEMPLATE` to
+     create unique log files, final config files, etc.
+
+     Example:
+     LOG_TIMESTAMP_TEMPLATE = %Y%m%d%H%M%S.{RUN_ID}
+
+     | *Used by:*  All
+
    ENSEMBLE_STAT_NC_ORANK_FLAG_LATLON
      Specify the value for 'nc_orank_flag.latlon' in the MET configuration file for EnsembleStat.
 

diff --git a/docs/Users_Guide/release-notes.rst b/docs/Users_Guide/release-notes.rst
@@ -39,6 +39,53 @@ describes the bugfix, enhancement, or new feature:
 https://github.com/dtcenter/METplus/issues
 
 
+METplus Version 5.0.0 Beta 3 Release Notes (2022-09-21)
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+.. warning:: **MAJOR CHANGES**:
+
+  * Use cases that include **EnsembleStat** wrapper will require config file updates. **NEW USER'S GUIDE SECTION OUTLINING REQUIRED CHANGES COMING SOON**
+  * The default value of :term:`SCRUB_STAGING_DIR` is now *True*.
+    This means some intermediate files that are auto-generated by METplus such as file lists and
+    uncompressed files will automatically be removed unless this option is set by the user.
+    These files are typically only used to debug unexpected issues.
+  * The default value of :term:`METPLUS_CONF` now includes the :term:`LOG_TIMESTAMP` so each METplus run
+    will generate a unique final config file, e.g. metplus_final.conf.20220921121733.
+
+
+* Enhancements:
+
+  * **Update the EnsembleStat wrapper and use case examples to remove ensemble post processing logic** (`#1816 <https://github.com/dtcenter/METplus/issues/1816>`_)
+  * Enhance logic to consistently create directories (`#1657 <https://github.com/dtcenter/METplus/issues/1657>`_)
+  * Create checksum for released code (`#262 <https://github.com/dtcenter/METplus/issues/262>`_)
+  * Add the user ID to the log output at beginning and end of each METplus wrappers run (`dtcenter/METplus-Internal#20 <https://github.com/dtcenter/METplus-Internal/issues/20>`_)
+  * Update logic to name final conf and intermediate files with a unique identifier (`dtcenter/METplus-Internal#32 <https://github.com/dtcenter/METplus-Internal/issues/32>`_)
+  * Change default logging time information (`dtcenter/METplus-Internal#34 <https://github.com/dtcenter/METplus-Internal/issues/34>`_)
+
+* Bugfixes:
+
+  * Allow NA value for <TOOL-NAME>_CLIMO_[MEAN/STDEV]_HOUR_INTERVAL (`#1787 <https://github.com/dtcenter/METplus/issues/1787>`_)
+
+* New Wrappers: 
+
+  * PlotPointObs (`#1489 <https://github.com/dtcenter/METplus/issues/1489>`_)
+
+* New Use Cases: 
+
+  * PANDA-C use cases  (`#1686 <https://github.com/dtcenter/METplus/issues/1686>`_)
+  * MJO-ENSO diagnostics (`#1330 <https://github.com/dtcenter/METplus/issues/1330>`_)
+
+
+* Documentation:
+
+
+* Internal:
+
+  * Add instructions in Release Guide for "Recreate an Existing Release" (`#1746 <https://github.com/dtcenter/METplus/issues/1746>`_)
+  * Add modulefiles used for installations on various machines (`#1749 <https://github.com/dtcenter/METplus/issues/1749>`_)
+
+
+
 METplus Version 5.0.0 Beta 2 Release Notes (2022-08-03)
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 

diff --git a/docs/Users_Guide/systemconfiguration.rst b/docs/Users_Guide/systemconfiguration.rst
@@ -1078,38 +1078,14 @@ the output directory.
 Loop Order
 ----------
 
-The METplus wrappers can be configured to loop first by times then
-processes or vice-versa. Looping by times first will run each process in
-the process list for a given run time, increment to the next run time, run
-each process in the process list, and so on. Looping by processes first
-will run all times for the first process, then run all times for the
-second process, and so on.
+The METplus wrappers will run all times for the first process defined in the
+:term:`PROCESS_LIST`, then run all times for the second process, and so on.
+The :term:`LOOP_ORDER` variable has been deprecated in v5.0.0.
+This is the behavior that was previously executed when LOOP_ORDER = processes.
 
-**Example 1 Configuration**::
-
-  [config]
-  LOOP_ORDER = times
-
-  PROCESS_LIST = PCPCombine, GridStat
-
-  VALID_BEG = 20190201
-  VALID_END = 20190203
-  VALID_INCREMENT = 1d
-
-will run in the following order::
-
-  * PCPCombine at 2019-02-01
-  * GridStat   at 2019-02-01
-  * PCPCombine at 2019-02-02
-  * GridStat   at 2019-02-02
-  * PCPCombine at 2019-02-03
-  * GridStat   at 2019-02-03
-
-
-**Example 2 Configuration**::
+**Example Configuration**::
 
   [config]
-  LOOP_ORDER = processes
 
   PROCESS_LIST = PCPCombine, GridStat
 
@@ -1126,12 +1102,7 @@ will run in the following order::
   * GridStat   at 2019-02-02
   * GridStat   at 2019-02-03
 
-.. note::
-    If running a MET tool that processes data over a time range, such as
-    SeriesAnalysis or StatAnalysis, the tool must be run with
-    LOOP_ORDER = processes.
 
-
 .. _Custom_Looping:
 
 Custom Looping
@@ -1782,15 +1753,13 @@ The possible values for the \*_RUNTIME_FREQ variables are:
   (init or valid and forecast lead combination).
   All filename templates are substituted with values.
 
-Note that :term:`LOOP_ORDER` must be set to processes to run these wrappers.
-Also note that the following example may not contain all of the configuration
+Note that the following example may not contain all of the configuration
 variables that are required for a successful run. The are intended to show
 how these variables affect how the data is processed.
 
 **SeriesAnalysis Examples**::
 
     [config]
-    LOOP_ORDER = processes
 
     LOOP_BY = INIT
     INIT_TIME_FMT = %Y%m%d%H