Update Develop-ref after #1820, #1823, and dtcenter/MET#2265 (#1824)

Co-authored-by: Hank Fisher <fisherh@seneca.rap.ucar.edu> Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> Co-authored-by: Mrinal Biswas <biswas@seneca.rap.ucar.edu> Co-authored-by: jprestop <jpresto@ucar.edu> Co-authored-by: johnhg <johnhg@ucar.edu> Co-authored-by: Mrinal Biswas <biswas@ucar.edu> Co-authored-by: Julie Prestopnik <jpresto@seneca.rap.ucar.edu> Co-authored-by: j-opatz <jopatz@ucar.edu> Co-authored-by: Kathryn Newman <knewman@seneca.rap.ucar.edu> Co-authored-by: j-opatz <59586397+j-opatz@users.noreply.github.com> Co-authored-by: Lisa Goodrich <lisag@ucar.edu> Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> Co-authored-by: Christina Kalb <kalb@mohawk.rap.ucar.edu> Co-authored-by: cristianastan2 <cstan@gmu.edu> Co-authored-by: John Halley Gotway <johnhg@ucar.edu> Co-authored-by: Hank Fisher <fisherh@ucar.edu> Co-authored-by: mrinalbiswas <biswas@ucar.edu> Co-authored-by: bikegeek <3753118+bikegeek@users.noreply.github.com> Co-authored-by: lisagoodrich <33230218+lisagoodrich@users.noreply.github.com> Co-authored-by: Christina Kalb <kalb@ucar.edu> Co-authored-by: Howard Soh <hsoh@seneca.rap.ucar.edu> Co-authored-by: Molly Smith <molly.b.smith@noaa.gov> Co-authored-by: hsoh-u <hsoh@ucar.edu> Co-authored-by: bikegeek <minnawin@ucar.edu>
dtcenter · Sep 21, 2022 · 0c52df6 · 0c52df6
1 parent bbaea06
commit 0c52df6
Show file tree

Hide file tree

Showing 388 changed files with 693 additions and 5,200 deletions.
diff --git a/.github/actions/run_tests/entrypoint.sh b/.github/actions/run_tests/entrypoint.sh
@@ -56,7 +56,7 @@ if [[ "$INPUT_CATEGORIES" == pytests* ]]; then
 	 .
 
   echo Running Pytests
-  command="export METPLUS_PYTEST_HOST=docker; cd internal_tests/pytests;"
+  command="export METPLUS_PYTEST_HOST=docker; cd internal/tests/pytests;"
   command+="status=0;"
   for x in `cat $PYTESTS_GROUPS_FILEPATH`; do
     marker="${x//_or_/ or }"

diff --git a/.github/jobs/get_requirements.py b/.github/jobs/get_requirements.py
@@ -14,7 +14,7 @@
                                                os.pardir))
 sys.path.insert(0, METPLUS_TOP_DIR)
 
-from internal_tests.use_cases.metplus_use_case_suite import METplusUseCaseSuite
+from internal.tests.use_cases.metplus_use_case_suite import METplusUseCaseSuite
 
 def main():
     all_requirements = set()

diff --git a/.github/jobs/get_use_case_commands.py b/.github/jobs/get_use_case_commands.py
@@ -13,7 +13,7 @@
                                                os.pardir))
 sys.path.insert(0, METPLUS_TOP_DIR)
 
-from internal_tests.use_cases.metplus_use_case_suite import METplusUseCaseSuite
+from internal.tests.use_cases.metplus_use_case_suite import METplusUseCaseSuite
 from metplus.util.met_util import expand_int_string_to_list
 from docker_utils import VERSION_EXT
 

diff --git a/.github/parm/use_case_groups.json b/.github/parm/use_case_groups.json
@@ -117,7 +117,7 @@
   {
     "category": "precipitation",
     "index_list": "3-7",
-    "run": false
+    "run": true
   },
   {
     "category": "precipitation",

diff --git a/.github/workflows/testing.yml b/.github/workflows/testing.yml
@@ -85,7 +85,7 @@ jobs:
           DOCKER_USERNAME: ${{ secrets.DOCKER_USERNAME }}
           DOCKER_PASSWORD: ${{ secrets.DOCKER_PASSWORD }}
           EXTERNAL_TRIGGER: ${{ needs.job_control.outputs.external_trigger }}
-          #MET_FORCE_TAG: 10.0.0
+          #MET_FORCE_TAG: 10.1.0
 
   update_data_volumes:
     name: Docker Setup - Update Data Volumes
@@ -109,7 +109,7 @@ jobs:
     name: Use Case Tests
     runs-on: ubuntu-latest
     needs: [get_image, update_data_volumes, job_control]
-    if: ${{ always() && needs.job_control.outputs.run_some_tests == 'true' }}
+    if: ${{ needs.job_control.outputs.run_some_tests == 'true' }}
     strategy:
       fail-fast: false
       matrix: ${{fromJson(needs.job_control.outputs.matrix)}}

diff --git a/docs/Contributors_Guide/add_use_case.rst b/docs/Contributors_Guide/add_use_case.rst
@@ -680,7 +680,7 @@ will be used in the final pull request.
 Add use case to the test suite
 ------------------------------
 
-The *internal_tests/use_cases/all_use_cases.txt* file in the METplus
+The *internal/tests/use_cases/all_use_cases.txt* file in the METplus
 repository contains the list of all use cases.
 Add the new use case to this file so it will be available in
 the tests. See the :ref:`cg-ci-all-use-cases` section for details.
@@ -848,7 +848,7 @@ steps were unsuccessful in lowering memory usage, please take the following step
   Utilize a Python memory profiler to identify as specifically as possible 
   where the script exceeds the memory limit.
 - Add the use case to the :ref:`memory-intense-use-cases` list.
-- In the *internal_tests/use_cases/all_use_cases.txt* file, ensure that the 
+- In the *internal/tests/use_cases/all_use_cases.txt* file, ensure that the 
   use case is listed as the lowest-listed use case in its respective category. 
   Change the number in front of the new use case to an 'X', preceded 
   by the ‘#’ character::

diff --git a/docs/Contributors_Guide/continuous_integration.rst b/docs/Contributors_Guide/continuous_integration.rst
@@ -601,7 +601,7 @@ All Use Cases
 ^^^^^^^^^^^^^
 
 All of the existing use cases are listed in **all_use_cases.txt**,
-found in *internal_tests/use_cases*.
+found in *internal/tests/use_cases*.
 
 The file is organized by use case category. Each category starts
 a line that following the format::

diff --git a/docs/Contributors_Guide/testing.rst b/docs/Contributors_Guide/testing.rst
@@ -1,7 +1,7 @@
 Testing
 =======
 
-Test scripts are found in the GitHub repository in the internal_tests
+Test scripts are found in the GitHub repository in the internal/tests
 directory.
 
 .. _cg-unit-tests:
@@ -13,15 +13,15 @@ Unit tests are run with pytest. They are found in the *pytests* directory.
 Each tool has its own subdirectory containing its test files.
 
 Unit tests can be run by running the 'pytest' command from the
-internal_tests/pytests directory of the repository.
+internal/tests/pytests directory of the repository.
 The 'pytest' Python package must be available.
 A report will be output showing which pytest categories failed.
 When running on a new computer, a **minimum_pytest.<HOST>.sh**
 file must be created to be able to run the script. This file contains
 information about the local environment so that the tests can run.
 
 All unit tests must include one of the custom markers listed in the
-internal_tests/pytests/pytest.ini file. Some examples include:
+internal/tests/pytests/pytest.ini file. Some examples include:
 
     * util
     * wrapper_a

diff --git a/docs/Release_Guide/release_steps/metplus/update_web_server_data.rst b/docs/Release_Guide/release_steps/metplus/update_web_server_data.rst
@@ -7,7 +7,7 @@ Create Directory for Next Release
 On the DTC web server where the sample input data for use cases is hosted,
 run the setup_next_release_data.py script for the next upcoming release
 to set up the data directory for the next major/minor version development.
-The script can be found in the METplus repository in internal_tests/use_cases.
+The script can be found in the METplus repository in internal/tests/use_cases.
 The file should be found in the home directory of the met_test user on
 the DTC web server host. It is linked to the file in the METplus repository.
 Pull the latest changes from the develop branch before running the script::

diff --git a/docs/Users_Guide/glossary.rst b/docs/Users_Guide/glossary.rst
@@ -984,12 +984,18 @@ METplus Configuration Glossary
      | *Used by:*  EnsembleStat
 
    ENSEMBLE_STAT_ENS_VLD_THRESH
+     .. warning:: **DEPRECATED:** Please use :term:`ENSEMBLE_STAT_VLD_THRESH` instead.
+
+   ENSEMBLE_STAT_VLD_THRESH
      Threshold for the ratio of the number of valid data values to the total number of expected ensemble members. This value is passed into the ensemble_stat config file to make sure the percentage of files that are valid meets the expectation.
 
      | *Used by:*  EnsembleStat
 
    ENSEMBLE_STAT_ENS_OBS_THRESH
-     Sets the ens.obs_thresh value in the ensemble_stat MET config file.
+     .. warning:: **DEPRECATED:** Please use :term:`ENSEMBLE_STAT_OBS_THRESH`.
+
+   ENSEMBLE_STAT_OBS_THRESH
+     Sets the obs_thresh value in the ensemble_stat MET config file.
 
      | *Used by:*  EnsembleStat
 
@@ -5586,19 +5592,13 @@ METplus Configuration Glossary
      | *Used by:*  EnsembleStat
 
    ENSEMBLE_STAT_NBRHD_PROB_WIDTH
-     Specify the value for 'nbrhd_prob.width' in the MET configuration file for EnsembleStat.
-
-     | *Used by:*  EnsembleStat
+          .. warning:: **DEPRECATED:** Please use :term:`GEN_ENS_PROD_NBRHD_PROB_WIDTH` in :ref:`gen_ens_prod_wrapper` instead.
 
    ENSEMBLE_STAT_NBRHD_PROB_SHAPE
-     Specify the value for 'nbrhd_prob.shape' in the MET configuration file for EnsembleStat.
-
-     | *Used by:*  EnsembleStat
+          .. warning:: **DEPRECATED:** Please use :term:`GEN_ENS_PROD_NBRHD_PROB_SHAPE` in :ref:`gen_ens_prod_wrapper` instead.
 
    ENSEMBLE_STAT_NBRHD_PROB_VLD_THRESH
-     Specify the value for 'nbrhd_prob.vld_thresh' in the MET configuration file for EnsembleStat.
-
-     | *Used by:*  EnsembleStat
+          .. warning:: **DEPRECATED:** Please use :term:`GEN_ENS_PROD_NBRHD_PROB_VLD_THRESH` in :ref:`gen_ens_prod_wrapper` instead.
 
    ENSEMBLE_STAT_CLIMO_CDF_CDF_BINS
      See :term:`ENSEMBLE_STAT_CLIMO_CDF_BINS`
@@ -5634,34 +5634,22 @@ METplus Configuration Glossary
      | *Used by:*  EnsembleStat
 
    ENSEMBLE_STAT_NMEP_SMOOTH_GAUSSIAN_DX
-     Specify the value for 'nmep_smooth.gaussian_dx' in the MET configuration file for EnsembleStat.
-
-     | *Used by:*  EnsembleStat
+          .. warning:: **DEPRECATED:** Please use :term:`GEN_ENS_PROD_NMEP_SMOOTH_GAUSSIAN_DX` in :ref:`gen_ens_prod_wrapper` instead.
 
    ENSEMBLE_STAT_NMEP_SMOOTH_GAUSSIAN_RADIUS
-     Specify the value for 'nmep_smooth.gaussian_radius' in the MET configuration file for EnsembleStat.
-
-     | *Used by:*  EnsembleStat
+          .. warning:: **DEPRECATED:** Please use :term:`GEN_ENS_PROD_NMEP_SMOOTH_GAUSSIAN_RADIUS` in :ref:`gen_ens_prod_wrapper` instead.
 
    ENSEMBLE_STAT_NMEP_SMOOTH_VLD_THRESH
-     Specify the value for 'nmep_smooth.vld_thresh' in the MET configuration file for EnsembleStat.
-
-     | *Used by:*  EnsembleStat
+          .. warning:: **DEPRECATED:** Please use :term:`GEN_ENS_PROD_NMEP_SMOOTH_VLD_THRESH` in :ref:`gen_ens_prod_wrapper` instead.
 
    ENSEMBLE_STAT_NMEP_SMOOTH_SHAPE
-     Specify the value for 'nmep_smooth.shape' in the MET configuration file for EnsembleStat.
-
-     | *Used by:*  EnsembleStat
+          .. warning:: **DEPRECATED:** Please use :term:`GEN_ENS_PROD_NMEP_SMOOTH_SHAPE` in :ref:`gen_ens_prod_wrapper` instead.
 
    ENSEMBLE_STAT_NMEP_SMOOTH_METHOD
-     Specify the value for 'nmep_smooth.type.method' in the MET configuration file for EnsembleStat.
-
-     | *Used by:*  EnsembleStat
+          .. warning:: **DEPRECATED:** Please use :term:`GEN_ENS_PROD_NMEP_SMOOTH_METHOD` in :ref:`gen_ens_prod_wrapper` instead.
 
    ENSEMBLE_STAT_NMEP_SMOOTH_WIDTH
-     Specify the value for 'nmep_smooth.type.width' in the MET configuration file for EnsembleStat.
-
-     | *Used by:*  EnsembleStat
+          .. warning:: **DEPRECATED:** Please use :term:`GEN_ENS_PROD_NMEP_SMOOTH_WIDTH` in :ref:`gen_ens_prod_wrapper` instead.
 
    ENSEMBLE_STAT_CENSOR_THRESH
      Specify the value for 'censor_thresh' in the MET configuration file for EnsembleStat.
@@ -5925,74 +5913,46 @@ METplus Configuration Glossary
      | *Used by:* EnsembleStat
 
    ENSEMBLE_STAT_ENSEMBLE_FLAG_LATLON
-     Specify the value for 'ensemble_flag.latlon' in the MET configuration file for EnsembleStat.
-
-     | *Used by:*  EnsembleStat
+          .. warning:: **DEPRECATED:** Please use :term:`ENSEMBLE_STAT_NC_ORANK_FLAG_LATLON` or :term:`GEN_ENS_PROD_ENSEMBLE_FLAG_LATLON` in :ref:`gen_ens_prod_wrapper` instead.
 
    ENSEMBLE_STAT_ENSEMBLE_FLAG_MEAN
-     Specify the value for 'ensemble_flag.mean' in the MET configuration file for EnsembleStat.
-
-     | *Used by:*  EnsembleStat
+          .. warning:: **DEPRECATED:** Please use :term:`ENSEMBLE_STAT_NC_ORANK_FLAG_MEAN` or :term:`GEN_ENS_PROD_ENSEMBLE_FLAG_MEAN` in :ref:`gen_ens_prod_wrapper` instead.
 
    ENSEMBLE_STAT_ENSEMBLE_FLAG_STDEV
-     Specify the value for 'ensemble_flag.stdev' in the MET configuration file for EnsembleStat.
-
-     | *Used by:*  EnsembleStat
+          .. warning:: **DEPRECATED:** Please use :term:`GEN_ENS_PROD_ENSEMBLE_FLAG_STDEV` in :ref:`gen_ens_prod_wrapper` instead.
 
    ENSEMBLE_STAT_ENSEMBLE_FLAG_MINUS
-     Specify the value for 'ensemble_flag.minus' in the MET configuration file for EnsembleStat.
-
-     | *Used by:*  EnsembleStat
+          .. warning:: **DEPRECATED:** Please use :term:`GEN_ENS_PROD_ENSEMBLE_FLAG_MINUS` in :ref:`gen_ens_prod_wrapper` instead.
 
    ENSEMBLE_STAT_ENSEMBLE_FLAG_PLUS
-     Specify the value for 'ensemble_flag.plus' in the MET configuration file for EnsembleStat.
-
-     | *Used by:*  EnsembleStat
+          .. warning:: **DEPRECATED:** Please use :term:`GEN_ENS_PROD_ENSEMBLE_FLAG_PLUS` in :ref:`gen_ens_prod_wrapper` instead.
 
    ENSEMBLE_STAT_ENSEMBLE_FLAG_MIN
-     Specify the value for 'ensemble_flag.min' in the MET configuration file for EnsembleStat.
-
-     | *Used by:*  EnsembleStat
+          .. warning:: **DEPRECATED:** Please use :term:`GEN_ENS_PROD_ENSEMBLE_FLAG_MIN` in :ref:`gen_ens_prod_wrapper` instead.
 
    ENSEMBLE_STAT_ENSEMBLE_FLAG_MAX
-     Specify the value for 'ensemble_flag.max' in the MET configuration file for EnsembleStat.
-
-     | *Used by:*  EnsembleStat
+          .. warning:: **DEPRECATED:** Please use :term:`GEN_ENS_PROD_ENSEMBLE_FLAG_MAX` in :ref:`gen_ens_prod_wrapper` instead.
 
    ENSEMBLE_STAT_ENSEMBLE_FLAG_RANGE
-     Specify the value for 'ensemble_flag.range' in the MET configuration file for EnsembleStat.
-
-     | *Used by:*  EnsembleStat
+          .. warning:: **DEPRECATED:** Please use :term:`GEN_ENS_PROD_ENSEMBLE_FLAG_RANGE` in :ref:`gen_ens_prod_wrapper` instead.
 
    ENSEMBLE_STAT_ENSEMBLE_FLAG_VLD_COUNT
-     Specify the value for 'ensemble_flag.vld_count' in the MET configuration file for EnsembleStat.
-
-     | *Used by:*  EnsembleStat
+          .. warning:: **DEPRECATED:** Please use :term:`ENSEMBLE_STAT_NC_ORANK_FLAG_VLD_COUNT` or :term:`GEN_ENS_PROD_ENSEMBLE_FLAG_VLD_COUNT` in :ref:`gen_ens_prod_wrapper` instead.
 
    ENSEMBLE_STAT_ENSEMBLE_FLAG_FREQUENCY
-     Specify the value for 'ensemble_flag.frequency' in the MET configuration file for EnsembleStat.
-
-     | *Used by:*  EnsembleStat
+          .. warning:: **DEPRECATED:** Please use :term:`GEN_ENS_PROD_ENSEMBLE_FLAG_FREQUENCY` in :ref:`gen_ens_prod_wrapper` instead.
 
    ENSEMBLE_STAT_ENSEMBLE_FLAG_NEP
-     Specify the value for 'ensemble_flag.nep' in the MET configuration file for EnsembleStat.
-
-     | *Used by:*  EnsembleStat
+          .. warning:: **DEPRECATED:** Please use :term:`GEN_ENS_PROD_ENSEMBLE_FLAG_NEP` in :ref:`gen_ens_prod_wrapper` instead.
 
    ENSEMBLE_STAT_ENSEMBLE_FLAG_NMEP
-     Specify the value for 'ensemble_flag.nmep' in the MET configuration file for EnsembleStat.
-
-     | *Used by:*  EnsembleStat
+          .. warning:: **DEPRECATED:** Please use :term:`GEN_ENS_PROD_ENSEMBLE_FLAG_NMEP` in :ref:`gen_ens_prod_wrapper` instead.
 
    ENSEMBLE_STAT_ENSEMBLE_FLAG_RANK
-     Specify the value for 'ensemble_flag.rank' in the MET configuration file for EnsembleStat.
-
-     | *Used by:*  EnsembleStat
+          .. warning:: **DEPRECATED:** Please use :term:`ENSEMBLE_STAT_NC_ORANK_FLAG_RANK` instead.
 
    ENSEMBLE_STAT_ENSEMBLE_FLAG_WEIGHT
-     Specify the value for 'ensemble_flag.weight' in the MET configuration file for EnsembleStat.
-
-     | *Used by:*  EnsembleStat
+          .. warning:: **DEPRECATED:** Please use :term:`ENSEMBLE_STAT_NC_ORANK_FLAG_WEIGHT` instead.
 
    GRID_STAT_MASK_GRID
      Specify the value for 'mask.grid' in the MET configuration file for GridStat.
@@ -6358,9 +6318,12 @@ METplus Configuration Glossary
      | *Used by:*  TCpairs
 
    ENS_ENSEMBLE_STAT_INPUT_DATATYPE
-     Set the file_type entry of the ens dictionary in the MET config file for EnsembleStat.
+          .. warning:: **DEPRECATED:** Please use :term:`GEN_ENS_PROD_INPUT_DATATYPE` in :ref:`gen_ens_prod_wrapper` instead.
 
-     | *Used by:*  EnsembleStat
+   GEN_ENS_PROD_INPUT_DATATYPE
+     Set the file_type entry of the ens dictionary in the MET config file for GenEnsProd.
+
+     | *Used by:*  GenEnsProd
 
    FCST_SERIES_ANALYSIS_INPUT_DATATYPE
      Set the file_type entry of the fcst dictionary in the MET config file for SeriesAnalysis.
@@ -8709,8 +8672,8 @@ METplus Configuration Glossary
 
      | *Used by:*  GenEnsProd
 
-   GEN_ENS_PROD_ENSEMBLE_FLAG_CLIMO_CDF
-     Specify the value for 'ensemble_flag.climo_cdf' in the MET configuration file for GenEnsProd.
+   GEN_ENS_PROD_ENSEMBLE_FLAG_CLIMO_CDP
+     Specify the value for 'ensemble_flag.climo_cdp' in the MET configuration file for GenEnsProd.
 
      | *Used by:*  GenEnsProd
 
@@ -9224,7 +9187,7 @@ METplus Configuration Glossary
      it will automatically be removed by the wrapper to prevent an error in the
      MET tool. This may require adjusting the value for
      :term:`ENSEMBLE_STAT_N_MEMBERS` and/or
-     :term:`ENSEMBLE_STAT_ENS_VLD_THRESH`.
+     :term:`ENSEMBLE_STAT_VLD_THRESH`.
 
      | *Used by:*  EnsembleStat
 
@@ -9937,3 +9900,38 @@ METplus Configuration Glossary
      See :ref:`Overriding Unsupported MET config file settings<met-config-overrides>` for more information
 
      | *Used by:* PlotPointObs
+
+   ENSEMBLE_STAT_NC_ORANK_FLAG_LATLON
+     Specify the value for 'nc_orank_flag.latlon' in the MET configuration file for EnsembleStat.
+
+     | *Used by:* EnsembleStat
+
+   ENSEMBLE_STAT_NC_ORANK_FLAG_MEAN
+     Specify the value for 'nc_orank_flag.mean' in the MET configuration file for EnsembleStat.
+
+     | *Used by:* EnsembleStat
+
+   ENSEMBLE_STAT_NC_ORANK_FLAG_RAW
+     Specify the value for 'nc_orank_flag.raw' in the MET configuration file for EnsembleStat.
+
+     | *Used by:* EnsembleStat
+
+   ENSEMBLE_STAT_NC_ORANK_FLAG_RANK
+     Specify the value for 'nc_orank_flag.rank' in the MET configuration file for EnsembleStat.
+
+     | *Used by:* EnsembleStat
+
+   ENSEMBLE_STAT_NC_ORANK_FLAG_PIT
+     Specify the value for 'nc_orank_flag.pit' in the MET configuration file for EnsembleStat.
+
+     | *Used by:* EnsembleStat
+
+   ENSEMBLE_STAT_NC_ORANK_FLAG_VLD_COUNT
+     Specify the value for 'nc_orank_flag.vld_count' in the MET configuration file for EnsembleStat.
+
+     | *Used by:* EnsembleStat
+
+   ENSEMBLE_STAT_NC_ORANK_FLAG_WEIGHT
+     Specify the value for 'nc_orank_flag.weight' in the MET configuration file for EnsembleStat.
+
+     | *Used by:* EnsembleStat
diff --git a/docs/Users_Guide/installation.rst b/docs/Users_Guide/installation.rst
@@ -227,7 +227,7 @@ The METplus Wrappers source code contains the following directory structure::
     build_components/
     docs/
     environment.yml
-    internal_tests/
+    internal/
     manage_exernals/
     metplus/
     parm/
@@ -253,7 +253,7 @@ developer has Doxygen installed on the host.
 The Doxygen documentation is useful to
 contributors and is not necessary for METplus end-users.
 
-The **internal_tests/** directory contains test scripts that are only
+The **internal/** directory contains scripts that are only
 relevant to METplus developers and contributors.
 
 The **manage_externals/** directory contains scripts used to