Update Develop-ref after #1908 and #1914 (#1921)

Co-authored-by: Kathryn Newman <knewman@seneca.rap.ucar.edu> Co-authored-by: j-opatz <59586397+j-opatz@users.noreply.github.com> Co-authored-by: Lisa Goodrich <lisag@ucar.edu> Co-authored-by: Julie Prestopnik <jpresto@seneca.rap.ucar.edu> Co-authored-by: johnhg <johnhg@ucar.edu> Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> Co-authored-by: Mrinal Biswas <biswas@ucar.edu> Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> Co-authored-by: Christina Kalb <kalb@mohawk.rap.ucar.edu> Co-authored-by: Hank Fisher <fisherh@seneca.rap.ucar.edu> Co-authored-by: jprestop <jpresto@ucar.edu> Co-authored-by: cristianastan2 <cstan@gmu.edu> Co-authored-by: John Halley Gotway <johnhg@ucar.edu> Co-authored-by: bikegeek <minnawin@ucar.edu> Co-authored-by: Hank Fisher <fisherh@ucar.edu> Co-authored-by: j-opatz <jopatz@ucar.edu> Co-authored-by: Christina Kalb <kalb@ucar.edu> Co-authored-by: lisagoodrich <33230218+lisagoodrich@users.noreply.github.com> Co-authored-by: Howard Soh <hsoh@seneca.rap.ucar.edu> Co-authored-by: Molly Smith <molly.b.smith@noaa.gov> Co-authored-by: hsoh-u <hsoh@ucar.edu> Co-authored-by: bikegeek <3753118+bikegeek@users.noreply.github.com>
dtcenter · Nov 7, 2022 · b468467 · b468467
1 parent 9840da5
commit b468467
Show file tree

Hide file tree

Showing 140 changed files with 4,049 additions and 3,830 deletions.
diff --git a/.github/actions/run_tests/entrypoint.sh b/.github/actions/run_tests/entrypoint.sh
@@ -61,7 +61,7 @@ if [[ "$INPUT_CATEGORIES" == pytests* ]]; then
   for x in `cat $PYTESTS_GROUPS_FILEPATH`; do
     marker="${x//_or_/ or }"
     marker="${marker//not_/not }"
-    command+="/usr/local/envs/pytest/bin/pytest -vv --cov=../../metplus -m \"$marker\""
+    command+="/usr/local/envs/pytest/bin/pytest -vv --cov=../../../metplus -m \"$marker\""
     command+=";if [ \$? != 0 ]; then status=1; fi;"
   done
   command+="if [ \$status != 0 ]; then echo ERROR: Some pytests failed. Search for FAILED to review; false; fi"

diff --git a/.github/jobs/create_output_data_volumes.sh b/.github/jobs/create_output_data_volumes.sh
@@ -28,7 +28,7 @@ if [ $? != 0 ]; then
   exit 1
 fi
 
-docker_data_output_dir=scripts/docker/docker_data_output
+docker_data_output_dir=.github/jobs/docker_data_output
 
 success=1
 for vol_name in use_cases_*; do

diff --git a/scripts/docker/docker_data_output/Dockerfile → .github/jobs/docker_data_output/Dockerfile b/scripts/docker/docker_data_output/Dockerfile → .github/jobs/docker_data_output/Dockerfile
diff --git a/.github/jobs/docker_setup.sh b/.github/jobs/docker_setup.sh
@@ -29,9 +29,9 @@ duration=$(( SECONDS - start_seconds ))
 echo "TIMING: docker pull ${DOCKERHUB_TAG} took `printf '%02d' $(($duration / 60))`:`printf '%02d' $(($duration % 60))` (MM:SS)"
 
 # set DOCKERFILE_PATH that is used by docker hook script get_met_version
-export DOCKERFILE_PATH=${GITHUB_WORKSPACE}/scripts/docker/Dockerfile
+export DOCKERFILE_PATH=${GITHUB_WORKSPACE}/internal/scripts/docker/Dockerfile
 
-MET_TAG=`${GITHUB_WORKSPACE}/scripts/docker/hooks/get_met_version`
+MET_TAG=`${GITHUB_WORKSPACE}/internal/scripts/docker/hooks/get_met_version`
 
 MET_DOCKER_REPO=met-dev
 if [ "${MET_TAG}" != "develop" ]; then

diff --git a/.github/jobs/docker_update_data_volumes.py b/.github/jobs/docker_update_data_volumes.py
diff --git a/.github/jobs/get_use_case_commands.py b/.github/jobs/get_use_case_commands.py
@@ -14,7 +14,7 @@
 sys.path.insert(0, METPLUS_TOP_DIR)
 
 from internal.tests.use_cases.metplus_use_case_suite import METplusUseCaseSuite
-from metplus.util.met_util import expand_int_string_to_list
+from metplus.util.string_manip import expand_int_string_to_list
 from docker_utils import VERSION_EXT
 
 

diff --git a/.github/parm/pytest_groups.txt b/.github/parm/pytest_groups.txt
@@ -1,3 +1,4 @@
+run_metplus
 util
 wrapper
 wrapper_a

diff --git a/.github/parm/use_case_groups.json b/.github/parm/use_case_groups.json
@@ -74,6 +74,11 @@
     "index_list": "6",
     "run": false
   },
+  {
+    "category": "marine_and_cryosphere",
+    "index_list": "7",
+    "run": false
+  },
   {
     "category": "medium_range",
     "index_list": "0",
@@ -134,6 +139,11 @@
     "index_list": "8",
     "run": false
   },
+  {
+    "category": "precipitation",
+    "index_list": "9",
+    "run": false
+  },
   {
     "category": "s2s",
     "index_list": "0",

diff --git a/docs/Contributors_Guide/continuous_integration.rst b/docs/Contributors_Guide/continuous_integration.rst
@@ -685,7 +685,7 @@ images using Conda that can be used to run use cases. These images are stored
 on DockerHub in *dtcenter/metplus-envs* and are named with a tag that
 corresponds to the keyword without the **_env** suffix.
 The environments were created using Docker commands via scripts that are found
-in *scripts/docker/docker_env*.
+in *internal/scripts/docker_env*.
 Existing keywords that set up Conda environments used for use cases are:
 
 * cfgrib_env
@@ -793,14 +793,14 @@ packages take a very long time to install in Docker. The new approach involves
 creating Docker images that use Conda to create a Python environment that can
 run the use case. To see what is available in each of the existing Python
 environments, refer to the comments in the scripts found in
-*scripts/docker/docker_env/scripts*.
+*internal/scripts/docker_env/scripts*.
 New environments must be added by a METplus developer,
 so please create a discussion on the
 `METplus GitHub Discussions <https://met.readthedocs.io/en/latest/Users_Guide/appendixF.html>`_
 forum if none of these environments contain the package requirements
 needed to run a new use case.
 
-A **README.md** file can be found in *scripts/docker/docker_env* that
+A **README.md** file can be found in *internal/scripts/docker_env* that
 provides commands that can be run to recreate a Docker image if the
 conda environment needs to be updated. Please note that Docker must
 be installed on the workstation used to create new Docker images and

diff --git a/docs/Contributors_Guide/deprecation.rst b/docs/Contributors_Guide/deprecation.rst
@@ -26,7 +26,7 @@ wrong variable and it is using WGRIB2 = wgrib2.
 
 check_for_deprecated_config()
 -----------------------------
-In **met_util.py** there is a function called
+In **metplus/util/config_metplus.py** there is a function called
 check_for_deprecated_config. It contains a dictionary of dictionaries
 called deprecated_dict that specifies the old config name, the section
 it was found in, and a suggested alternative (None if no alternative

diff --git a/docs/Contributors_Guide/user_support.rst b/docs/Contributors_Guide/user_support.rst
@@ -14,6 +14,10 @@ Support Responsibilities
 ========================
 
 Five staff members take turns monitoring Discussions each day of the week.
+
+Each day, all support members should follow up on existing Discussions in
+which they are assigned (i.e. tagged).
+
 The responsibilities for each daily assignee are described below.
 
 

diff --git a/docs/Users_Guide/getting_started.rst b/docs/Users_Guide/getting_started.rst
@@ -327,7 +327,7 @@ user configuration file and
 
 The last line of the screen output should match this format::
 
-    05/04 09:42:52.277 metplus (met_util.py:212) INFO: METplus has successfully finished running.
+    05/04 09:42:52.277 metplus INFO: METplus has successfully finished running.
 
 If this log message is not shown, there is likely an issue with one or more
 of the default configuration variable overrides in the
@@ -339,7 +339,7 @@ how the :ref:`common_config_variables` control a use case run.
 If the run was successful, the line above the success message should contain
 the path to the METplus log file that was generated::
 
-    05/04 09:44:21.534 metplus (met_util.py:211) INFO: Check the log file for more information: /path/to/output/logs/metplus.log.20210504094421
+    05/04 09:44:21.534 metplus INFO: Check the log file for more information: /path/to/output/logs/metplus.log.20210504094421
 
 * Review the log file and compare it to the Example.conf use case
   configuration file to see how the settings correspond to the result.

diff --git a/docs/Users_Guide/installation.rst b/docs/Users_Guide/installation.rst
@@ -221,16 +221,11 @@ The METplus Wrappers source code contains the following directory structure::
   METplus/
     build_components/
     docs/
-    environment.yml
     internal/
     manage_exernals/
     metplus/
     parm/
     produtil/
-    README.md
-    requirements.txt
-    scripts/
-    setup.py
     ush/
 
 The top-level METplus Wrappers directory consists of a README.md file
@@ -249,7 +244,8 @@ The Doxygen documentation is useful to
 contributors and is not necessary for METplus end-users.
 
 The **internal/** directory contains scripts that are only
-relevant to METplus developers and contributors.
+relevant to METplus developers and contributors, such as tests and files
+used with Docker.
 
 The **manage_externals/** directory contains scripts used to
 facilitate the downloading and management
@@ -262,9 +258,6 @@ METplus Wrappers.
 
 The **produtil/** directory contains part of the external utility produtil.
 
-The **scripts/** directory contains scripts that are used for creating
-Docker images.
-
 The **ush/** directory contains the run_metplus.py script that is
 executed to run use cases.
 

diff --git a/docs/Users_Guide/release-notes.rst b/docs/Users_Guide/release-notes.rst
@@ -30,6 +30,63 @@ When applicable, release notes are followed by the `GitHub issue <https://github
 describes the bugfix, enhancement, or new feature.
 
 
+METplus Version 5.0.0 Beta 4 Release Notes (2022-10-26)
+-------------------------------------------------------
+
+.. warning:: **MAJOR CHANGES**:
+
+  * The LOOP_ORDER config variable was removed. The variable set in a user's
+    config file will be ignored in favor of executing the logic that
+    corresponds to *LOOP_ORDER = processes*, where all times are processed for
+    the first item in the PROCESS_LIST, then all times are processed for the
+    second item in the PROCESS_LIST, etc. This may change the order that
+    commands are executed in a use case, but it should not change the results.
+
+
+* Enhancements:
+
+  * Remove LOOP_ORDER config variable (`#1687 <https://github.com/dtcenter/METplus/issues/1687>`_)
+
+  * Add unique identifier for each METplus run to configuration (`#1829 <https://github.com/dtcenter/METplus/issues/1829>`_)
+
+  * StatAnalysis - Support setting multiple jobs (`#1842 <https://github.com/dtcenter/METplus/issues/1842>`_)
+
+  * StatAnalysis - Set MET verbosity (`#1772 <https://github.com/dtcenter/METplus/issues/1772>`_)
+
+  * StatAnalysis - Support using both init/valid variables in string substitution (`#1861 <https://github.com/dtcenter/METplus/issues/1861>`_)
+
+  * StatAnalysis - Allow filename template tags in jobs (`#1862 <https://github.com/dtcenter/METplus/issues/1862>`_)
+
+  * StatAnalysis - Support looping over groups of list items (`#1870 <https://github.com/dtcenter/METplus/issues/1870>`_)
+
+  * StatAnalysis - Allow processing of time ranges other than daily (`#1871 <https://github.com/dtcenter/METplus/issues/1871>`_)
+
+  * StatAnalysis - Add support for using a custom loop list (`#1893 <https://github.com/dtcenter/METplus/issues/1893>`_)
+
+  * Remove MakePlots wrapper (`#1843 <https://github.com/dtcenter/METplus/issues/1843>`_)
+
+* Bugfixes:
+
+  * PCPCombine - custom loop list does not work for subtract method (`#1884 <https://github.com/dtcenter/METplus/issues/1884>`_)
+
+* New Wrappers: None
+
+* New Use Cases:
+
+  * Probability of Exceedence for 85th percentile temperatures (`#1808 <https://github.com/dtcenter/METplus/issues/1808>`_)
+
+  * FV3 Physics Tendency plotting via METplotpy (`#1852 <https://github.com/dtcenter/METplus/issues/1852>`_)
+
+* Documentation: None
+
+* Internal:
+
+  * Fix GitHub Actions warnings - update the version of actions and replace set-output (`#1863 <https://github.com/dtcenter/METplus/issues/1863>`_)
+
+  * Update diff logic to handle CSV files that have rounding differences (`#1865 <https://github.com/dtcenter/METplus/issues/1865>`_)
+
+  * Add unit tests for expected failure (`dtcenter/METplus-Internal#24 <https://github.com/dtcenter/METplus-Internal/issues/24>`_)
+
 METplus Version 5.0.0 Beta 3 Release Notes (2022-09-21)
 -------------------------------------------------------
 
@@ -67,8 +124,7 @@ METplus Version 5.0.0 Beta 3 Release Notes (2022-09-21)
   * MJO-ENSO diagnostics (`#1330 <https://github.com/dtcenter/METplus/issues/1330>`_)
 
 
-* Documentation:
-
+* Documentation: None
 
 * Internal: