Update develop-ref after #2248 and #2254 (#2255)

Co-authored-by: johnhg <johnhg@ucar.edu> Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> Co-authored-by: cristianastan2 <cstan@gmu.edu> Co-authored-by: John Halley Gotway <johnhg@ucar.edu> Co-authored-by: bikegeek <minnawin@ucar.edu> Co-authored-by: Lisa Goodrich <lisag@ucar.edu> Co-authored-by: Julie Prestopnik <jpresto@seneca.rap.ucar.edu> Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> Co-authored-by: Hank Fisher <fisherh@ucar.edu> Co-authored-by: Dan Adriaansen <dadriaan@ucar.edu> Co-authored-by: jprestop <jpresto@ucar.edu> Co-authored-by: Tracy Hertneky <hertneky@seneca.rap.ucar.edu> Co-authored-by: Giovanni Rosa <giovanni.rosa@unimol.it> Co-authored-by: j-opatz <59586397+j-opatz@users.noreply.github.com> Co-authored-by: Mrinal Biswas <biswas@seneca.rap.ucar.edu> Co-authored-by: j-opatz <jopatz@ucar.edu> Co-authored-by: Daniel Adriaansen <dadriaan@ucar.edu> Co-authored-by: Jonathan Vigh <jvigh@ucar.edu> Co-authored-by: bikegeek <3753118+bikegeek@users.noreply.github.com> Co-authored-by: Will Mayfield <59745143+willmayfield@users.noreply.github.com> Co-authored-by: lisagoodrich <33230218+lisagoodrich@users.noreply.github.com> Co-authored-by: metplus-bot <97135045+metplus-bot@users.noreply.github.com> Co-authored-by: Tracy Hertneky <39317287+hertneky@users.noreply.github.com> Co-authored-by: Giovanni Rosa <g.rosa1@studenti.unimol.it> Co-authored-by: mrinalbiswas <biswas@ucar.edu> Co-authored-by: Christina Kalb <kalb@ucar.edu> Co-authored-by: jason-english <73247785+jason-english@users.noreply.github.com> fix Contributor's Guide GitHub Workflow page (#1774) fix release (#1790) fix GitHub Actions warnings (#1864) fix #1884 develop PCPCombine {custom} in subtract method (#1887) fix #1939 develop - failure reading obs when zipped file also exists (#1941) Closes #1986 fix develop Fix broken documentation links (#2004) fix #2026 develop StatAnalysis looping (#2028) fix priority of obs_window config variables so that wrapper-specific version is preferred over generic OBS_WINDOW_BEGIN/END (#2062) fix #2070 var list numeric order (#2072) fix #2087 develop docs_pdf (#2091) fix #2096/#2098 develop - fix skip if output exists and do not error if no commands were run (#2099) Fix for Dockerfile smell DL4000 (#2112) fix #2082 develop regrid.convert/censor_thresh/censor_val (#2140) fix #2082 main_v5.0 regrid.convert/censor_thresh/censor_val (#2101) fix #2137 develop PointStat -obs_valid_beg/end (#2141) fix failured introduced by urllib3 (see urllib3/urllib3#2168) fix #2161 develop PCPCombine additional field arguments in -subtract mode (#2162) fix #2168 develop - StatAnalysis time shift (#2169) fix releases. (#2183) fix #2189 develop - spaces in complex thresholds (#2191) fix #2179 develop TCPairs fix -diag argument (#2187) fixes (#2200) fix diff tests (#2217) fix automated tests (#2237) fix #2235 rename multivar_itensity to multivar_intensity_flag (#2236) fix #2241 Create directory containing -out_stat file (#2242) fix #2245 use unique run ID to name logger instance (#2247) fix #2244 develop fix diff tests (#2254)
dtcenter · Jul 18, 2023 · 9225f15 · 9225f15
1 parent b3f5cf3
commit 9225f15
Show file tree

Hide file tree

Showing 56 changed files with 866 additions and 511 deletions.
diff --git a/.github/actions/run_tests/entrypoint.sh b/.github/actions/run_tests/entrypoint.sh
@@ -8,8 +8,6 @@ WS_PATH=$RUNNER_WORKSPACE/$REPO_NAME
 # set CI jobs directory variable to easily move it
 CI_JOBS_DIR=.github/jobs
 
-PYTESTS_GROUPS_FILEPATH=.github/parm/pytest_groups.txt
-
 source ${GITHUB_WORKSPACE}/${CI_JOBS_DIR}/bash_functions.sh
 
 # get branch name for push or pull request events
@@ -34,7 +32,7 @@ fi
 
 # running unit tests (pytests)
 if [[ "$INPUT_CATEGORIES" == pytests* ]]; then
-  export METPLUS_ENV_TAG="pytest.v5.1"
+  export METPLUS_ENV_TAG="test.v5.1"
   export METPLUS_IMG_TAG=${branch_name}
   echo METPLUS_ENV_TAG=${METPLUS_ENV_TAG}
   echo METPLUS_IMG_TAG=${METPLUS_IMG_TAG}
@@ -56,15 +54,9 @@ if [[ "$INPUT_CATEGORIES" == pytests* ]]; then
 	 .
 
   echo Running Pytests
-  command="export METPLUS_PYTEST_HOST=docker; cd internal/tests/pytests;"
-  command+="status=0;"
-  for x in `cat $PYTESTS_GROUPS_FILEPATH`; do
-    marker="${x//_or_/ or }"
-    marker="${marker//not_/not }"
-    command+="/usr/local/conda/envs/${METPLUS_ENV_TAG}/bin/pytest -vv --cov=../../../metplus --cov-append -m \"$marker\""
-    command+=";if [ \$? != 0 ]; then status=1; fi;"
-  done
-  command+="if [ \$status != 0 ]; then echo ERROR: Some pytests failed. Search for FAILED to review; false; fi"
+  command="export METPLUS_TEST_OUTPUT_BASE=/data/output;"
+  command+="/usr/local/conda/envs/${METPLUS_ENV_TAG}/bin/pytest internal/tests/pytests -vv --cov=metplus --cov-append --cov-report=term-missing;"
+  command+="if [ \$? != 0 ]; then echo ERROR: Some pytests failed. Search for FAILED to review; false; fi"
   time_command docker run -v $WS_PATH:$GITHUB_WORKSPACE --workdir $GITHUB_WORKSPACE $RUN_TAG bash -c "$command"
   exit $?
 fi

diff --git a/.github/jobs/run_diff_docker.py b/.github/jobs/run_diff_docker.py
@@ -24,6 +24,7 @@
 OUTPUT_DIR = '/data/output'
 DIFF_DIR = '/data/diff'
 
+
 def copy_diff_output(diff_files):
     """!  Loop through difference output and copy files
     to directory so it can be made available for comparison.
@@ -45,6 +46,7 @@ def copy_diff_output(diff_files):
             copy_to_diff_dir(diff_file,
                              'diff')
 
+
 def copy_to_diff_dir(file_path, data_type):
     """! Generate output path based on input file path,
     adding text based on data_type to the filename, then
@@ -85,6 +87,7 @@ def copy_to_diff_dir(file_path, data_type):
 
     return True
 
+
 def main():
     print('******************************')
     print("Comparing output to truth data")
@@ -97,5 +100,6 @@ def main():
     if diff_files:
         copy_diff_output(diff_files)
 
+
 if __name__ == '__main__':
     main()
diff --git a/.github/parm/pytest_groups.txt b/.github/parm/pytest_groups.txt
diff --git a/.github/parm/use_case_groups.json b/.github/parm/use_case_groups.json
@@ -1,7 +1,7 @@
 [
    {
     "category": "met_tool_wrapper",
-    "index_list": "0-29,59-61",
+    "index_list": "0-29,59-62",
     "run": false
   },
   {

diff --git a/docs/Contributors_Guide/add_use_case.rst b/docs/Contributors_Guide/add_use_case.rst
@@ -327,6 +327,12 @@ file.grib2, run the following command::
 
     wgrib2 file.grib2 | grep TMP | wgrib2 -i file.grib2 -grib_out subset.grib2
 
+The egrep command can be used for more complex subsetting of grib2 data.
+Example: To create a file called subset.grib2 from file.grib2 that contains
+PRMSL data and TMP data on 1000, 900, 800, 700, 500, and 100 mb levels::
+
+    wgrib2 file.grib2 -s | egrep '(:TMP:1000 mb:|:TMP:900 mb:|:TMP:800 mb:|:TMP:700 mb:|:TMP:500 mb:|:TMP:100 mb:|:PRMSL)' | wgrib2 -i file.grib2 -grib subset.grib2
+
 If the input data is in NetCDF format, the
 `ncks <http://nco.sourceforge.net/nco.html>`_ tool can be used to subset
 the file(s).

diff --git a/docs/Contributors_Guide/testing.rst b/docs/Contributors_Guide/testing.rst
@@ -9,24 +9,73 @@ directory.
 Unit Tests
 ----------
 
-Unit tests are run with pytest. They are found in the *pytests* directory.
+Unit tests are run with pytest.
+They are found in the *internal/tests/pytests* directory under the *wrappers*
+and *util* directories.
 Each tool has its own subdirectory containing its test files.
 
-Unit tests can be run by running the 'pytest' command from the
-internal/tests/pytests directory of the repository.
-The 'pytest' Python package must be available.
+Pytest Requirements
+^^^^^^^^^^^^^^^^^^^
+
+The following Python packages are required to run the tests.
+
+* **pytest**: Runs the tests
+* **python-dateutil**: Required to run METplus wrappers
+* **netCDF4**: Required for some METplus wrapper functionality
+* **pytest-cov** (optional): Only if generating code coverage stats
+* **pillow** (optional): Only used if running diff utility tests
+* **pdf2image** (optional): Only used if running diff utility tests
+
+Running
+^^^^^^^
+
+To run the unit tests, set the environment variable
+**METPLUS_TEST_OUTPUT_BASE** to a path where the user running has write
+permissions, nativate to the METplus directory, then call pytest::
+
+    export METPLUS_TEST_OUTPUT_BASE=/d1/personal/${USER}/pytest
+    cd METplus
+    pytest internal/tests/pytests
+
 A report will be output showing which pytest categories failed.
-When running on a new computer, a **minimum_pytest.<HOST>.sh**
-file must be created to be able to run the script. This file contains
-information about the local environment so that the tests can run.
+To view verbose test output, add the **-vv** argument::
+
+    pytest internal/tests/pytests -vv
+
+Code Coverage
+^^^^^^^^^^^^^
+
+If the *pytest-cov* package is installed, the code coverage report can
+be generated from the tests by running::
+
+    pytest internal/tests/pytests --cov=metplus --cov-report=term-missing
+
+In addition to the pass/fail report, the code coverage information will be
+displayed including line numbers that are not covered by any test.
 
-All unit tests must include one of the custom markers listed in the
+Subsetting Tests by Directory
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+A subset of the unit tests can be run by adjusting the path.
+Be sure to include the *--cov-append* argument so the results of the run
+are appended to the full code coverage results.
+To run only the GridStat unit tests::
+
+    pytest internal/tests/pytests/wrappers/grid_stat --cov=metplus --cov-report=term-missing --cov-append
+
+
+Subsetting Tests by Marker
+^^^^^^^^^^^^^^^^^^^^^^^^^^
+Unit tests can include one of the custom markers listed in the
 internal/tests/pytests/pytest.ini file. Some examples include:
 
+    * diff
+    * run_metplus
     * util
     * wrapper_a
     * wrapper_b
     * wrapper_c
+    * wrapper_d
     * wrapper
     * long
     * plotting
@@ -42,47 +91,52 @@ New pytest markers should be added to the pytest.ini file with a brief
 description. If they are not added to the markers list, then a warning will
 be output when running the tests.
 
-There are many unit tests for METplus and false failures can occur if all of
-the are attempted to run at once.
 To run only tests with a given marker, run::
 
-    pytest -m <MARKER-NAME>
+    pytest internal/tests/pytests -m <MARKER-NAME>
 
 To run all tests that do not have a given marker, run::
 
-    pytest -m "not <MARKER-NAME>"
+    pytest internal/tests/pytests -m "not <MARKER-NAME>"
+
+For example, **if you are running on a system that does not have the additional
+dependencies required to run the diff utility tests**, you can run all of the
+tests except those by running::
+
+    pytest internal/tests/pytests -m "not diff"
+
+Multiple marker groups can be run by using the *or* keyword::
+
+    pytest internal/tests/pytests -m "<MARKER-NAME1> or <MARKER-NAME2>"
+
+Writing Unit Tests
+^^^^^^^^^^^^^^^^^^
+
+metplus_config fixture
+""""""""""""""""""""""
 
-Multiple marker groups can be run by using the 'or' keyword::
+Many unit tests utilize a pytest fixture named **metplus_config**.
+This is defined in the **conftest.py** file in internal/tests/pytests.
+This is used to create a METplusConfig object that contains the minimum
+configurations needed to run METplus, like **OUTPUT_BASE**.
+Using this fixture in a pytest will initialize the METplusConfig object to use
+in the tests.
 
-    pytest -m "<MARKER-NAME1> or <MARKER-NAME2>"
+This also creates a unique output directory for each test where
+logs and output files are written. This directory is created under
+**$METPLUS_TEST_OUTPUT_BASE**/test_output and is named with the run ID.
+If the test passes, then the output directory is automatically removed.
+If the test fails, the output directory will not be removed so the content
+can be reviewed to debug the issue.
 
+To use it, add **metplus_config** as an argument to the test function::
 
-Use Case Tests
---------------
+    def test_something(metplus_config)
 
-Use case tests are run via a Python script called **test_use_cases.py**,
-found in the *use_cases* directory.
-Eventually the running of these tests will be automated using an external
-tool, such as GitHub Actions or Travis CI.
-The script contains a list of use cases that are found in the repository.
-For each computer that will run the use cases, a
-**metplus_test_env.<HOST>.sh** file must exist to set local configurations.
-All of the use cases can be run by executing the script
-**run_test_use_cases.sh**. The use case test script will output the results
-into a directory such as */d1/<USER>/test-use-case-b*, defined in the
-environment file.
-If */d1/<USER>/test-use-case-b* already exists, its content will be copied
-over to */d1/<USER>/test-use-case-a*. If data is found in
-the */d1/<USER>/test-use-case-b* directory  already exists, its content
-will be copied
-over to the */d1/<USER>/test-use-case-a* directory, the script will prompt
-the user to remove those files.
-Once the tests have finished running, the output found in the two
-directories can be compared to see what has changed. Suggested commands
-to run to compare the output will be shown on the screen after completion
-of the script.
+then set a variable called **config** using the fixture name::
 
-To see which files and directories are only found in one run::
+    config = metplus_config
 
-    diff -r /d1/mccabe/test-use-case-a /d1/mccabe/test-use-case-b | grep Only
+Additional configuration variables can be set by using the set method::
 
+    config.set('config', key, value)
diff --git a/docs/Users_Guide/quicksearch.rst b/docs/Users_Guide/quicksearch.rst
@@ -10,6 +10,7 @@ METplus Quick Search for Use Cases
 
 
 Use Cases by MET Tool:
+======================
 
 .. only:: html
 
@@ -163,6 +164,7 @@ Use Cases by METplus Feature:
    | `Runtime Frequency <../search.html?q=RuntimeFreqUseCase&check_keywords=yes&area=default>`_
    | `Series by Initialization <../search.html?q=SeriesByInitUseCase&check_keywords=yes&area=default>`_
    | `Series by Forecast Lead <../search.html?q=SeriesByLeadUseCase&check_keywords=yes&area=default>`_
+   | `Tropical Cyclone <../search.html?q=TropicalCycloneUseCase&check_keywords=yes&area=default>`_
    | `Validation of Models or Analyses <../search.html?q=ValidationUseCase&check_keywords=yes&area=default>`_
    | `User Defined Script <../search.html?q=UserScriptUseCase&check_keywords=yes&area=default>`_
 
@@ -191,6 +193,7 @@ Use Cases by METplus Feature:
    | **Runtime Frequency**: *RuntimeFreqUseCase*
    | **Series by Initialization**: *SeriesByInitUseCase*
    | **Series by Forecast Lead**: *SeriesByLeadUseCase*
+   | **Tropical Cyclone**: *TropicalCycloneUseCase*
    | **Validation of Models or Analyses**: *ValidationUseCase*
    | **User Defined Script**: *UserScriptUseCase*