Update develop-ref after #2402 (#2403)

Co-authored-by: Dan Adriaansen <dadriaan@ucar.edu>
Co-authored-by: johnhg <johnhg@ucar.edu>
Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com>
Co-authored-by: John Halley Gotway <johnhg@ucar.edu>
Co-authored-by: jprestop <jpresto@ucar.edu>
Co-authored-by: Tracy Hertneky <hertneky@seneca.rap.ucar.edu>
Co-authored-by: Giovanni Rosa <giovanni.rosa@unimol.it>
Co-authored-by: j-opatz <59586397+j-opatz@users.noreply.github.com>
Co-authored-by: Mrinal Biswas <biswas@seneca.rap.ucar.edu>
Co-authored-by: j-opatz <jopatz@ucar.edu>
Co-authored-by: Daniel Adriaansen <dadriaan@ucar.edu>
Co-authored-by: Jonathan Vigh <jvigh@ucar.edu>
Co-authored-by: root <root@localhost>
Co-authored-by: lisagoodrich <33230218+lisagoodrich@users.noreply.github.com>
Co-authored-by: Julie Prestopnik <jpresto@ucar.edu>
Co-authored-by: Tracy <tracy.hertneky@noaa.gov>
Co-authored-by: Hank Fisher <fisherh@ucar.edu>
Co-authored-by: bikegeek <3753118+bikegeek@users.noreply.github.com>
Co-authored-by: metplus-bot <97135045+metplus-bot@users.noreply.github.com>
Co-authored-by: Lisa Goodrich <lisag@ucar.edu>
Co-authored-by: Tracy Hertneky <39317287+hertneky@users.noreply.github.com>
Co-authored-by: Giovanni Rosa <g.rosa1@studenti.unimol.it>
Co-authored-by: mrinalbiswas <biswas@ucar.edu>
Co-authored-by: Christina Kalb <kalb@ucar.edu>
Co-authored-by: jason-english <73247785+jason-english@users.noreply.github.com>
Co-authored-by: John Sharples <41682323+John-Sharples@users.noreply.github.com>
Co-authored-by: reza-armuei <144857501+reza-armuei@users.noreply.github.com>
fix #1939 develop - failure reading obs when zipped file also exists (#1941)
Closes #1986
fix develop Fix broken documentation links (#2004)
fix #2026 develop StatAnalysis looping (#2028)
fix priority of obs_window config variables so that wrapper-specific version is preferred over generic OBS_WINDOW_BEGIN/END (#2062)
fix #2070 var list numeric order (#2072)
fix #2087 develop docs_pdf (#2091)
fix #2096/#2098 develop - fix skip if output exists and do not error if no commands were run (#2099)
Fix for Dockerfile smell DL4000 (#2112)
fix #2082 develop regrid.convert/censor_thresh/censor_val (#2140)
fix #2082 main_v5.0 regrid.convert/censor_thresh/censor_val (#2101)
fix #2137 develop PointStat -obs_valid_beg/end (#2141)
fix failured introduced by urllib3 (see urllib3/urllib3#2168)
fix #2161 develop PCPCombine additional field arguments in -subtract mode (#2162)
fix #2168 develop - StatAnalysis time shift (#2169)
fix releases. (#2183)
fix #2189 develop - spaces in complex thresholds (#2191)
fix #2179 develop TCPairs fix -diag argument (#2187)
fixes (#2200)
fix diff tests (#2217)
fix automated tests (#2237)
fix #2235 rename multivar_itensity to multivar_intensity_flag (#2236)
fix #2241 Create directory containing -out_stat file (#2242)
fix #2245 use unique run ID to name logger instance (#2247)
fix #2244 develop fix diff tests (#2254)
fixture to set pytest tmpdir (#2261)
fix #1853 develop - PointStat don't require mask variables to be set (#2262)
fix #2279 develop - buoy station file from 2022 (#2280)
fix (#2313)
fix ReadTheDocs requirements to include pillow which is a dependency of sphinx-gallery: see https://blog.readthedocs.com/defaulting-latest-build-tools/ for more info on why this was necessary

.coveragerc

@@ -1,3 +1,5 @@
 [run]
 relative_files = True
 source = metplus
+omit =
+  metplus/wrappers/cyclone_plotter_wrapper.py

.github/jobs/docker_utils.py

@@ -39,7 +39,7 @@ def get_dockerhub_url(branch_name):
 def docker_get_volumes_last_updated(current_branch):
     import requests
     dockerhub_url = get_dockerhub_url(current_branch)
-    dockerhub_request = requests.get(dockerhub_url)
+    dockerhub_request = requests.get(dockerhub_url, timeout=60)
     if dockerhub_request.status_code != 200:
         print(f"Could not find DockerHub URL: {dockerhub_url}")
         return None
@@ -61,7 +61,7 @@ def docker_get_volumes_last_updated(current_branch):
                 volumes_last_updated[repo_name] = repo['last_updated']
         if not page['next']:
             break
-        page = requests.get(page['next']).json()
+        page = requests.get(page['next'], timeout=60).json()
         attempts += 1
 
     return volumes_last_updated

.github/jobs/free_disk_space.sh

@@ -13,3 +13,25 @@ sudo rm -rf "$AGENT_TOOLSDIRECTORY"
 printf "\nChecking disk usage after cleanup"
 
 df -h
+
+echo Pruning docker files
+
+cmd="docker images"
+printf "\nBEFORE CLEANUP: $cmd"
+$cmd
+
+cmd="docker image prune -af"
+printf "\nRunning $cmd"
+$cmd
+
+cmd=docker system prune -af
+printf "\nRunning $cmd"
+$cmd
+
+cmd="docker images"
+printf "\nAFTER CLEANUP: $cmd"
+$cmd
+
+printf "\nChecking disk usage after Docker cleanup"
+
+df -h

.github/jobs/get_data_volumes.py

@@ -40,7 +40,7 @@ def main(args):
     branch_name = get_branch_name()
     if not branch_name:
         print("Could not get current branch. Exiting.")
-        sys.exit(1)
+        return None
 
     # remove -ref from end of branch name if found
     if branch_name.endswith('-ref'):
@@ -88,7 +88,7 @@ def main(args):
         # use it, otherwise use develop version of data volume
         elif (metplus_version == 'develop' and
               f'{branch_name}-{model_app_name}' in available_volumes):
-                volume_name = f'{branch_name}-{model_app_name}'
+            volume_name = f'{branch_name}-{model_app_name}'
         else:
             volume_name = f'{metplus_version}-{model_app_name}'
 
@@ -97,18 +97,23 @@ def main(args):
         cmd = (f'docker create --name {model_app_name} '
                f'{full_volume_name}')
         if not run_commands(cmd):
-            continue
+            print(f'ERROR: Could not create data volume for {full_volume_name}')
+            return None
 
         # add name to volumes from list to pass to docker build
         volume_list.append(f'--volumes-from {model_app_name}')
 
     return ' '.join(volume_list)
 
+
 if __name__ == "__main__":
     # split up command line args that have commas before passing into main
     args = []
 
     for arg in sys.argv[1:]:
         args.extend(arg.split(','))
     out = main(args)
+    if out is None:
+        print("ERROR: Something went wrong")
+        sys.exit(1)
     print(out)

.github/jobs/setup_and_run_diff.py

@@ -43,6 +43,9 @@
 output_category = f"output-{output_data_branch}-{artifact_name}"
 
 VOLUMES_FROM = get_data_volumes.main([output_category])
+if VOLUMES_FROM is None:
+    print("ERROR: Could not get truth data to run diff")
+    sys.exit(1)
 
 print(f"Output Volumes: {VOLUMES_FROM}")
 

.github/jobs/setup_and_run_use_cases.py

@@ -49,6 +49,10 @@ def main():
     )
     # get input data volumes
     volumes_from = get_data_volumes.main(categories_list)
+    if volumes_from is None:
+        print('ERROR: Could not get input data to run use cases')
+        sys.exit(1)
+
     print(f"Input Volumes: {volumes_from}")
 
     # build Docker image with conda environment and METplus branch image

.github/parm/use_case_groups.json

@@ -277,6 +277,6 @@
   {
     "category": "unstructured_grids",
     "index_list": "0",
-    "run": true
+    "run": false
   }
 ]

.github/workflows/testing.yml

@@ -145,8 +145,10 @@ jobs:
         env:
           METPLUS_TEST_OUTPUT_BASE: ${{ runner.workspace }}/pytest_output
       - name: Generate coverage report
-        run: coverage report -m
+        run: |
+          coverage report -m --fail-under=90 || echo "::error file=coverage,line=1,col=1::Code coverage is below 90%"
         if: always()
+        continue-on-error: true
       - name: Run Coveralls
         uses: AndreMiras/coveralls-python-action@8799c9f4443ac4201d2e2f2c725d577174683b99
         if: always()

docs/Contributors_Guide/conda_env.rst

@@ -1,10 +1,11 @@
 .. _conda_env:
 
+**************************************
 Instructions for the Conda Environment
-======================================
+**************************************
 
-Overview:  Replicating the Python 3.6.3 environment for running METplus
-_______________________________________________________________________
+Overview:  Replicating the Python 3.6.3 environment for Running METplus
+=======================================================================
 
 If the host (i.e. the computer on which the METplus and MET tools are
 running) doesn't already have all the necessary packages installed, it is
@@ -19,8 +20,8 @@ METplus source code:
 
    *METplus/environment.yml*
 
-Pre-condition- installing the conda package manager if it doesn't exist
-_______________________________________________________________________
+Pre-condition- Installing the Conda Package Manager If It Doesn't Exist
+=======================================================================
 
 **These instructions only need to be followed once.**
 
@@ -72,7 +73,7 @@ _______________________________________________________________________
 
 
 Creating the METplus conda env
-______________________________
+==============================
 
 **These instructions only need to be implemented once.**
 
@@ -127,8 +128,8 @@ to end the application.
 
        conda deactivate
 
-Activating and deactivating the METplus conda env
-_________________________________________________
+Activating and Deactivating the METplus conda env
+=================================================
 
 Once the user has followed the instructions under the "Creating the METplus
 conda env", follow these instructions to start running METplus :
@@ -163,8 +164,8 @@ conda env", follow these instructions to start running METplus :
 
 
 
-Optional: Checking for missing packages and mismatched version
-______________________________________________________________
+Optional: Checking for Missing Packages and Mismatched Version
+==============================================================
 
 To check for missing and mismatched packages run the **check_python.py**
 script in the METplus directory:
@@ -192,7 +193,7 @@ The results are sent to stdout (screen) and three files are created:
 
 
 Link Conda Directory to Data Disk (RAL Linux Machines)
-______________________________________________________
+======================================================
 
 By default, Conda environments are stored in a directory called ".conda" that is found in the user's home directory, i.e. /home/user/.conda (Note that the dot at the beginning of the directory name is a hidden directory that does not always show up in a directory listing). Conda environments can take up a lot of disk space which can quickly fill up the /home disk. It is recommended that you create a directory on a data disk that has more disk space and create a symbolic link from the .conda directory so the environments will be stored on the data disk. Keep in mind that deleting the directory on the data disk will delete all of your conda environments and they cannot be easily recovered.
 

docs/Contributors_Guide/create_wrapper.rst

@@ -1,26 +1,27 @@
+***************************
 How to Create a New Wrapper
-===========================
+***************************
 
 Naming
-------
+======
 
 File Name
-^^^^^^^^^
+---------
 
 Create the new wrapper in the *metplus/wrappers* directory and
 name it to reflect the wrapper's function, e.g.: new_tool_wrapper.py is
 a wrapper around an application named "new_tool."
 Copy the **example_wrapper.py** to start the process.
 
 Class Name
-^^^^^^^^^^
+----------
 
 The name of the class should match the wrapper's function without underscores
 and with the first letter of each word capitalized followed by "Wrapper."
 For example, the new_tool wrapper would be named **NewToolWrapper**.
 
 Add Entry to LOWER_TO_WRAPPER_NAME Dictionary
----------------------------------------------
+=============================================
 
 In *metplus/util/constants.py*, add entries to the LOWER_TO_WRAPPER_NAME
 dictionary so that the wrapper can be found in the PROCESS_LIST even if
@@ -57,12 +58,12 @@ More than one entry is rarely needed, but
 they will not hurt anything as long as they do not cause any conflicts.
 
 Wrapper Components
-------------------
+==================
 
 Open the wrapper file for editing the new class.
 
 Naming
-^^^^^^
+------
 
 Rename the class to match the wrapper's class from the above sections.
 Most wrappers should be a subclass of the RuntimeFreqWrapper::
@@ -80,7 +81,7 @@ To create EnsembleStat wrapper from GridStat, replace
 **GridStat** with **EnsembleStat**.
 
 Parent Class
-^^^^^^^^^^^^
+------------
 
 If the new tool falls under one of the existing tool categories,
 then make the tool a subclass of one of the existing classes.
@@ -91,15 +92,15 @@ See :ref:`bc_class_hierarchy` for more information on existing classes to
 determine which class to use as the parent class.
 
 Class Variables for Runtime Frequency
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+-------------------------------------
 
 **RUNTIME_FREQ_DEFAULT** and **RUNTIME_FREQ_SUPPORTED** should be set for all
 wrappers that inherit from **RuntimeFreqWrapper**.
 
 See :ref:`bc_class_vars` for more information.
 
 Init Function
-^^^^^^^^^^^^^
+-------------
 
 Modify the init function to initialize NewTool from its base class
 to set the self.app_name variable to the name of the application.
@@ -114,7 +115,7 @@ See the Basic Components :ref:`bc_init_function` section for more information::
         super().__init__(config, instance=instance)
 
 Read Configuration Variables
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+----------------------------
 
 The create_c_dict function is called during the initialization step of each
 wrapper. It is where values from the self.config object are read.
@@ -132,13 +133,13 @@ The function should also always return the c_dict variable::
     return c_dict
 
 File Input/Output
-"""""""""""""""""
+^^^^^^^^^^^^^^^^^
 
 METplus configuration variables that end with _DIR and _TEMPLATE are used
 to define the criteria to search for input files.
 
 Allow Multiple Files
-""""""""""""""""""""
+^^^^^^^^^^^^^^^^^^^^
 
 If the application can take more than one file as input for a given category
 (i.e. FCST, OBS, ENS, etc.) then ALLOW_MULTIPLE_FILES must be set to True::
@@ -151,7 +152,7 @@ If it is set to False and a list of files are found for an input
 then the wrapper will produce an error and not build the command.
 
 Run Functions
-^^^^^^^^^^^^^
+-------------
 
 * The **run_at_time_once** function or some the functions that it calls will
   need to be overridden in the wrapper.
@@ -240,7 +241,7 @@ Refer to the :ref:`basic_components_of_wrappers` section of the Contributor's
 Guide for more information on what should be added.
 
 Documentation
--------------
+=============
 
 * Add a section for the new wrapper in the 'Python Wrappers' section of the
   User's Guide. This includes a list of all configuration variables specific

docs/Contributors_Guide/deprecation.rst

@@ -1,12 +1,14 @@
+**********************************
 Deprecating an Old Config Variable
-==================================
+**********************************
 
 If a config variable changes names, an alert is needed to let
 the user know that they need to update the config files if they
 are using a deprecated variable.
 
 Example
--------
+=======
+
 ::
 
     [exe]
@@ -25,7 +27,8 @@ read and the user will have no way to know that they are setting the
 wrong variable and it is using WGRIB2 = wgrib2.
 
 check_for_deprecated_config()
------------------------------
+=============================
+
 In **metplus/util/constants.py** there is a dictionary called
 DEPRECATED_DICT that specifies the old config name as the key.
 The value is a dictionary of info that is used to help users update their