Update develop-ref after dtcenter/MET#2524 (#2171)

Co-authored-by: Julie Prestopnik <jpresto@ucar.edu>
Co-authored-by: johnhg <johnhg@ucar.edu>
Co-authored-by: cristianastan2 <cstan@gmu.edu>
Co-authored-by: John Halley Gotway <johnhg@ucar.edu>
Co-authored-by: bikegeek <minnawin@ucar.edu>
Co-authored-by: Lisa Goodrich <lisag@ucar.edu>
Co-authored-by: Julie Prestopnik <jpresto@seneca.rap.ucar.edu>
Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com>
Co-authored-by: Hank Fisher <fisherh@ucar.edu>
Co-authored-by: Dan Adriaansen <dadriaan@ucar.edu>
Co-authored-by: jprestop <jpresto@ucar.edu>
Co-authored-by: Tracy Hertneky <hertneky@seneca.rap.ucar.edu>
Co-authored-by: Giovanni Rosa <giovanni.rosa@unimol.it>
Co-authored-by: j-opatz <59586397+j-opatz@users.noreply.github.com>
Co-authored-by: lisagoodrich <33230218+lisagoodrich@users.noreply.github.com>
Co-authored-by: bikegeek <3753118+bikegeek@users.noreply.github.com>
Co-authored-by: j-opatz <jopatz@ucar.edu>
Co-authored-by: Will Mayfield <59745143+willmayfield@users.noreply.github.com>
Co-authored-by: metplus-bot <97135045+metplus-bot@users.noreply.github.com>
Co-authored-by: Tracy Hertneky <39317287+hertneky@users.noreply.github.com>
Co-authored-by: Giovanni Rosa <g.rosa1@studenti.unimol.it>
fixing errors #650
fixing errors take 2 #650
fixing errors take 3 #650
fixing errors take 4 #650
fixing errors take 5 #650
fixing question order #650
fix #1706 fix PhaseDiagram use case to avoid writing into INPUT_BASE (#1708)
fix #1713 develop METPLOTPY_BASE (#1715)
fix #1691 remove whitespace from output file paths (#1721)
fix Contributor's Guide GitHub Workflow page (#1774)
fix release (#1790)
fix GitHub Actions warnings (#1864)
fix #1884 develop PCPCombine {custom} in subtract method (#1887)
fix #1939 develop - failure reading obs when zipped file also exists (#1941)
Closes #1986
fix develop Fix broken documentation links (#2004)
fix #2026 develop StatAnalysis looping (#2028)
fix priority of obs_window config variables so that wrapper-specific version is preferred over generic OBS_WINDOW_BEGIN/END (#2062)
fix #2070 var list numeric order (#2072)
fix #2087 develop docs_pdf (#2091)
fix #2096/#2098 develop - fix skip if output exists and do not error if no commands were run (#2099)
Fix for Dockerfile smell DL4000 (#2112)
fix #2082 develop regrid.convert/censor_thresh/censor_val (#2140)
fix #2082 main_v5.0 regrid.convert/censor_thresh/censor_val (#2101)
fix #2137 develop PointStat -obs_valid_beg/end (#2141)
fix failured introduced by urllib3 (see urllib3/urllib3#2168)
fix #2161 develop PCPCombine additional field arguments in -subtract mode (#2162)

.github/jobs/get_metviewer.sh

@@ -7,13 +7,13 @@ export METVIEWER_DIR=$RUNNER_WORKSPACE/output/metviewer
 export METVIEWER_DOCKER_IMAGE=dtcenter/metviewer:develop
 
 # install docker-compose
-apk add docker-compose
+apk add docker-cli-compose
 
 # download docker-compose.yml file from METviewer develop branch
 wget https://raw.githubusercontent.com/dtcenter/METviewer/develop/docker/docker-compose.yml
 
 # Run docker-compose to create the containers
-docker-compose up -d
+docker compose up -d
 
 # sleep for a few seconds to ensure database has fully started
 sleep 20

.github/workflows/testing.yml

@@ -53,7 +53,7 @@ on:
 jobs:
 
   event_info:
-    name: "Trigger: ${{ github.event_name != 'workflow_dispatch' && github.event_name || github.event.inputs.repository }} ${{ github.event_name != 'workflow_dispatch' && 'local' || github.event.inputs.actor }} ${{ github.event_name != 'workflow_dispatch' && 'event' || github.event.inputs.sha }}"
+    name: "Trigger: ${{ github.event_name != 'workflow_dispatch' && github.event_name || github.event.inputs.repository }} ${{ github.event_name != 'workflow_dispatch' && 'event' || github.event.inputs.sha }} ${{ github.event_name != 'workflow_dispatch' && 'local' || github.event.inputs.actor }} "
     runs-on: ubuntu-latest
     steps:
       - name: Print GitHub values for reference

.github/workflows/update_input_data.yml

@@ -0,0 +1,28 @@
+name: Update Input Test Data
+on:
+  workflow_dispatch:
+    inputs:
+      branch:
+        description: 'Branch to update if not in above pull down list (e.g. PR coming from a fork)'
+
+jobs:
+  update_data_volumes:
+    name: Update Data Volumes
+    runs-on: ubuntu-latest
+    steps:
+      - run: |
+          branch_name=$(echo ${{ github.event.inputs.branch || github.ref }} | cut -d/ -f3)
+          echo "branch_name=$branch_name" >> "$GITHUB_ENV"
+      - uses: dtcenter/metplus-action-data-update@v2
+        with:
+          docker_name: ${{ secrets.DOCKER_USERNAME }}
+          docker_pass: ${{ secrets.DOCKER_PASSWORD }}
+          repo_name: ${{ github.repository }}
+          data_prefix: sample_data
+          branch_name: ${{ env.branch_name }}
+          docker_data_dir: /data/input/METplus_Data
+          data_repo_dev: metplus-data-dev
+          data_repo_stable: metplus-data
+          use_feature_data: true
+          tag_max_pages: 15
+

.readthedocs.yaml

@@ -9,10 +9,14 @@ version: 2
 #formats: all
 formats: [pdf]
 
+build:
+  os: ubuntu-22.04
+  tools:
+    python: "3.10"
+
 # Optionally set the version of Python and requirements required to build your
 # docs
 python:
-   version: 3.8
    install:
    - requirements: docs/requirements.txt
    - requirements: requirements.txt

docs/Contributors_Guide/add_use_case.rst

@@ -882,6 +882,8 @@ chapter under
 Pull Request Reviewer Instructions
 ==================================
 
+.. _update-the-develop-data-directory:
+
 Update the develop data directory
 ---------------------------------
 
@@ -1071,8 +1073,9 @@ then combine the two list items into a single item::
 
     - "met_tool_wrapper:0-54"
 
-Create a pull request from develop into develop-ref
----------------------------------------------------
+
+Update the Truth Data
+---------------------
 
 The addition of a new use case results in new output data. When this happens,
 the reference branch needs to be updated so that future pull requests will
@@ -1082,45 +1085,48 @@ the destination branch. This is done so that the pull request number
 responsible for the changes in the truth data can be referenced to easily
 track where differences occurred.
 
-Merging develop into develop-ref often causes strange conflicts. It really is
-necessary and important to update develop-ref with the latest content of
-develop. Follow these command line instructions in the METplus repository to
-reconcile the conflicts before creating the pull request.
-
-* Reconcile conflicts between develop and develop-ref branches.
-
-::
-
-    git checkout develop-ref
-    git pull
-    git checkout develop
-    git pull
-    git merge -s ours develop-ref
-    git push origin develop
-
-* Next click
-  `here <https://github.com/dtcenter/METplus/compare/develop-ref...develop>`_
-  and click the green "Create pull request" button to create the pull request.
-
-.. figure:: figure/develop_into_develop-ref.png
-
-* Set the name of the pull request to "Update develop-ref after #XXXX" where
-  XXXX is the pull request number that introduced the differences.
-
-* Delete the template content and add the pull request number (formatted #XXXX)
-  and a brief description of what has changed. The description is optional
-  because the link to the pull request should contain this information.
-
+A GitHub Action workflow is available to handle this step.
+
+* Ensure that the develop data directory has been updated to include all of the
+  new input data.
+  Check with the reviewers of recent pull requests that add a new use case to
+  confirm that the steps under :ref:`update-the-develop-data-directory` have
+  been completed. If this step has not been completed, then the new use case(s)
+  will fail and the new output data will not be added to the truth data set.
+* Navigate to https://github.com/dtcenter/METplus/actions/workflows/update_truth.yml
+  or from the METplus GitHub page, click on the Actions tab,
+  then click on "Update Truth Data" under menu on the left.
+* Click on the "Run workflow" button on the right.
+* Click on the Branch pull down and select "develop" unless you are updating
+  the truth data for a bugfix on a main_vX.Y branch.
+* Enter the pull request numbers that warranted the update.
+  Include the '#' symbol before the number to create a link to the PR.
+  PRs from a repository other than METplus should include
+  the repository name before '#' symbol.
+* Enter a brief summary of the changes.
+  Developers can navigate to the PRs for more information.
+
+.. figure:: figure/update_truth_data.png
+
+* Click the "Run workflow" button.
+* A new workflow run should appear at the top of the list and complete quickly.
+* Click on the "Pull Requests" tab.
+  A new pull request should have been created with the information that
+  was entered. Click on the new pull request.
+* Verify that the information in this pull request is correct.
+  If the "develop" branch was selected in the "Run workflow" menu,
+  then the pull request should show **develop-ref <- develop**.
 * Add the appropriate project and milestone values on the right hand side.
-
-* Create the pull request.
-
-* Squash and merge the pull request. It is not necessary to wait for the
+* Scroll to the bottom of the pull request and click "Squash and merge."
+* Click "Confirm squash and merge." It is not necessary to wait for the
   automation checks to complete for this step.
-
 * Monitor the Testing automation run for the develop-ref branch and ensure that
   all of the use cases run successfully and the final step named
   "Create Output Docker Data Volumes" completed successfully.
+* If any use cases fail, check that the input data has been updated following
+  the instructions under :ref:`update-the-develop-data-directory` and rerun
+  all of the jobs of the -ref workflow.
+
 
 Clean Up DTC Web Server
 -----------------------

metplus/wrappers/pcp_combine_wrapper.py

@@ -366,6 +366,9 @@ def setup_subtract_method(self, time_info, accum, data_src):
         if self.c_dict.get(f"{data_src}_LEVELS"):
             field_args['level'] = self.c_dict[f"{data_src}_LEVELS"][0]
 
+        if self.c_dict.get(f"{data_src}_OPTIONS"):
+            field_args['extra'] = self.c_dict[f"{data_src}_OPTIONS"][0]
+
         # if data is GRIB and second lead is 0, then
         # run PCPCombine in -add mode with just the first file
         if lead2 == 0 and not self.c_dict[f'{data_src}_USE_ZERO_ACCUM']: