diff --git a/.github/ISSUE_TEMPLATE/update_truth.md b/.github/ISSUE_TEMPLATE/update_truth.md new file mode 100644 index 0000000000..0b18a6cd88 --- /dev/null +++ b/.github/ISSUE_TEMPLATE/update_truth.md @@ -0,0 +1,60 @@ +--- +name: Update Truth +about: Review use case differences that are caused by changes in an external repository and update truth dataset if necessary. +title: 'Update Truth: ' +labels: 'alert: NEED ACCOUNT KEY, alert: NEED MORE DEFINITION, alert: NEED CYCLE ASSIGNMENT, type: update truth, priority: blocker, component: CI/CD, requestor: METplus Team' +assignees: '' + +--- + +## Describe Expected Changes ## + +*Write a short summary of the differences that will likely be found in the GitHub Actions testing workflow that was triggered by the pull request that warranted this issue* + +## Define the Metadata ## + +### Title ### +- [ ] Add a link to the pull request that warranted this issue to the issue title using format dtcenter/{REPO}#{PR_NUMBER}. + +**Example:** Update Truth: For dtcenter/MET#2655 + +### Assignee ### + +*Assign this issue to the author of the pull request that warranted this issue. Optionally assign anyone else who should review the differences in the output.* + +- [ ] Select **engineer(s)** or **no engineer** required +- [ ] Select **scientist(s)** or **no scientist** required + +### Projects and Milestone ### +- [ ] Select **Repository** and/or **Organization** level **Project(s)** or add **alert: NEED CYCLE ASSIGNMENT** label +- [ ] Select **Milestone** as the next official version or **Future Versions** + +## Update Truth Checklist ### +- [ ] Review the GitHub Actions workflow that was triggered by the PR merge + - If no differences were found, note this in a comment. + - If all of the differences are expected, note this in a comment. + Include any details of how the review was performed. + - If unexpected differences are found, the following instructions can + help uncover potential explanations. If none of these apply and the + source of the differences cannot be determined, contact the + METplus wrappers lead engineer (@georgemccabe) for assistance. + - Search for other open issues that have the label `type: update truth` + applied by clicking on the label on this issue. Coordinate with the + author of these issues to ensure all diffs are properly reviewed. + - Check if any additional GitHub Actions testing workflows have been + triggered since the workflow that corresponds to this issue was run. + Review the latest run to ensure that there are no diffs that are + unrelated to this issue. + - If the incorrect differences are caused by the changes from the + issue that warranted this issue, consider reverting the PR and + re-opening the issue. + - Iterate until one of the above conditions apply. +- [ ] Approve the update of the truth data + - Contact the METplus wrappers lead engineer (@georgemccabe) or + backup lead (@jprestop) to let them know that the truth data can + be updated. +- [ ] Update the truth data. + This should be handled by a METplus wrappers engineer. + See the (instructions to update the truth data)[https://metplus.readthedocs.io/en/develop/Contributors_Guide/continuous_integration.html#update-truth-data-update-truth-data-yml] + for more info. +- [ ] Close this issue. diff --git a/docs/Contributors_Guide/add_use_case.rst b/docs/Contributors_Guide/add_use_case.rst index e6445f5944..2deea9dba5 100644 --- a/docs/Contributors_Guide/add_use_case.rst +++ b/docs/Contributors_Guide/add_use_case.rst @@ -656,36 +656,11 @@ Trigger Input Data Ingest If working in a forked METplus repository, the newly added input data will not become available for the tests unless it is triggered from the dtcenter -repository. A METplus developer will need to run the following steps. Please -provide them with the name of the forked repository and the branch that will -be used to create the pull request with the new use case. In this example, -the branch feature_XYZ exists in the *my_fake_user/METplus* repository. First, -clone the *dtcenter/METplus* repository, the run the following:: - - git remote add my_fake_user https://github.com/my_fake_user/METplus - git checkout develop - git checkout -b feature_XYZ - git pull my_fake_user feature_XYZ - git push origin feature_XYZ - git remote remove my_fake_user - -These commands will add a new remote to the forked repository, create a branch -off of the develop branch with the same name as the branch on the fork, pull -in the changes from the forked branch, then push the new branch up to -*dtcenter/METplus* on GitHub. Finally, the remote is removed to avoid clutter. - -Once these steps have been completed, go to *dtcenter/METplus* on GitHub -in a web browser and navigate to the -`Actions tab `_. -Click on the job named -"Docker Setup - Update Data Volumes" then click on "Update Data Volumes" and -verify that the new data tarfile was found on the DTC web server and the new -Docker data volume was created successfully. See -:ref:`verify-new-input-data-was-found`. If the input data was ingested -properly, then delete the feature branch from *dtcenter/METplus*. -This will avoid -confusion if this branch diverges from the branch on the forked repository that -will be used in the final pull request. +repository. A METplus developer will need to run the +:ref:`cg-ci-update-input-test-data` GitHub Actions workflow to trigger it. +Please provide them with the name of the branch that will +be used to create the pull request with the new use case. + .. _add_use_case_to_test_suite: @@ -1143,52 +1118,9 @@ Update the Truth Data The addition of a new use case results in new output data. When this happens, the reference branch needs to be updated so that future pull requests will compare their results to a "truth" data set that contains the new files. -Create a pull request with "develop" as the source branch and "develop-ref" as -the destination branch. This is done so that the pull request number -responsible for the changes in the truth data can be referenced to easily -track where differences occurred. - -A GitHub Action workflow is available to handle this step. - -* Ensure that the develop data directory has been updated to include all of the - new input data. - Check with the reviewers of recent pull requests that add a new use case to - confirm that the steps under :ref:`update-the-develop-data-directory` have - been completed. If this step has not been completed, then the new use case(s) - will fail and the new output data will not be added to the truth data set. -* Navigate to https://github.com/dtcenter/METplus/actions/workflows/update_truth.yml - or from the METplus GitHub page, click on the Actions tab, - then click on "Update Truth Data" under menu on the left. -* Click on the "Run workflow" button on the right. -* Click on the Branch pull down and select "develop" unless you are updating - the truth data for a bugfix on a main_vX.Y branch. -* Enter the pull request numbers that warranted the update. - Include the '#' symbol before the number to create a link to the PR. - PRs from a repository other than METplus should include - the repository name before '#' symbol. -* Enter a brief summary of the changes. - Developers can navigate to the PRs for more information. - -.. figure:: figure/update_truth_data.png - -* Click the "Run workflow" button. -* A new workflow run should appear at the top of the list and complete quickly. -* Click on the "Pull Requests" tab. - A new pull request should have been created with the information that - was entered. Click on the new pull request. -* Verify that the information in this pull request is correct. - If the "develop" branch was selected in the "Run workflow" menu, - then the pull request should show **develop-ref <- develop**. -* Add the appropriate project and milestone values on the right hand side. -* Scroll to the bottom of the pull request and click "Squash and merge." -* Click "Confirm squash and merge." It is not necessary to wait for the - automation checks to complete for this step. -* Monitor the Testing automation run for the develop-ref branch and ensure that - all of the use cases run successfully and the final step named - "Create Output Docker Data Volumes" completed successfully. -* If any use cases fail, check that the input data has been updated following - the instructions under :ref:`update-the-develop-data-directory` and rerun - all of the jobs of the -ref workflow. + +Follow the instructions for using the :ref:`cg-ci-update-truth-data` GitHub +Actions workflow to perform this step. Clean Up DTC Web Server diff --git a/docs/Contributors_Guide/continuous_integration.rst b/docs/Contributors_Guide/continuous_integration.rst index 0fbe5fff72..b4283f9a97 100644 --- a/docs/Contributors_Guide/continuous_integration.rst +++ b/docs/Contributors_Guide/continuous_integration.rst @@ -76,6 +76,106 @@ at the bottom of the workflow summary page when the workflow has completed. .. figure:: figure/ci-doc-artifacts.png +.. _cg-ci-update-truth-data: + +Update Truth Data (update_truth.yml) +------------------------------------ + +The METplus use case test truth data includes output from use cases that is +used to compare with new use case test results to flag any differences. +Differences can occur due to changes to the METplus wrappers +source code/configuration files or changes to any of its dependent +METplus components such as MET, METplotpy, METcalcpy, and METdataio. +Differences can also occur when a new use case is added, as the new use case +creates output that does not yet exist in the truth dataset. + +Once all differences are confirmed to be expected, +the reference branch, e.g. develop-ref, needs to be updated. This triggers a +:ref:`cg-ci-testing-workflow` that runs all of the use cases, creates +Docker images with the new truth data, and pushes them to DockerHub. +This is done so that future pull requests will +compare their results to the updated truth dataset. + +This process involves creating a pull request with "develop" as the source +branch and "develop-ref" as the destination branch. +This is done so that the pull request responsible for the changes in the +truth data can be referenced to easily track where differences occurred. + +The **Update Truth Data** workflow is available to handle this step. + +* Ensure that the develop data directory has been updated to include all of the + new input data. + Check with the reviewers of recent pull requests that add a new use case to + confirm that the steps under :ref:`update-the-develop-data-directory` have + been completed. If this step has not been completed, then the new use case(s) + will fail and the new output data will not be added to the truth data set. +* Navigate to https://github.com/dtcenter/METplus/actions/workflows/update_truth.yml + or from the METplus GitHub page, click on the Actions tab, + then click on "Update Truth Data" under menu on the left. +* Click on the "Run workflow" button on the right. +* Click on the Branch pull down and select "develop" unless you are updating + the truth data for a bugfix on a main_vX.Y branch. +* Enter the pull request numbers that warranted the update. + Include the '#' symbol before the number to create a link to the PR. + PRs from a repository other than METplus should include + the repository name before '#' symbol. +* Enter a brief summary of the changes. + Developers can navigate to the PRs for more information. + +.. figure:: figure/update_truth_data.png + +* Click the "Run workflow" button. +* A new workflow run should appear at the top of the list and complete quickly. +* Click on the "Pull Requests" tab. + A new pull request should have been created with the information that + was entered. Click on the new pull request. +* Verify that the information in this pull request is correct. + If the "develop" branch was selected in the "Run workflow" menu, + then the pull request should show **develop-ref <- develop**. +* Add the appropriate project and milestone values on the right hand side. +* If a GitHub issue exists to track the review of the differences, click on + the gear icon next to *Development* on the right side menu and add the issue. +* Scroll to the bottom of the pull request and click "Squash and merge." +* Click "Confirm squash and merge." It is not necessary to wait for the + automation checks to complete for this step. +* Monitor the Testing automation run for the develop-ref branch and ensure that + all of the use cases run successfully and the final step named + "Create Output Docker Data Volumes" completed successfully. +* If any use cases fail, check that the input data has been updated following + the instructions under :ref:`update-the-develop-data-directory` and rerun + all of the jobs of the -ref workflow. + +.. _cg-ci-update-input-test-data: + +Update Input Test Data (update_input_data.yml) +---------------------------------------------- + +New/updated input data for a METplus use case is read from the +DTC web server as described in the :ref:`use_case_input_data` section of the +**Adding Use Cases** chapter of the METplus Contributor's Guide. +This automatically happens as part of the :ref:`cg-ci-testing-workflow` when +a push event occurs on a dtcenter/METplus branch. +This step can be forced by using the **Update Input Test Data** workflow. + +This is workflow is typically used when a new use case is being provided by +an external contributor and their pull request is coming from a forked +repository. +Only dtcenter/METplus workflows have permission to update the input test data. + +To force the ingest of this input data, navigate to +https://github.com/dtcenter/METplus/actions/workflows/update_input_data.yml . +Click on the **Run workflow** pull-down, type the name of the branch that +matches the directory that contains the new data on the DTC web server, +and click the **Run workflow** button. + +The value in the branch pull-down under the text that says +**Use workflow from** is ignored if there is a value typed for the branch name. +If the branch name exists in the dtcenter/METplus repository, leave the +branch name text box blank and select the branch name from the pull-down menu. + +Verify that the workflow ran successfully and properly obtained the new data +by reviewing the log output from the workflow run. + Release Published (release_published.yml) - DEPRECATED ------------------------------------------------------