Skip to content

Commit

Permalink
Feature 1694 s2s subgroups (#1697)
Browse files Browse the repository at this point in the history
* per #1694, moved 4 use cases from s2s to s2s_mjo, updated paths accordingly, and turned on all s2s use cases to test that they all run successfully after the changes

* per #1694, fixed paths to s2s_mjo conf files

* updated documentation for use cases that were moved from s2s to s2s_mjo

* attempt to free up unused disk space in GHA runner environment

* moved 4 s2s use cases into s2s_mid_lat

* added new model application categories to contrib guide for adding new use cases

* per #947, changed convection_allowing_models use cases to short_range

* changed which use case tests run to the ones that are failing and added other METdbLoad use case to see if that fails as well

* test to determine which files are preventing MySQL database from being created properly

* test 2 to determine which files are preventing MySQL database from being created properly

* test 3 to see if removing these files is not the cause of the METdbLoad failure

* updated references to METdatadb to METdataio since the repository was renamed

* fixed typo

* changed path to sql file needed to create database because it was moved from METviewer to METdataio

* fixed path to sql file that was moved from METviewer to METdataio

* removed temporary fix because metdataio conda env was created in the dtcenter/metplus-envs:metdataio Docker image

* added note to update path when METviewer Dockerfile changes to reflect METdatadb rename to METdataio, ci-skip-unit-tests

* fixed path to METdataio repo

* add back commands to free up disk space because issue with METdbLoad use case was likely not related, ci-skip-unit-tests

* run all tests with ci-run-all-diff

* remove use case group added for testing, ci-skip-all

* changed exit code for diff tests to 2 so it is easier to see if a use case test job failed due to an actual failure or due to differences in the output

* changed grouping of s2s mid lat use cases to original grouping to prevent warning that artifact contains more than 10,000 files. The 2 WeatherRegime use cases produce a lot of output files, so splitting them up should resolve this warning

* per #1694, changed all references to convection allowing models to short range in the Verification Datasets section of the documentation

* changed URLs to develop version of documentation to a URL relative to the current version of the documentation to match the quick search links from the METplus User's Guide

* per #947, changed references to convection_allowing_model (without the s) to short_range that were missed

* updated use case test scripts to rename convection_allowing_models to short_range and added note to alert developers that the list of use cases in the script is not maintained and therefore not complete
  • Loading branch information
georgemccabe committed Jul 18, 2022
1 parent 2da1f91 commit a20c0bb
Show file tree
Hide file tree
Showing 99 changed files with 435 additions and 400 deletions.
1 change: 1 addition & 0 deletions .github/jobs/free_disk_space.sh
Original file line number Diff line number Diff line change
Expand Up @@ -11,4 +11,5 @@ sudo rm -rf "/usr/local/share/boost"
sudo rm -rf "$AGENT_TOOLSDIRECTORY"

printf "\nChecking disk usage after cleanup"

df -h
2 changes: 1 addition & 1 deletion .github/jobs/get_metviewer.sh
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ sleep 20
# print list of currently running containers to
# verify mysql and metviewer are running
docker ps -a

# TODO: when METviewer Dockerfile changes METdatadb to METdataio, update path to sql file accordingly
# commands to run inside METviewer container
cmd="mysql -hmysql_mv -uroot -pmvuser -e\"create database mv_metplus_test;\";"
cmd+="mysql -hmysql_mv -uroot -pmvuser mv_metplus_test < /METviewer-python/METdataio/METdbLoad/sql/mv_mysql.sql"
Expand Down
8 changes: 4 additions & 4 deletions .github/jobs/get_use_case_commands.py
Original file line number Diff line number Diff line change
Expand Up @@ -72,13 +72,13 @@ def handle_automation_env(host_name, reqs, work_dir):
'cd -;'
)

# if metdatadb is in requirements list, add command to obtain METdatadb
if 'metdatadb' in str(reqs).lower():
# if metdataio is in requirements list, add command to obtain METdataio
if 'metdataio' in str(reqs).lower():
setup_env += (
f'cd {METPLUS_DOCKER_LOC};'
f'{work_dir}/manage_externals/checkout_externals'
f' -e {work_dir}/.github/parm/Externals_metdatadb.cfg;'
f'{python_path} -m pip install {METPLUS_DOCKER_LOC}/../METdatadb;'
f' -e {work_dir}/.github/parm/Externals_metdataio.cfg;'
f'{python_path} -m pip install {METPLUS_DOCKER_LOC}/../METdataio;'
'cd -;'
)

Expand Down
2 changes: 1 addition & 1 deletion .github/jobs/run_difference_tests.sh
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ if [ "$( ls -A ${RUNNER_WORKSPACE}/diff)" ]; then
echo ::set-output name=upload_diff::true
mkdir -p artifact/diff-${artifact_name}
cp -r ${RUNNER_WORKSPACE}/diff/* artifact/diff-${artifact_name}
exit 1
exit 2
fi

echo ::set-output name=upload_diff::false
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
[METdatadb]
local_path = ../METdatadb
local_path = ../METdataio
protocol = git
required = True
repo_url = https://github.com/dtcenter/METdatadb
repo_url = https://github.com/dtcenter/METdataio
branch = develop

[externals_description]
Expand Down
42 changes: 21 additions & 21 deletions .github/parm/use_case_groups.json
Original file line number Diff line number Diff line change
Expand Up @@ -25,32 +25,32 @@
"run": false
},
{
"category": "convection_allowing_models",
"category": "short_range",
"index_list": "0",
"run": false
},
{
"category": "convection_allowing_models",
"category": "short_range",
"index_list": "1",
"run": false
},
{
"category": "convection_allowing_models",
"category": "short_range",
"index_list": "2-6",
"run": false
},
{
"category": "convection_allowing_models",
"category": "short_range",
"index_list": "7",
"run": false
},
{
"category": "convection_allowing_models",
"category": "short_range",
"index_list": "8",
"run": false
},
{
"category": "convection_allowing_models",
"category": "short_range",
"index_list": "9",
"run": false
},
Expand Down Expand Up @@ -136,52 +136,52 @@
},
{
"category": "s2s",
"index_list": "1-3",
"index_list": "1",
"run": false
},
{
"category": "s2s",
"index_list": "4",
"index_list": "2",
"run": false
},
{
"category": "s2s",
"index_list": "5",
"index_list": "3",
"run": false
},
{
"category": "s2s",
"index_list": "6-7",
"index_list": "4",
"run": false
},
{
"category": "s2s",
"index_list": "7-9",
"index_list": "5",
"run": false
},
{
"category": "s2s",
"index_list": "10",
"index_list": "6",
"run": false
},
{
"category": "s2s",
"index_list": "11",
"category": "s2s_mid_lat",
"index_list": "0-2",
"run": false
},
{
"category": "s2s",
"index_list": "12",
"run": false
"category": "s2s_mid_lat",
"index_list": "3",
"run": false
},
{
"category": "s2s",
"index_list": "13",
"category": "s2s_mjo",
"index_list": "0-2",
"run": false
},
{
"category": "s2s",
"index_list": "14",
"category": "s2s_mjo",
"index_list": "3",
"run": false
},
{
Expand Down
4 changes: 3 additions & 1 deletion docs/Contributors_Guide/add_use_case.rst
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,6 @@ one of the following:

* air_quality_and_comp
* climate
* convection_allowing_models
* data_assimilation
* extremes
* land_surface
Expand All @@ -52,6 +51,9 @@ one of the following:
* pbl
* precipitation
* s2s (Subseasonal to Seasonal)
* s2s_mid_lat (Subseasonal to Seasonal: Mid-Latitude)
* s2s_mjo (Subseasonal to Seasonal: Madden-Julian Oscillation)
* short_range (formerly convection_allowing_models)
* space_weather
* tc_and_extra_tc (Tropical Cyclone and Extratropical Cyclone)

Expand Down
8 changes: 6 additions & 2 deletions docs/Users_Guide/quicksearch.rst
Original file line number Diff line number Diff line change
Expand Up @@ -70,28 +70,32 @@ Use Cases by Application:

| `Air Quality and Composition <../search.html?q=AirQualityAndCompAppUseCase&check_keywords=yes&area=default>`_
| `Climate <../search.html?q=ClimateAppUseCase&check_keywords=yes&area=default>`_
| `Convection Allowing Models <../search.html?q=ConvectionAllowingModelsAppUseCase&check_keywords=yes&area=default>`_
| `Short Range <../search.html?q=ShortRangeAppUseCase&check_keywords=yes&area=default>`_
| `Data Assimilation <../search.html?q=DataAssimilationAppUseCase&check_keywords=yes&area=default>`_
| `Ensemble <../search.html?q=EnsembleAppUseCase&check_keywords=yes&area=default>`_
| `Marine and Cryosphere <../search.html?q=MarineAndCryosphereAppUseCase&check_keywords=yes&area=default>`_
| `Medium Range <../search.html?q=MediumRangeAppUseCase&check_keywords=yes&area=default>`_
| `Precipitation <../search.html?q=PrecipitationAppUseCase&check_keywords=yes&area=default>`_
| `Space Weather <../search.html?q=SpaceWeatherAppUseCase&check_keywords=yes&area=default>`_
| `Subseasonal to Seasonal <../search.html?q=S2SAppUseCase&check_keywords=yes&area=default>`_
| `Subseasonal to Seasonal: Madden-Julian Oscillation <../search.html?q=S2SMJOAppUseCase&check_keywords=yes&area=default>`_
| `Subseasonal to Seasonal: Mid-Latitude <../search.html?q=S2SMidLatAppUseCase&check_keywords=yes&area=default>`_
| `Tropical Cyclone and Extra-Tropical Cyclone <../search.html?q=TCandExtraTCAppUseCase&check_keywords=yes&area=default>`_
.. only:: latex

| **Air Quality and Composition**: *AirQualityAndCompAppUseCase*
| **Climate**: *ClimateAppUseCase*
| **Convection Allowing Models**: *ConvectionAllowingModelsAppUseCase*
| **Short Range**: *ShortRangeAppUseCase*
| **Data Assimilation**: *DataAssimilationAppUseCase*
| **Ensemble**: *EnsembleAppUseCase*
| **Marine and Cryosphere**: *MarineAndCryosphereAppUseCase*
| **Medium Range**: *MediumRangeAppUseCase*
| **Precipitation**: *PrecipitationAppUseCase*
| **Space Weather**: *SpaceWeatherAppUseCase*
| **Subseasonal to Seasonal**: *S2SAppUseCase*
| **Subseasonal to Seasonal: Madden-Julian Oscillation**: *S2SMJOAppUseCase*
| **Subseasonal to Seasonal: Mid-Latitude**: *S2SMidLatAppUseCase*
| **Tropical Cyclone and Extra-Tropical Cyclone**: *TCandExtraTCAppUseCase*
Use Cases by Organization:
Expand Down
2 changes: 1 addition & 1 deletion docs/Verification_Datasets/datasets/aeronet.rst
Original file line number Diff line number Diff line change
Expand Up @@ -52,4 +52,4 @@ METplus Use Cases
Keywords
.. note:: **Current Dataset:** VxDataAERONET

.. note:: **Data Labels:** DataTypePoint, DataLevelSurface, DataLevelUpperAir, DataProviderNASA, DataApplicationConvectionAllowingModels, DataApplicationMediumRange
.. note:: **Data Labels:** DataTypePoint, DataLevelSurface, DataLevelUpperAir, DataProviderNASA, DataApplicationShortRange, DataApplicationMediumRange
2 changes: 1 addition & 1 deletion docs/Verification_Datasets/datasets/gdas_prepbufr.rst
Original file line number Diff line number Diff line change
Expand Up @@ -48,4 +48,4 @@ METplus Use Cases
Keywords
.. note:: **Current Dataset:** VxDataGDASPREP

.. note:: **Data Labels:** DataTypePoint, DataLevelSurface, DataLevelUpperAir, DataProviderNOAA, DataApplicationConvectionAllowingModels, DataApplicationMediumRange
.. note:: **Data Labels:** DataTypePoint, DataLevelSurface, DataLevelUpperAir, DataProviderNOAA, DataApplicationShortRange, DataApplicationMediumRange
2 changes: 1 addition & 1 deletion docs/Verification_Datasets/datasets/goes_level_1b.rst
Original file line number Diff line number Diff line change
Expand Up @@ -57,4 +57,4 @@ METplus Use Cases
Keywords
.. note:: **Current Dataset:** VxDataGOESLEV1B

.. note:: **Data Labels:** DataTypeGridded, DataLevelSatellite, DataProviderNASA, DataApplicationConvectionAllowingModels
.. note:: **Data Labels:** DataTypeGridded, DataLevelSatellite, DataProviderNASA, DataApplicationShortRange
Original file line number Diff line number Diff line change
Expand Up @@ -63,4 +63,4 @@ METplus Use Cases
Keywords
.. note:: **Current Dataset:** VxDataGOESLEV2AERO

.. note:: **Data Labels:** DataTypeGridded, DataLevelSatellite, DataProviderNASA, DataApplicationConvectionAllowingModels
.. note:: **Data Labels:** DataTypeGridded, DataLevelSatellite, DataProviderNASA, DataApplicationShortRange
2 changes: 1 addition & 1 deletion docs/Verification_Datasets/datasets/madis.rst
Original file line number Diff line number Diff line change
Expand Up @@ -47,4 +47,4 @@ METplus Use Cases
Keywords
.. note:: **Current Dataset:** VxDataMADIS

.. note:: **Data Labels:** DataTypePoint, DataLevelSurface, DataProviderNOAA, DataApplicationPrecipitation, DataApplicationConvectionAllowingModels, DataApplicationMediumRange
.. note:: **Data Labels:** DataTypePoint, DataLevelSurface, DataProviderNOAA, DataApplicationPrecipitation, DataApplicationShortRange, DataApplicationMediumRange
2 changes: 1 addition & 1 deletion docs/Verification_Datasets/datasets/metar_isu.rst
Original file line number Diff line number Diff line change
Expand Up @@ -46,4 +46,4 @@ METplus Use Cases
Keywords
.. note:: **Current Dataset:** VxDataMETAR

.. note:: **Data Labels:** DataTypePoint, DataLevelSurface, DataProviderISU, DataApplicationPrecipitation, DataApplicationConvectionAllowingModels, DataApplicationMediumRange
.. note:: **Data Labels:** DataTypePoint, DataLevelSurface, DataProviderISU, DataApplicationPrecipitation, DataApplicationShortRange, DataApplicationMediumRange
2 changes: 1 addition & 1 deletion docs/Verification_Datasets/datasets/mping.rst
Original file line number Diff line number Diff line change
Expand Up @@ -48,4 +48,4 @@ METplus Use Cases
Keywords
.. note:: **Current Dataset:** VxDataMPING

.. note:: **Data Labels:** DataTypePoint, DataLevelSurface, DataProviderOU, DataApplicationPrecipitation, DataApplicationConvectionAllowingModels
.. note:: **Data Labels:** DataTypePoint, DataLevelSurface, DataProviderOU, DataApplicationPrecipitation, DataApplicationShortRange
Original file line number Diff line number Diff line change
Expand Up @@ -41,4 +41,4 @@ METplus Use Cases
Keywords
.. note:: **Current Dataset:** VxDataMRMSCompositeReflectivity

.. note:: **Data Labels:** DataTypeGridded, DataLevelSurface, DataProviderNOAA, DataApplicationConvectionAllowingModels
.. note:: **Data Labels:** DataTypeGridded, DataLevelSurface, DataProviderNOAA, DataApplicationShortRange
2 changes: 1 addition & 1 deletion docs/Verification_Datasets/datasets/nexrad_level_2.rst
Original file line number Diff line number Diff line change
Expand Up @@ -49,4 +49,4 @@ METplus Use Cases
Keywords
.. note:: **Current Dataset:** VxDataNexradLevel2

.. note:: **Data Labels:** DataTypeGridded, DataLevelSurface, DataProviderNOAA, DataApplicationConvectionAllowingModels, DataApplicationMediumRange
.. note:: **Data Labels:** DataTypeGridded, DataLevelSurface, DataProviderNOAA, DataApplicationShortRange, DataApplicationMediumRange
2 changes: 1 addition & 1 deletion docs/Verification_Datasets/datasets/nexrad_level_3.rst
Original file line number Diff line number Diff line change
Expand Up @@ -52,4 +52,4 @@ METplus Use Cases
Keywords
.. note:: **Current Dataset:** VxDataNexradLevel3

.. note:: **Data Type Labels:** DataTypeGridded, DataLevelSurface, DataProviderNOAA, DataApplicationConvectionAllowingModels, DataApplicationMediumRange
.. note:: **Data Type Labels:** DataTypeGridded, DataLevelSurface, DataProviderNOAA, DataApplicationShortRange, DataApplicationMediumRange
2 changes: 1 addition & 1 deletion docs/Verification_Datasets/datasets/opera_eumetnet.rst
Original file line number Diff line number Diff line change
Expand Up @@ -51,4 +51,4 @@ METplus Use Cases
Keywords
.. note:: **Current Dataset:** VxDataOperaEumetnet

.. note:: **Data Labels:** DataTypeGridded, DataLevelSurface, DataProviderEUMETNET, DataApplicationPrecipitation, DataApplicationConvectionAllowingModels
.. note:: **Data Labels:** DataTypeGridded, DataLevelSurface, DataProviderEUMETNET, DataApplicationPrecipitation, DataApplicationShortRange
2 changes: 1 addition & 1 deletion docs/Verification_Datasets/datasets/template.rst
Original file line number Diff line number Diff line change
Expand Up @@ -80,7 +80,7 @@ Keywords
DataProviderUSAF
Select at least one DataApplication label:
DataApplicationClimate
DataApplicationConvectionAllowingModels
DataApplicationShortRange
DataApplicationEnsemble
DataApplicationMarineAndCryo
DataApplicationMediumRange
Expand Down
2 changes: 1 addition & 1 deletion docs/Verification_Datasets/datasets/viirs.rst
Original file line number Diff line number Diff line change
Expand Up @@ -52,4 +52,4 @@ METplus Use Cases

Keywords
.. note:: **Current Dataset:** VxDataVIIRS
.. note:: **Data Labels:** DataTypeGridded, DataLevelSatellite, DataProviderNASA, DataApplicationConvectionAllowingModels, DataApplicationMediumRange
.. note:: **Data Labels:** DataTypeGridded, DataLevelSatellite, DataProviderNASA, DataApplicationShortRange, DataApplicationMediumRange
42 changes: 21 additions & 21 deletions docs/Verification_Datasets/quicksearch.rst
Original file line number Diff line number Diff line change
Expand Up @@ -5,33 +5,33 @@ Quick Search for Datasets

Dataset Types
-------------
| `Point Data <https://metplus.readthedocs.io/en/develop/search.html?q=DataTypePoint&check_keywords=yes&area=default>`_
| `Gridded Data <https://metplus.readthedocs.io/en/develop/search.html?q=DataTypeGridded&check_keywords=yes&area=default>`_
| `Point Data <../search.html?q=DataTypePoint&check_keywords=yes&area=default>`_
| `Gridded Data <../search.html?q=DataTypeGridded&check_keywords=yes&area=default>`_
Dataset Levels
--------------
| `Surface Data <https://metplus.readthedocs.io/en/develop/search.html?q=DataLevelSurface&check_keywords=yes&area=default>`_
| `Upper-Air Data <https://metplus.readthedocs.io/en/develop/search.html?q=DataLevelUpperAir&check_keywords=yes&area=default>`_
| `Satellite Data <https://metplus.readthedocs.io/en/develop/search.html?q=DataLevelSatellite&check_keywords=yes&area=default>`_
| `Surface Data <../search.html?q=DataLevelSurface&check_keywords=yes&area=default>`_
| `Upper-Air Data <../search.html?q=DataLevelUpperAir&check_keywords=yes&area=default>`_
| `Satellite Data <../search.html?q=DataLevelSatellite&check_keywords=yes&area=default>`_
Dataset Providers
-----------------
| `EUMETNET <https://metplus.readthedocs.io/en/develop/search.html?q=DataProviderEUMETNET&check_keywords=yes&area=default>`_
| `ISU <https://metplus.readthedocs.io/en/develop/search.html?q=DataProviderISU&check_keywords=yes&area=default>`_
| `NOAA <https://metplus.readthedocs.io/en/develop/search.html?q=DataProviderNOAA&check_keywords=yes&area=default>`_
| `NASA <https://metplus.readthedocs.io/en/develop/search.html?q=DataProviderNASA&check_keywords=yes&area=default>`_
| `OU <https://metplus.readthedocs.io/en/develop/search.html?q=DataProviderOU&check_keywords=yes&area=default>`_
| `USAF <https://metplus.readthedocs.io/en/develop/search.html?q=DataProviderUSAF&check_keywords=yes&area=default>`_
| `NSIDC <https://metplus.readthedocs.io/en/develop/search.html?q=DataProviderNSIDC&check_keywords=yes&area=default>`_
| `EUMETNET <../search.html?q=DataProviderEUMETNET&check_keywords=yes&area=default>`_
| `ISU <../search.html?q=DataProviderISU&check_keywords=yes&area=default>`_
| `NOAA <../search.html?q=DataProviderNOAA&check_keywords=yes&area=default>`_
| `NASA <../search.html?q=DataProviderNASA&check_keywords=yes&area=default>`_
| `OU <../search.html?q=DataProviderOU&check_keywords=yes&area=default>`_
| `USAF <../search.html?q=DataProviderUSAF&check_keywords=yes&area=default>`_
| `NSIDC <../search.html?q=DataProviderNSIDC&check_keywords=yes&area=default>`_
Dataset Applications
--------------------
| `Climate <https://metplus.readthedocs.io/en/develop/search.html?q=DataApplicationClimate&check_keywords=yes&area=default>`_
| `Convection Allowing Models <https://metplus.readthedocs.io/en/develop/search.html?q=DataApplicationConvectionAllowingModels&check_keywords=yes&area=default>`_
| `Ensemble <https://metplus.readthedocs.io/en/develop/search.html?q=DataApplicationEnsemble&check_keywords=yes&area=default>`_
| `Marine and Cryosphere <https://metplus.readthedocs.io/en/develop/search.html?q=DataApplicationMarineAndCryo&check_keywords=yes&area=default>`_
| `Medium Range <https://metplus.readthedocs.io/en/develop/search.html?q=DataApplicationMediumRange&check_keywords=yes&area=default>`_
| `Precipitation <https://metplus.readthedocs.io/en/develop/search.html?q=DataApplicationPrecipitation&check_keywords=yes&area=default>`_
| `Space Weather <https://metplus.readthedocs.io/en/develop/search.html?q=DataApplicationSpaceWeather&check_keywords=yes&area=default>`_
| `Subseasonal to Seasonal <https://metplus.readthedocs.io/en/develop/search.html?q=DataApplicationS2S&check_keywords=yes&area=default>`_
| `Tropical Cyclone and Extra-Tropical Cyclone <https://metplus.readthedocs.io/en/develop/search.html?q=DataApplicationTCandExtraTC&check_keywords=yes&area=default>`_
| `Climate <../search.html?q=DataApplicationClimate&check_keywords=yes&area=default>`_
| `Ensemble <../search.html?q=DataApplicationEnsemble&check_keywords=yes&area=default>`_
| `Marine and Cryosphere <../search.html?q=DataApplicationMarineAndCryo&check_keywords=yes&area=default>`_
| `Medium Range <../search.html?q=DataApplicationMediumRange&check_keywords=yes&area=default>`_
| `Precipitation <../search.html?q=DataApplicationPrecipitation&check_keywords=yes&area=default>`_
| `Short Range <../search.html?q=DataApplicationShortRange&check_keywords=yes&area=default>`_
| `Space Weather <../search.html?q=DataApplicationSpaceWeather&check_keywords=yes&area=default>`_
| `Subseasonal to Seasonal <../search.html?q=DataApplicationS2S&check_keywords=yes&area=default>`_
| `Tropical Cyclone and Extra-Tropical Cyclone <../search.html?q=DataApplicationTCandExtraTC&check_keywords=yes&area=default>`_
File renamed without changes
File renamed without changes
File renamed without changes
File renamed without changes
File renamed without changes
File renamed without changes

This file was deleted.

3 changes: 3 additions & 0 deletions docs/use_cases/model_applications/s2s_mid_lat/README.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
Subseasonal to Seasonal: Mid-Latitude
-------------------------------------
Subseasonal-to-Seasonal model configurations relating to middle latitudes
Loading

0 comments on commit a20c0bb

Please sign in to comment.