Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature 1694 s2s subgroups #1697

Merged
merged 28 commits into from
Jul 18, 2022
Merged
Show file tree
Hide file tree
Changes from 24 commits
Commits
Show all changes
28 commits
Select commit Hold shift + click to select a range
900443a
per #1694, moved 4 use cases from s2s to s2s_mjo, updated paths accor…
georgemccabe Jul 12, 2022
12bcfba
per #1694, fixed paths to s2s_mjo conf files
georgemccabe Jul 12, 2022
8b00df5
updated documentation for use cases that were moved from s2s to s2s_mjo
georgemccabe Jul 12, 2022
bddfc05
attempt to free up unused disk space in GHA runner environment
georgemccabe Jul 13, 2022
7122d44
moved 4 s2s use cases into s2s_mid_lat
georgemccabe Jul 13, 2022
8a1cd74
added new model application categories to contrib guide for adding ne…
georgemccabe Jul 13, 2022
f37dfaf
per #947, changed convection_allowing_models use cases to short_range
georgemccabe Jul 13, 2022
fe64437
changed which use case tests run to the ones that are failing and add…
georgemccabe Jul 13, 2022
1e8cd33
test to determine which files are preventing MySQL database from bein…
georgemccabe Jul 13, 2022
7da856a
test 2 to determine which files are preventing MySQL database from be…
georgemccabe Jul 13, 2022
d14b4ee
test 3 to see if removing these files is not the cause of the METdbLo…
georgemccabe Jul 13, 2022
00b1530
updated references to METdatadb to METdataio since the repository was…
georgemccabe Jul 13, 2022
743a371
fixed typo
georgemccabe Jul 13, 2022
50e94d7
changed path to sql file needed to create database because it was mov…
georgemccabe Jul 13, 2022
851732a
fixed path to sql file that was moved from METviewer to METdataio
georgemccabe Jul 13, 2022
3950eed
removed temporary fix because metdataio conda env was created in the …
georgemccabe Jul 13, 2022
c8accba
added note to update path when METviewer Dockerfile changes to reflec…
georgemccabe Jul 14, 2022
528abee
fixed path to METdataio repo
georgemccabe Jul 14, 2022
e669be2
add back commands to free up disk space because issue with METdbLoad …
georgemccabe Jul 14, 2022
2275076
run all tests with ci-run-all-diff
georgemccabe Jul 14, 2022
d4eea0c
remove use case group added for testing, ci-skip-all
georgemccabe Jul 14, 2022
c3ba81e
changed exit code for diff tests to 2 so it is easier to see if a use…
georgemccabe Jul 14, 2022
f52cb99
changed grouping of s2s mid lat use cases to original grouping to pre…
georgemccabe Jul 14, 2022
9dd8dfe
merged develop and resolved conflicts
georgemccabe Jul 18, 2022
89ea557
per #1694, changed all references to convection allowing models to sh…
georgemccabe Jul 18, 2022
2675c31
changed URLs to develop version of documentation to a URL relative to…
georgemccabe Jul 18, 2022
36a6d06
per #947, changed references to convection_allowing_model (without th…
georgemccabe Jul 18, 2022
d451f40
updated use case test scripts to rename convection_allowing_models to…
georgemccabe Jul 18, 2022
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .github/jobs/free_disk_space.sh
Original file line number Diff line number Diff line change
Expand Up @@ -11,4 +11,5 @@ sudo rm -rf "/usr/local/share/boost"
sudo rm -rf "$AGENT_TOOLSDIRECTORY"

printf "\nChecking disk usage after cleanup"

df -h
2 changes: 1 addition & 1 deletion .github/jobs/get_metviewer.sh
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ sleep 20
# print list of currently running containers to
# verify mysql and metviewer are running
docker ps -a

# TODO: when METviewer Dockerfile changes METdatadb to METdataio, update path to sql file accordingly
# commands to run inside METviewer container
cmd="mysql -hmysql_mv -uroot -pmvuser -e\"create database mv_metplus_test;\";"
cmd+="mysql -hmysql_mv -uroot -pmvuser mv_metplus_test < /METviewer-python/METdataio/METdbLoad/sql/mv_mysql.sql"
Expand Down
8 changes: 4 additions & 4 deletions .github/jobs/get_use_case_commands.py
Original file line number Diff line number Diff line change
Expand Up @@ -72,13 +72,13 @@ def handle_automation_env(host_name, reqs, work_dir):
'cd -;'
)

# if metdatadb is in requirements list, add command to obtain METdatadb
if 'metdatadb' in str(reqs).lower():
# if metdataio is in requirements list, add command to obtain METdataio
if 'metdataio' in str(reqs).lower():
setup_env += (
f'cd {METPLUS_DOCKER_LOC};'
f'{work_dir}/manage_externals/checkout_externals'
f' -e {work_dir}/.github/parm/Externals_metdatadb.cfg;'
f'{python_path} -m pip install {METPLUS_DOCKER_LOC}/../METdatadb;'
f' -e {work_dir}/.github/parm/Externals_metdataio.cfg;'
f'{python_path} -m pip install {METPLUS_DOCKER_LOC}/../METdataio;'
'cd -;'
)

Expand Down
2 changes: 1 addition & 1 deletion .github/jobs/run_difference_tests.sh
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ if [ "$( ls -A ${RUNNER_WORKSPACE}/diff)" ]; then
echo ::set-output name=upload_diff::true
mkdir -p artifact/diff-${artifact_name}
cp -r ${RUNNER_WORKSPACE}/diff/* artifact/diff-${artifact_name}
exit 1
exit 2
fi

echo ::set-output name=upload_diff::false
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
[METdatadb]
local_path = ../METdatadb
local_path = ../METdataio
protocol = git
required = True
repo_url = https://github.com/dtcenter/METdatadb
repo_url = https://github.com/dtcenter/METdataio
branch = develop

[externals_description]
Expand Down
42 changes: 21 additions & 21 deletions .github/parm/use_case_groups.json
Original file line number Diff line number Diff line change
Expand Up @@ -25,32 +25,32 @@
"run": false
},
{
"category": "convection_allowing_models",
"category": "short_range",
"index_list": "0",
"run": false
},
{
"category": "convection_allowing_models",
"category": "short_range",
"index_list": "1",
"run": false
},
{
"category": "convection_allowing_models",
"category": "short_range",
"index_list": "2-6",
"run": false
},
{
"category": "convection_allowing_models",
"category": "short_range",
"index_list": "7",
"run": false
},
{
"category": "convection_allowing_models",
"category": "short_range",
"index_list": "8",
"run": false
},
{
"category": "convection_allowing_models",
"category": "short_range",
"index_list": "9",
"run": false
},
Expand Down Expand Up @@ -136,52 +136,52 @@
},
{
"category": "s2s",
"index_list": "1-3",
"index_list": "1",
"run": false
},
{
"category": "s2s",
"index_list": "4",
"index_list": "2",
"run": false
},
{
"category": "s2s",
"index_list": "5",
"index_list": "3",
"run": false
},
{
"category": "s2s",
"index_list": "6-7",
"index_list": "4",
"run": false
},
{
"category": "s2s",
"index_list": "7-9",
"index_list": "5",
"run": false
},
{
"category": "s2s",
"index_list": "10",
"index_list": "6",
"run": false
},
{
"category": "s2s",
"index_list": "11",
"category": "s2s_mid_lat",
"index_list": "0-2",
"run": false
},
{
"category": "s2s",
"index_list": "12",
"run": false
"category": "s2s_mid_lat",
"index_list": "3",
"run": false
},
{
"category": "s2s",
"index_list": "13",
"category": "s2s_mjo",
"index_list": "0-2",
"run": false
},
{
"category": "s2s",
"index_list": "14",
"category": "s2s_mjo",
"index_list": "3",
"run": false
},
{
Expand Down
4 changes: 3 additions & 1 deletion docs/Contributors_Guide/add_use_case.rst
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,6 @@ one of the following:

* air_quality_and_comp
* climate
* convection_allowing_models
* data_assimilation
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Need to add in "short_range" to replace the "convection_allowing_models".

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You are looking at the "before" text. This change is removing convection_allowing_models and adding short_range below.

* extremes
* land_surface
Expand All @@ -52,6 +51,9 @@ one of the following:
* pbl
* precipitation
* s2s (Subseasonal to Seasonal)
* s2s_mid_lat (Subseasonal to Seasonal: Mid-Latitude)
* s2s_mjo (Subseasonal to Seasonal: Madden-Julian Oscillation)
* short_range (formerly convection_allowing_models)
* space_weather
* tc_and_extra_tc (Tropical Cyclone and Extratropical Cyclone)

Expand Down
8 changes: 6 additions & 2 deletions docs/Users_Guide/quicksearch.rst
Original file line number Diff line number Diff line change
Expand Up @@ -70,28 +70,32 @@ Use Cases by Application:

| `Air Quality and Composition <../search.html?q=AirQualityAndCompAppUseCase&check_keywords=yes&area=default>`_
| `Climate <../search.html?q=ClimateAppUseCase&check_keywords=yes&area=default>`_
| `Convection Allowing Models <../search.html?q=ConvectionAllowingModelsAppUseCase&check_keywords=yes&area=default>`_
| `Short Range <../search.html?q=ShortRangeAppUseCase&check_keywords=yes&area=default>`_
| `Data Assimilation <../search.html?q=DataAssimilationAppUseCase&check_keywords=yes&area=default>`_
| `Ensemble <../search.html?q=EnsembleAppUseCase&check_keywords=yes&area=default>`_
| `Marine and Cryosphere <../search.html?q=MarineAndCryosphereAppUseCase&check_keywords=yes&area=default>`_
| `Medium Range <../search.html?q=MediumRangeAppUseCase&check_keywords=yes&area=default>`_
| `Precipitation <../search.html?q=PrecipitationAppUseCase&check_keywords=yes&area=default>`_
| `Space Weather <../search.html?q=SpaceWeatherAppUseCase&check_keywords=yes&area=default>`_
| `Subseasonal to Seasonal <../search.html?q=S2SAppUseCase&check_keywords=yes&area=default>`_
| `Subseasonal to Seasonal: Madden-Julian Oscillation <../search.html?q=S2SMJOAppUseCase&check_keywords=yes&area=default>`_
| `Subseasonal to Seasonal: Mid-Latitude <../search.html?q=S2SMidLatAppUseCase&check_keywords=yes&area=default>`_
| `Tropical Cyclone and Extra-Tropical Cyclone <../search.html?q=TCandExtraTCAppUseCase&check_keywords=yes&area=default>`_

.. only:: latex

| **Air Quality and Composition**: *AirQualityAndCompAppUseCase*
| **Climate**: *ClimateAppUseCase*
| **Convection Allowing Models**: *ConvectionAllowingModelsAppUseCase*
| **Short Range**: *ShortRangeAppUseCase*
| **Data Assimilation**: *DataAssimilationAppUseCase*
| **Ensemble**: *EnsembleAppUseCase*
| **Marine and Cryosphere**: *MarineAndCryosphereAppUseCase*
| **Medium Range**: *MediumRangeAppUseCase*
| **Precipitation**: *PrecipitationAppUseCase*
| **Space Weather**: *SpaceWeatherAppUseCase*
| **Subseasonal to Seasonal**: *S2SAppUseCase*
| **Subseasonal to Seasonal: Madden-Julian Oscillation**: *S2SMJOAppUseCase*
| **Subseasonal to Seasonal: Mid-Latitude**: *S2SMidLatAppUseCase*
| **Tropical Cyclone and Extra-Tropical Cyclone**: *TCandExtraTCAppUseCase*

Use Cases by Organization:
Expand Down

This file was deleted.

3 changes: 3 additions & 0 deletions docs/use_cases/model_applications/s2s_mid_lat/README.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
Subseasonal to Seasonal: Mid-Latitude
-------------------------------------
Subseasonal-to-Seasonal model configurations relating to middle latitudes
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
========================================================================================

model_applications/
s2s/
s2s_mid_lat/
UserScript_fcstGFS_obsERA_Blocking.py

"""
Expand Down Expand Up @@ -82,13 +82,13 @@
#
# METplus first loads all of the configuration files found in parm/metplus_config,
# then it loads any configuration files passed to METplus via the command line
# i.e. parm/use_cases/model_applications/s2s/UserScript_fcstGFS_obsERA_Blocking.py.
# i.e. parm/use_cases/model_applications/s2s_mid_lat/UserScript_fcstGFS_obsERA_Blocking.py.
# The file UserScript_fcstGFS_obsERA_Blocking.conf runs the python program, and the
# variables for all steps of the Blocking calculation are given in the [user_env_vars]
# section of the .conf file.
#
# .. highlight:: bash
# .. literalinclude:: ../../../../parm/use_cases/model_applications/s2s/UserScript_fcstGFS_obsERA_Blocking.conf
# .. literalinclude:: ../../../../parm/use_cases/model_applications/s2s_mid_lat/UserScript_fcstGFS_obsERA_Blocking.conf
#

##############################################################################
Expand All @@ -112,7 +112,7 @@
#
# This use case uses Python scripts to perform the blocking calculation
#
# parm/use_cases/model_applications/s2s/UserScript_fcstGFS_obsERA_Blocking/Blocking_driver.py:
# parm/use_cases/model_applications/s2s_mid_lat/UserScript_fcstGFS_obsERA_Blocking/Blocking_driver.py:
# This script calls the requested steps in the blocking analysis for a forecast, observation, or both. The possible
# steps are computing CBLs, plotting CBLs, computing IBLs, plotting IBLs, computing GIBLs, computing blocks, and
# plotting blocks.
Expand All @@ -127,7 +127,7 @@
# pre-processing steps. See the METcalcpy `Utility script <https://github.com/dtcenter/METcalcpy/blob/develop/metcalcpy/contributed/blocking_weather_regime/Blocking_WeatherRegime_util.py>`_ for more information.
#
# .. highlight:: python
# .. literalinclude:: ../../../../parm/use_cases/model_applications/s2s/UserScript_fcstGFS_obsERA_Blocking/Blocking_driver.py
# .. literalinclude:: ../../../../parm/use_cases/model_applications/s2s_mid_lat/UserScript_fcstGFS_obsERA_Blocking/Blocking_driver.py
#

##############################################################################
Expand All @@ -138,11 +138,11 @@
#
# 1) Passing in UserScript_fcstGFS_obsERA_Blocking.py then a user-specific system configuration file::
#
# run_metplus.py -c /path/to/METplus/parm/use_cases/model_applications/s2s/UserScript_fcstGFS_obsERA_Blocking.py -c /path/to/user_system.conf
# run_metplus.py -c /path/to/METplus/parm/use_cases/model_applications/s2s_mid_lat/UserScript_fcstGFS_obsERA_Blocking.py -c /path/to/user_system.conf
#
# 2) Modifying the configurations in parm/metplus_config, then passing in UserScript_fcstGFS_obsERA_Blocking.py::
#
# run_metplus.py -c /path/to/METplus/parm/use_cases/model_applications/s2s/UserScript_fcstGFS_obsERA_Blocking.py
# run_metplus.py -c /path/to/METplus/parm/use_cases/model_applications/s2s_mid_lat/UserScript_fcstGFS_obsERA_Blocking.py
#
# The following variables must be set correctly:
#
Expand All @@ -163,7 +163,7 @@
# ---------------
#
# Refer to the value set for **OUTPUT_BASE** to find where the output data was generated. Output for this use
# case will be found in model_applications/s2s/Blocking (relative to **OUTPUT_BASE**) and will contain output
# case will be found in model_applications/s2s_mid_lat/Blocking (relative to **OUTPUT_BASE**) and will contain output
# for the steps requested. This may include the regridded data, daily averaged files, running mean files,
# and anomaly files. In addition, output CBL, IBL, and Blocking frequency plots can be generated. The location
# of these output plots can be specified as BLOCKING_PLOT_OUTPUT_DIR. If it is not specified, plots will be sent
Expand All @@ -184,11 +184,12 @@
# * PCPCombineUseCase
# * StatAnalysisUseCase
# * S2SAppUseCase
# * S2SMidLatAppUseCase
# * NetCDFFileUseCase
# * GRIB2FileUseCase
#
# Navigate to the :ref:`quick-search` page to discover other similar use cases.
#
#
#
# sphinx_gallery_thumbnail_path = '_static/s2s-IBL_frequency.png'
# sphinx_gallery_thumbnail_path = '_static/s2s_mid_lat-UserScript_fcstGFS_obsERA_Blocking.png'
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
==================================================================================================

model_applications/
s2s/
s2s_mid_lat/
UserScript_fcstGFS_obsERA_WeatherRegime.py

"""
Expand Down Expand Up @@ -93,12 +93,12 @@
#
# METplus first loads all of the configuration files found in parm/metplus_config,
# then it loads any configuration files passed to METplus via the command line
# i.e. parm/use_cases/model_applications/s2s/UserScript_fcstGFS_obsERA_WeatherRegime.py.
# i.e. parm/use_cases/model_applications/s2s_mid_lat/UserScript_fcstGFS_obsERA_WeatherRegime.py.
# The file UserScript_fcstGFS_obsERA_WeatherRegime.conf runs the python program and
# sets the variables for all steps of the Weather Regime use case including data paths.
#
# .. highlight:: bash
# .. literalinclude:: ../../../../parm/use_cases/model_applications/s2s/UserScript_fcstGFS_obsERA_WeatherRegime.conf
# .. literalinclude:: ../../../../parm/use_cases/model_applications/s2s_mid_lat/UserScript_fcstGFS_obsERA_WeatherRegime.conf
#

##############################################################################
Expand All @@ -121,7 +121,7 @@
#
# This use case uses Python scripts to perform the blocking calculation
#
# parm/use_cases/model_applications/s2s/UserScript_fcstGFS_obsERA_WeatherRegime/WeatherRegime_driver.py:
# parm/use_cases/model_applications/s2s_mid_lat/UserScript_fcstGFS_obsERA_WeatherRegime/WeatherRegime_driver.py:
# This script calls the requested steps in the blocking analysis for a forecast, observation, or both. The possible
# steps are computing the elbow, computing EOFs, and computing weather regimes using k means clustering.
#
Expand All @@ -135,7 +135,7 @@
# pre-processing steps. See the METcalcpy `Utility script <https://github.com/dtcenter/METcalcpy/blob/develop/metcalcpy/contributed/blocking_weather_regime/Blocking_WeatherRegime_util.py>`_ for more information.
#
# .. highlight:: python
# .. literalinclude:: ../../../../parm/use_cases/model_applications/s2s/UserScript_fcstGFS_obsERA_WeatherRegime/WeatherRegime_driver.py
# .. literalinclude:: ../../../../parm/use_cases/model_applications/s2s_mid_lat/UserScript_fcstGFS_obsERA_WeatherRegime/WeatherRegime_driver.py
#

##############################################################################
Expand All @@ -146,11 +146,11 @@
#
# 1) Passing in UserScript_fcstGFS_obsERA_WeatherRegime.py then a user-specific system configuration file::
#
# run_metplus.py -c /path/to/METplus/parm/use_cases/model_applications/s2s/UserScript_fcstGFS_obsERA_WeatherRegime.py -c /path/to/user_system.conf
# run_metplus.py -c /path/to/METplus/parm/use_cases/model_applications/s2s_mid_lat/UserScript_fcstGFS_obsERA_WeatherRegime.py -c /path/to/user_system.conf
#
# 2) Modifying the configurations in parm/metplus_config, then passing in UserScript_fcstGFS_obsERA_WeatherRegime.py::
#
# run_metplus.py -c /path/to/METplus/parm/use_cases/model_applications/s2s/UserScript_fcstGFS_obsERA_WeatherRegime.py
# run_metplus.py -c /path/to/METplus/parm/use_cases/model_applications/s2s_mid_lat/UserScript_fcstGFS_obsERA_WeatherRegime.py
#
# The following variables must be set correctly:
#
Expand All @@ -171,7 +171,7 @@
# ---------------
#
# Refer to the value set for **OUTPUT_BASE** to find where the output data was generated. Output for this use
# case will be found in model_applications/s2s/WeatherRegime (relative to **OUTPUT_BASE**) and will contain output
# case will be found in model_applications/s2s_mid_lat/WeatherRegime (relative to **OUTPUT_BASE**) and will contain output
# for the steps requested. This may include the regridded data, daily averaged files, a text file containing the
# list of input files, and text files for the weather regime classification and time frequency (if KMEANS and
# TIMEFREQ are run for both the forecast and observation data). In addition, output elbow, EOF, and Kmeans weather
Expand All @@ -193,11 +193,12 @@
# * PCPCombineUseCase
# * StatAnalysisUseCase
# * S2SAppUseCase
# * S2SMidLatAppUseCase
# * NetCDFFileUseCase
# * GRIB2FileUseCase
#
# Navigate to the :ref:`quick-search` page to discover other similar use cases.
#
#
#
# sphinx_gallery_thumbnail_path = '_static/s2s-OBS_ERA_weather_regime.png'
# sphinx_gallery_thumbnail_path = '_static/s2s_mid_lat-UserScript_fcstGFS_obsERA_WeatherRegime.png'
Loading