Internal: Add support for creating multiple input datasets for use case categories

[georgemccabe]

Currently the automated tests are set up so that each model_applications category corresponds to an input data set that contains all of the data required to run all of the use cases in that category. The s2s input data set has become so large that while it doesn't exceed the maximum allowable size for the Docker data volume that stores it to use in the tests, but use case test groups that use this data run out of disk space when they write output data from the use cases.

The use case groups that fail also use the Conda environments required for METplotpy and METcalcpy, which are very large in size due to the many Python package dependencies. This also contributes to the total disk size that can be used in the test environment. The newly created metplotpy environment for #1566 is much larger in size than the existing environment, so this may cause disk space issues when that work is completed.

Size of current conda environments:

du -sh /usr/local/envs/*
897M    /usr/local/envs/metplotpy
185M    /usr/local/envs/metplus_base

Size of conda environments using Python 3.8.6 and updated package requirements:

du -sh /usr/local/envs/*
2.2G    /usr/local/envs/metplotpy.v5
168M    /usr/local/envs/metplus_base.v5

We may need to reconsider new requirements of use cases and how to group them in the tests, including:

• Size of input data
• Size out output data generated
• Size of conda environment required to run
• Others?

Describe the Enhancement

• Come up with a good naming convention for the additional input data sets. Currently they are named after the category, i.e. s2s. We will need another data set such as s2s_2.
• Update the automated test logic to support multiple input data sets for a given category
• Update the Contributor's Guide Add Use Cases chapter with the updated process for adding new data
• Update User's Guide to describe how to find input data for use cases since they will not just correspond to the model_applications sub-directory name anymore

Time Estimate

1-3 days

Sub-Issues

Consider breaking the enhancement down into sub-issues.

• Add a checkbox for each sub-issue here.

Relevant Deadlines

ASAP

Funding Source

2702691 2792541

Define the Metadata

Assignee

• Select engineer(s) or no engineer required
• Select scientist(s) or no scientist required

Labels

• Select component(s)
• Select priority
• Select requestor(s)

Projects and Milestone

• Select Repository and/or Organization level Project(s) or add alert: NEED PROJECT ASSIGNMENT label
• Select Milestone as the next official version or Future Versions

Define Related Issue(s)

Consider the impact to the other METplus components.

• METplus, MET, METdatadb, METviewer, METexpress, METcalcpy, METplotpy

Enhancement Checklist

See the METplus Workflow for details.

• Complete the issue definition above, including the Time Estimate and Funding Source.
• Fork this repository or create a branch of develop.
  Branch name: feature_<Issue Number>_<Description>
• Complete the development and test your changes.
• Add/update log messages for easier debugging.
• Add/update unit tests.
• Add/update documentation.
• Add any new Python packages to the METplus Components Python Requirements table.
• Push local changes to GitHub.
• Submit a pull request to merge into develop.
  Pull request: feature <Issue Number> <Description>
• Define the pull request metadata, as permissions allow.
  Select: Reviewer(s) and Linked issues
  Select: Repository level development cycle Project for the next official release
  Select: Milestone as the next official version
• Iterate until the reviewer(s) accept and merge your changes.
• Delete your fork or branch.
• Close this issue.

[hankenstein2]

Look into paying for more disk space to host bigger data sets
Look into hosting a TDS(Thredds) server to dynamically host data, i.e. just grab what you want.
Probably need to split up datasets more starting with S2S

[hankenstein2]

Try to split s2s data into two parts that are two descriptive s2s categories, i.e. s2s_ocean or s2s_fubar

[hankenstein2]

Phase, OMI, RMM can be grouped (7-10) NJO
Blocking, Weather Regime (1-3, 11) Blocking
s2s_mjo
s2s_mid_lat
s2s - for all the rest (0,4-6,12-14)

[georgemccabe]

Proposed groupings for splitting up s2s use cases

s2s_mjo

• UserScript_obsERA_obsOnly_PhaseDiagram
• UserScript_fcstGFS_obsERA_OMI
• UserScript_obsERA_obsOnly_OMI
• UserScript_obsERA_obsOnly_RMM

s2s_mid_lat

• UserScript_fcstGFS_obsERA_Blocking
• UserScript_obsERA_obsOnly_Blocking
• UserScript_obsERA_obsOnly_WeatherRegime
• UserScript_fcstGFS_obsERA_WeatherRegime

s2s

• GridStat_SeriesAnalysis_fcstNMME_obsCPC_seasonal_forecast
• TCGen_fcstGFSO_obsBDECKS_GDF_TDF
• UserScript_obsPrecip_obsOnly_Hovmoeller
• UserScript_obsPrecip_obsOnly_CrossSpectraPlot
• UserScript_obsERA_obsOnly_Stratosphere
• SeriesAnalysis_fcstCFSv2_obsGHCNCAMS_climoStandardized_MultiStatisticTool
• GridStat_fcstCFSv2_obsGHCNCAMS_MultiTercile

Should the s2s group have an additional identifier? Should/can this group be divided into smaller groups?

[TaraJensen]

@CPKalb @georgemccabe @j-opatz - please run these stratifications past CPC, PSL, etc... to see if they make sense to them. After all - they are the S2S community we are serving. Thanks!

[CPKalb]

I just heard back from Maria, and she thinks s2s_mjo is great! I'll let you know when I hear back about the blocking weather regime

[CPKalb]

I just heard back from Doug, and he thinks s2s_mid_lat is a good name as well.