-
Notifications
You must be signed in to change notification settings - Fork 120
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[develop] Split EnsembleStat tasks into GenEnsProd and EnsembleStat #695
Closed
gsketefian
wants to merge
125
commits into
ufs-community:develop
from
gsketefian:feature/split_gepes
Closed
[develop] Split EnsembleStat tasks into GenEnsProd and EnsembleStat #695
gsketefian
wants to merge
125
commits into
ufs-community:develop
from
gsketefian:feature/split_gepes
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
… for Hera and modify run_WE2E_tests.py so that this location is obtained in a platform-independent way.
…ation of staged forecast files; modify code to correctly set VX_FCST_INPUT_BASEDIR in WE2E experiments.
…o that it is exactly consistent with the one used in run_WE2E_tests.py.
…t each task has its own values for NNODES, PPN, MEM, WTIME, MAXTRIES.
…t the DTC team wants.
…uired by NCO); also rename vx task top-level sections in config_defaults.yaml so that they're all lower case (to match names of ex-scripts).
…ks; Add ensemble member name (if running ensemble vx) to the names of the vx tasks in the ROCOTO xml.
…t convention; fix bugs in the two ex-scripts exregional_run_met_pointstat_ens[mean|prob].sh that call METplus twice.
…output. Details below. In the ex-scripts for EnsembleStat: (1) generate a new variable FCST_INPUT_FN_TEMPLATE that contains a list of forecast input template files for each of the ensemble members and use this in the METplus configuration files, and (2) change the input base directory (INPUT_BASE) to use VX_FCST_INPUT_BASEDIR, which depending on whether forecasts are being run is set either to the location of the UPP output from the forecasts or to the staged forecast UPP output (this allows staged ensemble staged files to be used to perform ensemble vx). In config_defaults.yaml, introduce new template variables (FCST_SUBDIR_TEMPLATE, FCST_FN_TEMPLATE, and FCST_FN_METPROC_TEMPLATE) that are/will be used in forming the variable FCST_INPUT_FN_TEMPLATE in the EnsembleStat ex-scripts.
…leStat for point-obs, ensure that METplus is called only once and for the correct variable (instead of for both SFC and UPA).
… This should later be replaced with a python version.
… This should later be replaced with a python version.
…able the ensemble member name (if doing an ensemble forecast). This makes it easier in METviewer to identify curves corresponding to different members.
…_LOCAL_MODULE_FN and VX_LOCAL_MODULE_FN; fix spacing.
…P to templates and add necessary variables to correspondig ROCOTO template XML tasks.
…o templates and add necessary variables to the correspondig ROCOTO template XML tasks.
…e METplus conf files.
…semble verification with staged forecasts.
…s on Hera. This test runs ensemble vx from staged forecast and obs.
…f the MET/METplus tools; remove "met_tool_[sc|pc]" arguments to the vx ex-scripts and directly use the environment variable MET_TOOL along with the new function (set_met_tool_name) to set the MET/METplus tool names needed in the vx tasks.
… now done in GenEnsProd (to avoid duplication).
…e output of GenEnsProd (instead of EnsembleStat, which no longer generates the necessary outputs).
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
DESCRIPTION OF CHANGES:
IMPORTANT: This PR builds on PR #683. Thus, that one has to be merged first. Currently, the changes shown in this PR also include the ones from PR #683. Once #683 is merged, the number of files changed will be far fewer (maybe around 35, all except 2 or 3 of which are the METplus conf files).
The latest versions of MET/METplus have introduced a new tool named
GenEnsProd
that performs many of the ensemble-related calculations thatEnsembleStat
used to do. AlthoughEnsembleStat
can still perform these calculations, that will change in future versions of MET/METplus. Thus, this PR creates new vx tasks that callGenEnsProd
to perform the necessary calculations, removing them from theEnsembleStat
tasks. Details:GenEnsProd
.EnsembleStat
config files so thatEnsembleStat
no longer calculates the ensemble statistics that are now the purview ofGenEnsProd
.config_defaults.yaml
for the newGenEnsProd
tasks.GenEnsProd
. Note that these are identical to the existingEnsembleStat
tasks (they call the same j-job, define the same environment variables, etc) except for a change of task name and resource names (number of nodes, walltime, etc). Thus, they are very "loopable".GenEnsProd
tasks, not the output of theEnsembleStat
tasks.Type of change
TESTS CONDUCTED:
I ran the following
fundamental
suite of tests on Hera with intel, and all passed:In addition, I ran the
MET_ensemble_verification
test in NCO mode, and that also passed.DEPENDENCIES:
This PR builds on PR #683. Thus, that has to be merged first.
DOCUMENTATION:
Documentation is needed but will be added once the whole set of vx PRs has been merged (see Issue #630).
CHECKLIST