[production/AQM.v7] Code merge for ecflow updates into the production/AQM.v7 #cf184119 #812

lgannoaa · 2023-05-25T17:57:14Z

Background:
This merge is for Github issue #797 to bring the current development version ecflow updates into the production/AQM.v7 #cf184119. This version of the ecflow is not to be used for NCO code delivery.
There will be incremental updates following this merge. Once the AQM package is ready for implementation, the ecflow workflow will be configured for NCO code delivery in a later update.

Condition of the production/AQM.v7 #cf184119:
Here is a list of known issues from the production/AQM.v7 #cf184119:

forecast restart does not reproduce.
forecast job does not create field_table, data_table, and input.nml configuration file within its DATA directory. It reads these configuration files from the workflow configuration directory. There are files will need to be modified if forecast job configuration change is necessary.
data_table file used in the forecast job is a size zero file.
emission jobs have known internal issues which result in incorrect output from the science point of view.
lbcs job executable has known unstable silent failed condition.
bias correction jobs unable to reproduce due to exception handling issues.
There are four jobs that write output in its DATA location for other jobs to access. Therefore these four jobs can not remove the DATA when the job is completed. There are nexus_gfs_sfc, forecast, get_extrn_ics, and get_extrn_lbcs

The scrope of this merge:
This merge is to bring AQM ecflow workflow into the production/AQM.v7.
The merged AQM ecflow workflow will have all jobs completed with exit code 0 status.
This merge does not address AQM package internal issues outlined in the condition of the production/AQM.v7 #cf184119 session above.
The goal of this merge is not to fix EE2 issues of the AQM package.

Change log:

Adding ecflow code delivery instruction in docs
Adding ecflow workflow within AQM package in ecf directory
Adding ecflow workflow control directory in parm/config
Modified ush/preamble.sh to enable TRACE for NCO
Modified ush/machine/wcoss2.yaml for EMC realtime
Modified the ush/config.aqm.nco.realtime.yaml to use NCO standard of KEEPDATA as YES/NO
Modified the ush/config_defaults.yaml to:
Use NCO standard of KEEPDATA.
Fixed bug with COMINgefs configuration.
Match COMINbicor and COMOUTbicor with AQM realtime parallel
Remove extra AQM_AIRNOW_HIST_DIR
Modified ush/job_preamble.sh to:
Use updated share_pid for DATA directory assignment.
Remove unnecessary COMROOT assignment for nco environment.
Remove unnecessary KEEPDATA logic.
The DATA cleanup logic in job_postamble function does not work for the NCO environment during some emergency, production switch, and rerun conditions. Modified job_postamble function to do DATA clean up for those jobs that called it. There are four jobs that use DATA store output files for other jobs to access. These four jobs will not be using job_postamble to clean out the DATA.
Modified versions/run.ver.wcoss2 to include aqm_ver and arrange the model version in the head of the file.
Add jobs/JAQM_MANAGER to enable EMC retro development parallel
Add jobs/JDATA_CLEANUP to enable NCO DATA cleanup for job nexus_gfs_sfc, forecast, get_extrn_ics, and get_extrn_lbcs.
Modified the following J-Jobs to use COMIN assigned by NCO ecflow workflow for INPUT_DATA value within NCO environment
jobs/JREGIONAL_AQM_ICS
jobs/JREGIONAL_AQM_LBCS
jobs/JREGIONAL_MAKE_ICS
jobs/JREGIONAL_MAKE_LBCS
jobs/JREGIONAL_NEXUS_EMISSION
jobs/JREGIONAL_NEXUS_POST_SPLIT
jobs/JREGIONAL_POINT_SOURCE
Modified the following J-Jobs to use necessary environment variables.
jobs/JREGIONAL_BIAS_CORRECTION_O3
jobs/JREGIONAL_BIAS_CORRECTION_PM25
jobs/JREGIONAL_FIRE_EMISSION
jobs/JREGIONAL_POST_STAT_O3
jobs/JREGIONAL_POST_STAT_PM25
Modified jobs/JREGIONAL_GET_EXTRN_MDL_FILES:
Fixed application logic for ICS_OR_LBCS exception handling issue.
Updated staging directory location naming to be consistent.
Removed from DATA cleaning function job_postamble because the DATA of this job will be removed from DATA_CLEANUP job.
Modified jobs/JREGIONAL_NEXUS_GFS_SFC:
Removed from DATA cleaning function job_postamble because the DATA of this job will be removed from DATA_CLEANUP job.
Modified jobs/JREGIONAL_RUN_FCST:
Removed from DATA cleaning function job_postamble because the DATA of this job will be removed from DATA_CLEANUP job.
Modified jobs/JREGIONAL_RUN_POST:
Use DATAFCST from workflow assignment if default is set.
Fixed application logic for FCST_LEN_HRS value from array assignment.
Removed wired in rocoto developer logic for file *_task_complete.txt
Update application logic to use job_postamble.
Add scripts/exaqm_manager.sh for j-job JAQM_MANAGER
Add scripts/exdata_cleanup.sh for j-job JDATA_CLEANUP
Modified scripts/exregional_aqm_lbcs.sh:
Fixed application logic in FCST_LEN_HRS assignment.
Fixed AQM_MOFILE_FP path assignment issue.
Modified scripts/exregional_bias_correction_o3.sh
Fixed application logic in FCST_LEN_HRS assignment.
Replace AQM_AIRNOW_HIST_DIR with NCO standard COMINbicor
Fixed application logic in write output file
Modified scripts/exregional_bias_correction_pm25.sh
Fixed application logic in FCST_LEN_HRS assignment.
Replace AQM_AIRNOW_HIST_DIR with NCO standard COMINbicor
Fixed application logic in write output file
Fixed application logic in FCST_LEN_HRS assignment.
scripts/exregional_get_extrn_mdl_files.sh
scripts/exregional_nexus_gfs_sfc.sh
scripts/exregional_nexus_post_split.sh
scripts/exregional_point_source.sh
scripts/exregional_post_stat_o3.sh
scripts/exregional_pre_post_stat.sh
scripts/exregional_run_fcst.sh
Modified scripts/exregional_make_ics.sh
Use NCO standard naming for DATA location
Modified scripts/exregional_make_lbcs.sh
Use NCO standard naming for DATA location
Fixed application logic in FCST_LEN_HRS assignment.
Modified scripts/exregional_nexus_emission.sh
Fixed application logic in FCST_LEN_HRS assignment.
Use NCO standard naming for DATA location
Modified scripts/exregional_run_post.sh
Use NCO standard naming for DATA location
Remove unconditional remove of the forecast DATA because it will be handled by data_cleanup job

TESTS CONDUCTED:

wcoss2.intel

CHECKLIST

My code follows the style guidelines in the Contributor's Guide
I have performed a self-review of my own code using the Code Reviewer's Guide
I have commented my code, particularly in hard-to-understand areas
My changes generate no new warnings

The scrope of this merge: This merge is to bring AQM ecflow workflow into the production/AQM.v7. This merge does not address AQM package internal issues or science error in job output. This merge does not fix all EE2 issues of the AQM package. Change log: - Adding ecflow code delivery instruction in docs. - Adding ecflow workflow within AQM package in ecf directory. - Adding ecflow workflow control directory in parm/config. - Modified ush/preamble.sh to enable TRACE for NCO. - Modified ush/machine/wcoss2.yaml for EMC realtime. - Modified the ush/config.aqm.nco.realtime.yaml to use NCO standard of KEEPDATA as YES/NO. - Modified the ush/config_defaults.yaml to: Use NCO standard of KEEPDATA. Fixed bug with COMINgefs configuration. Match COMINbicor and COMOUTbicor with AQM realtime parallel. Remove extra AQM_AIRNOW_HIST_DIR. - Modified ush/job_preamble.sh to: Use updated share_pid for DATA directory assignment. Remove unnecessary COMROOT assignment for nco environment. Remove unnecessary KEEPDATA logic. The DATA cleanup logic in job_postamble function does not work for the NCO environment during some emergency, production switch, and rerun conditions. Modified job_postamble function to do DATA clean up for those jobs that called it. There are four jobs that use DATA store output files for other jobs to access. These four jobs will not be using job_postamble to clean out the DATA. - Modified versions/run.ver.wcoss2 to include aqm_ver and arrange the model version in the head of the file. - Add jobs/JAQM_MANAGER to enable EMC retro development parallel. - Add jobs/JDATA_CLEANUP to enable NCO DATA cleanup for job nexus_gfs_sfc, forecast, get_extrn_ics, and get_extrn_lbcs. - Modified the following J-Jobs to use COMIN assigned by NCO ecflow workflow for INPUT_DATA value within NCO environment: jobs/JREGIONAL_AQM_ICS jobs/JREGIONAL_AQM_LBCS jobs/JREGIONAL_MAKE_ICS jobs/JREGIONAL_MAKE_LBCS jobs/JREGIONAL_NEXUS_EMISSION jobs/JREGIONAL_NEXUS_POST_SPLIT jobs/JREGIONAL_POINT_SOURCE - Modified the following J-Jobs to use necessary environment variables: jobs/JREGIONAL_BIAS_CORRECTION_O3 jobs/JREGIONAL_BIAS_CORRECTION_PM25 jobs/JREGIONAL_FIRE_EMISSION jobs/JREGIONAL_POST_STAT_O3 jobs/JREGIONAL_POST_STAT_PM25 - Modified jobs/JREGIONAL_GET_EXTRN_MDL_FILES: Fixed application logic for ICS_OR_LBCS exception handling issue. Updated staging directory location naming to be consistent. Removed from DATA cleaning function job_postamble because the DATA of this job will be removed from DATA_CLEANUP job. - Modified jobs/JREGIONAL_NEXUS_GFS_SFC: Removed from DATA cleaning function job_postamble because the DATA of this job will be removed from DATA_CLEANUP job. - Modified jobs/JREGIONAL_RUN_FCST: Removed from DATA cleaning function job_postamble because the DATA of this job will be removed from DATA_CLEANUP job. - Modified jobs/JREGIONAL_RUN_POST: Use DATAFCST from workflow assignment if default is set. Fixed application logic for FCST_LEN_HRS value from array assignment. Removed wired in rocoto developer logic for file *_task_complete.txt Update application logic to use job_postamble. - Add scripts/exaqm_manager.sh for j-job JAQM_MANAGER - Add scripts/exdata_cleanup.sh for j-job JDATA_CLEANUP - Modified scripts/exregional_aqm_lbcs.sh: Fixed application logic in FCST_LEN_HRS assignment. Fixed AQM_MOFILE_FP path assignment issue. - Modified scripts/exregional_bias_correction_o3.sh Fixed application logic in FCST_LEN_HRS assignment. Replace AQM_AIRNOW_HIST_DIR with NCO standard COMINbicor Fixed application logic in write output file - Modified scripts/exregional_bias_correction_pm25.sh Fixed application logic in FCST_LEN_HRS assignment. Replace AQM_AIRNOW_HIST_DIR with NCO standard COMINbicor Fixed application logic in write output file - Fixed application logic in FCST_LEN_HRS assignment. scripts/exregional_get_extrn_mdl_files.sh scripts/exregional_nexus_gfs_sfc.sh scripts/exregional_nexus_post_split.sh scripts/exregional_point_source.sh scripts/exregional_post_stat_o3.sh scripts/exregional_pre_post_stat.sh scripts/exregional_run_fcst.sh - Modified scripts/exregional_make_ics.sh Use NCO standard naming for DATA location - Modified scripts/exregional_make_lbcs.sh Use NCO standard naming for DATA location Fixed application logic in FCST_LEN_HRS assignment. - Modified scripts/exregional_nexus_emission.sh Fixed application logic in FCST_LEN_HRS assignment. Use NCO standard naming for DATA location - Modified scripts/exregional_run_post.sh Use NCO standard naming for DATA location Remove unconditional remove of the forecast DATA because it will be handled by data_cleanup job ufs-community#797

JianpingHuang-NOAA · 2023-05-26T13:04:22Z

Thanks Lin for the hard work

JianpingHuang-NOAA · 2023-05-26T14:31:06Z

@lgannoaa Can you add me now?

JianpingHuang-NOAA · 2023-05-26T14:39:03Z

The ecflow-base package has been tested by the rocoto workflow and both workflow show the identical results for model raw and bias correction grib2 files as well as wmo products.

ecf/scripts/aqm_manager/aqm_manager.ecf

ecf/scripts/aqm_manager/data_cleanup.ecf

chan-hoo · 2023-05-26T14:54:06Z

jobs/JREGIONAL_BIAS_CORRECTION_O3

@@ -19,6 +19,10 @@
 source_config_for_task "cpl_aqm_parm|task_run_post|task_bias_correction_o3" ${GLOBAL_VAR_DEFNS_FP}
 . $USHdir/job_preamble.sh
 #
+export DCOMROOT=${DCOMROOT:-/lfs/h1/ops/prod/dcom}
+export DCOMINairnow=${DCOMINairnow:-/lfs/h1/ops/prod/dcom}
+export COMINbicor=${COMINbicor:-$(compath.py ${NET}/${aqm_ver})}


@lgannoaa, please define DCOMIN in ush/machine/wcoss and COMIN in job_preamble.sh as others.

Looks to me, the default value have been defined in var_defns.sh.
grep -i "comin" var_defns.sh |wc -l
11

As can be seen in the j-job scripts, job_preamble.sh is sourced in the very beginning of the scripts. In this script, it is sourced at Line 20. Please put COMINbicor into job_preamble.sh. DCOMROOT has been already defined in config_defaults.yaml as DCOMROOT: '{{ OPSROOT }}/dcom'. You don't have do define it here again. DCOMINairnowis defined inush/machine/wcoss2.yaml`.

I think I see @chan-hoo 's point here - is there a reason why these variables can't be collected within the job_preamble.sh script sourced in the J-jobs?

chan-hoo · 2023-05-26T14:54:48Z

jobs/JREGIONAL_BIAS_CORRECTION_O3

@@ -67,6 +71,7 @@ if [ "${RUN_ENVIR}" = "community" ]; then
  cd $DATA
 fi

+COMOUTwmo="${COMOUTwmo:-${COMOUT}/wmo}"


@lgannoaa, COMOUTwmo is defined in job_preamble.sh.

Looks to me, COMOUTwmo is defaulted in job_preamble.sh
grep -i "export COMOUTwmo=" job_preamble.sh |wc -l
1
It is common in NCO production job to identify the environment export variable within the job.

The job_preamble.sh is sourced in the very beginning of each j-job script. This means it is defined within the j-job script. This line is just duplicated. Please check Line 20 in the same script.

chan-hoo · 2023-05-26T14:55:37Z

jobs/JREGIONAL_BIAS_CORRECTION_PM25

+export DCOMROOT=${DCOMROOT:-/lfs/h1/ops/prod/dcom}
+export DCOMINairnow=${DCOMINairnow:-/lfs/h1/ops/prod/dcom}
+export COMINbicor=${COMINbicor:-$(compath.py ${NET}/${aqm_ver})}
+export COMOUTbicor=${COMOUTbicor:-$(compath.py ${NET}/${aqm_ver})}


Same as above. Please move these to machine/wcoss2 and job_preamble.sh.

Answered same as above.

chan-hoo · 2023-05-26T15:20:06Z

ush/config_defaults.yaml

@@ -963,7 +963,7 @@ nco:
  SENDDBN_NTC: false
  SENDCOM: false
  SENDWEB: false
-  KEEPDATA: true
+  KEEPDATA: YES


Not correct.

Same explanation as above.

chan-hoo · 2023-05-26T15:20:24Z

ush/config_defaults.yaml

@@ -3052,4 +3052,6 @@ cpl_aqm_parm:
  DCOMINpt_src: "/path/to/point/source/base/directory"

  DCOMINairnow: "/path/to/airnow/obaservation/data"
-  AQM_AIRNOW_HIST_DIR: "/path/to/historical/airnow/data/dir"
+    #AQM_AIRNOW_HIST_DIR: "/path/to/historical/airnow/data/dir"


Remove this line if not necessary

Will have a global developer code clean up later.

chan-hoo · 2023-05-26T15:21:06Z

ush/job_preamble.sh

    export COMIN="${COMIN:-$(compath.py -o ${NET}/${model_ver}/${RUN}.${PDY}/${cyc})}"
    export COMOUT="${COMOUT:-$(compath.py -o ${NET}/${model_ver}/${RUN}.${PDY}/${cyc})}"
    export COMINm1="${COMINm1:-$(compath.py -o ${NET}/${model_ver}/${RUN}.${PDYm1})}"

    export COMINgfs="${COMINgfs:-$(compath.py ${envir}/gfs/${gfs_ver})}"
    export COMINgefs="${COMINgefs:-$(compath.py ${envir}/gefs/${gefs_ver})}"

-    export KEEPDATA="${KEEPDATA:-$KEEPDATA}"
+#    export KEEPDATA="${KEEPDATA:-$KEEPDATA}"


Not necessary

It is wrong to code the KEEPDATA assignment with two identical value.

I don't see any purpose for this line - if KEEPDATA isn't defined, it won't help anything.

chan-hoo · 2023-05-26T15:21:51Z

ush/job_preamble.sh

    # Remove temp directory
-    if [ "${RUN_ENVIR}" = "nco" ] && [ "${KEEPDATA}" = "FALSE" ]; then
+    if [ "${RUN_ENVIR}" = "nco" ] && [ "${KEEPDATA}" = "NO" ]; then
+        echo "DATA clean up for directory: ${DATA}"


@lgannoaa, this change makes KEEPDATA:false not work correctly.

The job_postamble function solution can result in NCO unable to recover failed cycle during some production emergency condition.
It is replaced by dedicated data_cleanup job designed. Please read the justification above.

chan-hoo · 2023-05-26T15:22:27Z

ush/machine/wcoss2.yaml

  DCOMINpt_src: /lfs/h2/emc/physics/noscrub/Youhua.Tang/nei2016v1-pt/v2023-01-PT
  DCOMINairnow: /lfs/h1/ops/prod/dcom
-  AQM_AIRNOW_HIST_DIR: /lfs/h2/emc/physics/noscrub/jianping.huang/Bias_correction/aqmv7.0
+    #AQM_AIRNOW_HIST_DIR: /lfs/h2/emc/physics/noscrub/jianping.huang/Bias_correction/aqmv7.0e


Remove commented-out lines

Will do the global development code clean up later.

lgannoaa · 2023-05-26T17:08:45Z

@chan-hoo I believe all your questions have been answered.
As we are aware of the AQM package is current freeze awaiting for this PR merged. As agreed from management on May 16th. All future update to the production/AQM.v7 for this implementation should be reviewed and approved from @JianpingHuang-NOAA. Must pass test with both EMC rocoto parallel and EMC AQM development ecflow.

chan-hoo · 2023-05-30T10:40:38Z

@lgannoaa, let's focus on adding ecFlow to production/AQM.v7 in this PR. To review this PR correctly, I should know how to run ecFlow on WCOSS2. Please let me know how to do it.

MatthewPyle-NOAA

Still need to understand a few things better before approving.

MatthewPyle-NOAA · 2023-06-02T16:44:32Z

jobs/JREGIONAL_BIAS_CORRECTION_O3

@@ -19,6 +19,10 @@
 source_config_for_task "cpl_aqm_parm|task_run_post|task_bias_correction_o3" ${GLOBAL_VAR_DEFNS_FP}
 . $USHdir/job_preamble.sh
 #
+export DCOMROOT=${DCOMROOT:-/lfs/h1/ops/prod/dcom}
+export DCOMINairnow=${DCOMINairnow:-/lfs/h1/ops/prod/dcom}
+export COMINbicor=${COMINbicor:-$(compath.py ${NET}/${aqm_ver})}


I think I see @chan-hoo 's point here - is there a reason why these variables can't be collected within the job_preamble.sh script sourced in the J-jobs?

MatthewPyle-NOAA · 2023-06-02T16:48:35Z

jobs/JREGIONAL_GET_EXTRN_MDL_FILES

I had thought this J-job was getting eliminated - is it needed? Are files needed for the AQM not available on WCOSS?

MatthewPyle-NOAA · 2023-06-02T17:08:24Z

parm/config/var_defns.sh

+GRID_GEN_METHOD='ESGgrid'
+PREDEF_GRID_NAME='AQM_NA_13km'
+DATE_FIRST_CYCL='202304250000'
+DATE_LAST_CYCL='209904251800'


@chan-hoo Is the DATE_FIRST_CYCL the item that makes it case specific? Is this primarily used in configuring the workflow, or is it executed in every AQM run?

MatthewPyle-NOAA · 2023-06-02T17:16:50Z

scripts/exaqm_manager.sh

Is this new aqm_manager purely for dealing with ecFlow? Is it only needed for non-production use of ecFlow? Just trying to understand the need for it - don't see similar items in production systems.

MatthewPyle-NOAA · 2023-06-02T17:22:27Z

scripts/exregional_aqm_lbcs.sh

-  FCST_LEN_HRS=${FCST_LEN_CYCL[$CYCLE_IDX]}
-fi
+case $cyc in
+  00 )


@lgannoaa Is your objection just the more complicated shell syntax? If the original version is working, and if there is a desire to keep this relatively close to the development SRW app, I'm not sure I see the need for this change

MatthewPyle-NOAA · 2023-06-02T17:28:32Z

scripts/exregional_run_post.sh

-fi
+# if [ $RUN_ENVIR = "nco" ] && [ $KEEPDATA = "FALSE" ]; then
+#   rm -rf $DATAFCST
+# fi


I would agree that having the post script be able to remove the fcst run directory isn't a good practice. I think the introduction of an external cleanup job/script like is included in this PR makes more sense.

MatthewPyle-NOAA · 2023-06-02T17:31:56Z

ush/config.aqm.nco.realtime.yaml

@@ -34,7 +34,7 @@ nco:
  model_ver: v7.0
  RUN: aqm_nco_aqmna13km
  OPSROOT: /path/to/custom/opsroot
-  KEEPDATA: false
+  KEEPDATA: NO


Agree with @lgannoaa here - KEEPDATA is a boolean variable (YES/NO) as described on page 5 of the implementation standards doc.

MatthewPyle-NOAA · 2023-06-02T17:34:09Z

ush/job_preamble.sh

    export COMIN="${COMIN:-$(compath.py -o ${NET}/${model_ver}/${RUN}.${PDY}/${cyc})}"
    export COMOUT="${COMOUT:-$(compath.py -o ${NET}/${model_ver}/${RUN}.${PDY}/${cyc})}"
    export COMINm1="${COMINm1:-$(compath.py -o ${NET}/${model_ver}/${RUN}.${PDYm1})}"

    export COMINgfs="${COMINgfs:-$(compath.py ${envir}/gfs/${gfs_ver})}"
    export COMINgefs="${COMINgefs:-$(compath.py ${envir}/gefs/${gefs_ver})}"

-    export KEEPDATA="${KEEPDATA:-$KEEPDATA}"
+#    export KEEPDATA="${KEEPDATA:-$KEEPDATA}"


I don't see any purpose for this line - if KEEPDATA isn't defined, it won't help anything.

…s-community#812)

…community#778) * Construct var_defns components from dictionary. * Bring back config_defaults.yaml * Add support for sourcing yaml file into shell script. * Remove newline for printing config, json config fix. * Make QUILTING a sub-dictionary in predef_grids * Reorganize config_defaults.yaml by task and feature. * Bug fix with QUILTING=true. * Structure a dictionary based on a template dictionary. * Convert all WE2E config files to yaml. * Take care of problematic chars when converting to shell string. * Process only selected keys of config. * Add symlinked yaml config files. * Actually use yaml config files for WE2E tests. * Delete all shell WE2E configs. * Don't check for single quotes in test description. * Make WE2E work with yaml configs. * Make yaml default config format. * Bug fix in run_WE2E script. * Add utility to check validity of yaml config file. * Add config utility interface in ush directory. * Remove unused check_expt_config_vars script. * Add description to default config. * Reorganize source_config. * Add XML as one of the config formats. * Update custom_ESGgrid config. * Bug fix due to update. * Change ensemble seed. * Change POST_OUTPUT group due to merge. * Make xml and ini configs work. * Maintain config structure down to var_defns. * Add function to load structured shell config, put description under metadata * Flatten dicts before importing env now that shell config is structured. * Support python regex for selecting dict keys. * Add capability of sourcing task specific portion of config file. * Access var_defns via env variable. * Make names of tasks consistent with ex- and j- job script names. * Append pid to temp file. * Prettify user config, don't use " in xml texts. * Compare timestamp of csv vs all files instead of directory. * Fixes for some pylint suggestions. * Convert new configs to yaml. * Format python files with black (no functional change). * More readable yaml/json formats by using more data types. Only datetime type is now in quotes. * More readable yaml config files for WE2E and default configs. * Make config_defaults itself more readable. * Correct pyyaml list indentation issue. * Fix indentation in all config files. * Use unquoted WTIME in config_defaults * Cosmotic changes. * Fix due to merge. * Make __init__.py clearer. * Fixes due to merge. * Minor edits of comments. * Remove wcoss_dell_p3 from workflow (ufs-community#810) * remove wcoss_dell_p3 * remove block for tide and gyre * Replace deprecated NCAR python environment with conda on Cheyenne (ufs-community#812) * Fix issue on get_extrn_lbcs when FCST_LEN_HRS>=40 with netcdf (ufs-community#814) * activate b file on hpss for >40h * add a new we2e test for fcst_len_hrs>40 * reduce fcst time for we2e * Convert new test case to yaml. * Fix formatting due to merge. * Convert new test case to yaml. * Fix unittest. * Merge develop * Remove exception logic from __init__.py * Minor change to cmd concat. * Make grid gen methods return dictionary, simplifis code a lot. * Add a comment why we are suppressing yaml import exception. * Minor change to beautify unittest output. * Add status badge for functional tests. * Reorder tasks in config_default and we2e test cases to match order in FV3LAM.xml * Keep single quotes and newlines in we2e test description. * Revert back to not rounding to 10 digits Co-authored-by: Chan-Hoo.Jeon-NOAA <60152248+chan-hoo@users.noreply.github.com> Co-authored-by: Michael Kavulich <kavulich@ucar.edu>

lgannoaa marked this pull request as ready for review May 26, 2023 12:31

lgannoaa requested review from chan-hoo, RatkoVasic-NOAA and BenjaminBlake-NOAA as code owners May 26, 2023 12:31

chan-hoo changed the title ~~Code merge for ecflow updates into the production/AQM.v7 #cf184119~~ [production/AQM.v7] Code merge for ecflow updates into the production/AQM.v7 #cf184119 May 26, 2023

chan-hoo linked an issue May 26, 2023 that may be closed by this pull request

[production/AQM.v7] A request of merging the ecflow-tested package into the production/AQM.v7 branch #797

Closed

chan-hoo requested a review from JianpingHuang-NOAA May 26, 2023 14:31

JianpingHuang-NOAA approved these changes May 26, 2023

View reviewed changes

JianpingHuang-NOAA mentioned this pull request May 26, 2023

[production/AQM.v7] A request of merging the ecflow-tested package into the production/AQM.v7 branch #797

Closed

chan-hoo requested changes May 26, 2023

View reviewed changes

chan-hoo requested a review from MatthewPyle-NOAA May 30, 2023 10:42

MatthewPyle-NOAA reviewed Jun 2, 2023

View reviewed changes

chan-hoo mentioned this pull request Jun 6, 2023

[production/AQM.v7] Add ecFlow option to production branch for AQM #821

Merged

15 tasks

michelleharrold pushed a commit to michelleharrold/ufs-srweather-app that referenced this pull request Jun 7, 2023

Replace deprecated NCAR python environment with conda on Cheyenne (uf…

917cb7b

…s-community#812)

lgannoaa closed this Jun 8, 2023

lgannoaa mentioned this pull request Oct 16, 2023

[production/AQM.v7] Finalizing the ecflow to support the AQMv7 implementation #942

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[production/AQM.v7] Code merge for ecflow updates into the production/AQM.v7 #cf184119 #812

[production/AQM.v7] Code merge for ecflow updates into the production/AQM.v7 #cf184119 #812

lgannoaa commented May 25, 2023 •

edited

Loading

JianpingHuang-NOAA commented May 26, 2023

JianpingHuang-NOAA commented May 26, 2023

JianpingHuang-NOAA commented May 26, 2023

chan-hoo May 26, 2023

lgannoaa May 26, 2023

chan-hoo May 30, 2023

MatthewPyle-NOAA Jun 2, 2023

chan-hoo May 26, 2023

lgannoaa May 26, 2023

chan-hoo May 30, 2023

chan-hoo May 26, 2023

lgannoaa May 26, 2023

chan-hoo May 26, 2023

lgannoaa May 26, 2023

chan-hoo May 26, 2023

lgannoaa May 26, 2023

chan-hoo May 26, 2023

lgannoaa May 26, 2023

MatthewPyle-NOAA Jun 2, 2023

chan-hoo May 26, 2023

lgannoaa May 26, 2023

chan-hoo May 26, 2023

lgannoaa May 26, 2023

lgannoaa commented May 26, 2023 •

edited

Loading

chan-hoo commented May 30, 2023

MatthewPyle-NOAA left a comment

MatthewPyle-NOAA Jun 2, 2023

MatthewPyle-NOAA Jun 2, 2023

MatthewPyle-NOAA Jun 2, 2023

MatthewPyle-NOAA Jun 2, 2023

MatthewPyle-NOAA Jun 2, 2023

MatthewPyle-NOAA Jun 2, 2023

MatthewPyle-NOAA Jun 2, 2023

MatthewPyle-NOAA Jun 2, 2023

[production/AQM.v7] Code merge for ecflow updates into the production/AQM.v7 #cf184119 #812

[production/AQM.v7] Code merge for ecflow updates into the production/AQM.v7 #cf184119 #812

Conversation

lgannoaa commented May 25, 2023 • edited Loading

TESTS CONDUCTED:

CHECKLIST

JianpingHuang-NOAA commented May 26, 2023

JianpingHuang-NOAA commented May 26, 2023

JianpingHuang-NOAA commented May 26, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lgannoaa commented May 26, 2023 • edited Loading

chan-hoo commented May 30, 2023

MatthewPyle-NOAA left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lgannoaa commented May 25, 2023 •

edited

Loading

lgannoaa commented May 26, 2023 •

edited

Loading