Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

metatasks enkfgdas_fcst and enkfgdas_epmn no longer created for C96C48_ufs_hybatmDA #3192

Closed
RussTreadon-NOAA opened this issue Dec 26, 2024 · 5 comments · Fixed by #3193
Closed
Assignees
Labels
bug Something isn't working triage Issues that are triage

Comments

@RussTreadon-NOAA
Copy link
Contributor

What is wrong?

Attempts to run g-w CI for case C96C48_ufs_hybatmDA fail following the merger of g-w PR #3185 into develop.

rocotorun -d C96C48_ufs_hybatmDA.db -w C96C48_ufs_hybatmDA.xml fails with

12/26/24 15:25:54 UTC :: C96C48_ufs_hybatmDA_test.xml :: Error: Extra element dependency in interleave.
12/26/24 15:25:54 UTC :: C96C48_ufs_hybatmDA_test.xml :: Error: Element task failed to validate content at C96C48_ufs_hybatmDA_test.xml:93.
12/26/24 15:25:54 UTC :: C96C48_ufs_hybatmDA_test.xml :: Error: Extra element task in interleave.
12/26/24 15:25:54 UTC :: C96C48_ufs_hybatmDA_test.xml :: Error: Element workflow failed to validate content at C96C48_ufs_hybatmDA_test.xml:17.
12/26/24 15:25:54 UTC :: C96C48_ufs_hybatmDA_test.xml :: Error: Element workflow failed to validate content at C96C48_ufs_hybatmDA_test.xml:17.

Line 93 is the block defining task gdas_atmanlinit. This task depends on metatask enkfgdas_epmn.

Metatask enkfgdas_epmn is not defined in the xml. Upon closer inspection, metatask enkfgdas_fcst is also not defined in the xml ... even though it is listed as a dependency for enkfgdas_echgres

What should have happened?

We should be able to run g-w CI case C96C48_ufs_hybatmDA

What machines are impacted?

All or N/A

What global-workflow hash are you using?

g-w develop at 1c37f90

Steps to reproduce

  1. clone and install g-w develop
  2. set up g-w CI for C96C48_ufs_hybatmDA
  3. cd to C96C48_ufs_hybatmDA EXPDIR
  4. rocotorun -d C96C48_ufs_hybatmDA.db -w C96C48_ufs_hybatmDA.xml

The rocotorun command will fail. This is wrong. The rocotorun command should not fail.

Additional information

As a test install g-w develop at bdc0e29. This is the commit prior to PR #3185.

Set up g-w CI for C96C48_ufs_hybatmDA using the $HOMEgfs for the bdc0e29 install. The resulting C96C48_ufs_hybatmDA.xml defines metatasks enkfgdas_fcst and enkfgdas_epmn. rocotorun -d C96C48_ufs_hybatmDA.db -w C96C48_ufs_hybatmDA.xml runs without error. This is correct.

Do you have a proposed solution?

Please carefully check bdc0e29 for an error when UFS DA is active. This is a blocker issue. We can not cycle UFS ATM DA using g-w develop at 1c37f90.

A check of g-w PR #3185 does not show C96C48_ufs_hybatmDA as being run. This case is skipped on gaea, orion, hercules, and wcoss2. PR #3185 ran g-w CI on hercules. Thus, this error slipped through the cracks.

@RussTreadon-NOAA RussTreadon-NOAA added bug Something isn't working triage Issues that are triage labels Dec 26, 2024
@RussTreadon-NOAA
Copy link
Contributor Author

FYI: @DavidNew-NOAA

As of g-w develop at 1c37f90, g-w CI for C96C48_ufs_hybatmDA does not work. The xml file created by g-w does not define metatasks enkgdas_fcst and enkfgdas_epmn. Both tasks are listed at dependencies for other jobs in the xml.

@RussTreadon-NOAA
Copy link
Contributor Author

@TravisElless-NOAA - g-w PR #3185 generates an incomplete xml for C96C48_ufs_hybatmDA. I haven't looked into the details of #3185. Do you recall anything in #3185 which might be a problem when building xml for JEDI-based atmospheric DA?

@DavidNew-NOAA
Copy link
Contributor

@RussTreadon-NOAA thanks for the heads up. #3185 only had Hercules CI run to test it, and it appears from the emcbot output that C96C48_ufs_hybatmDA is not run on that machine, so that's how this bug fell through the cracks.

@RussTreadon-NOAA
Copy link
Contributor Author

Make the following change in a working copy of workflow/applications/gfs_cycled.py

@@ -311,6 +311,8 @@ class GFSCycledAppConfig(AppConfig):
                         task_names[run] += ['atmensanlobs', 'atmensanlsol']
                     else:
                         task_names[run] += ['atmensanlletkf']
+                    task_names[run].append('efcs') if 'gdas' in run else 0
+                    task_names[run].append('epos') if 'gdas' in run else 0
 
                 else:
                     task_names[run] += ['eobs', 'eupd']

With this change metatasks enkfgdas_fcst and enkfgdas_epmn are added to the xml file for C96C48_ufs_hybatmDA.

@RussTreadon-NOAA
Copy link
Contributor Author

Work for this bugfix will be done in RussTreadon-NOAA:bugfix/ufs_hybatmDA

RussTreadon-NOAA added a commit to RussTreadon-NOAA/global-workflow that referenced this issue Dec 27, 2024
@RussTreadon-NOAA RussTreadon-NOAA self-assigned this Dec 27, 2024
RussTreadon-NOAA added a commit to RussTreadon-NOAA/global-workflow that referenced this issue Dec 28, 2024
RussTreadon-NOAA added a commit to RussTreadon-NOAA/global-workflow that referenced this issue Jan 1, 2025
tsga added a commit to tsga/global-workflow that referenced this issue Jan 4, 2025
* develop:
  Ensure OCNRES and ICERES have 3 digits in the archive script (NOAA-EMC#3199)
  Set runtime shell requirements within Jenkins Pipeline (NOAA-EMC#3171)
  Add efcs and epos to ufs_hybatm xml (NOAA-EMC#3192) (NOAA-EMC#3193)
  Fix GEFS and SFS compile flags in build_all.sh (NOAA-EMC#3197)
  Remove early-cycle EnKF forecast (NOAA-EMC#3185)
  Fix mod_icec bug in atmos_prod (NOAA-EMC#3167)
  Create compute build option (NOAA-EMC#3186)
  Support global-workflow using Rocky 8 on CSPs (NOAA-EMC#2998)
danholdaway added a commit to danholdaway/global-workflow that referenced this issue Jan 27, 2025
* develop:
  Remove WAFS files and references from `develop` (NOAA-EMC#3263)
  fix intel stack version number on c5 (NOAA-EMC#3258)
  Update gsi_monitor and ufs_utils hashes to recent hashes for C5/C6 build and run (NOAA-EMC#3252)
  Enable DA cycling on gaea C5/C6 (NOAA-EMC#3255)
  Copy post-processed sea ice increment for diagnostics (NOAA-EMC#3235)
  Only run METplus in the 3Dvar tests (NOAA-EMC#3245)
  Clone, build, and run C48_ATM and C48_S2SW on Gaea C5 and C6 (NOAA-EMC#3106)
  Add echgres as a dependency only for RUN=enkfgdas, not enkfgfs (NOAA-EMC#3246)
  Add domain level to wave gridded COM path (NOAA-EMC#3137)
  CI JJOB Tests using CMake (NOAA-EMC#3214)
  Make assorted updates to waves (NOAA-EMC#3190)
  Move WCOSS2 LD_LIBRARY_PATH patches to load_ufsda_modules.sh (NOAA-EMC#3236)
  Adding a gefs_arch task to GEFS workflow (NOAA-EMC#3211)
  Add additional GEFS variables needed for AI/ML applications  (NOAA-EMC#3221)
  Add bmat task dependency to marine LETKF task (NOAA-EMC#3224)
  Resolve bug with LMOD_TMOD_FIND_FIRST setting affecting build on WCOSS2 (NOAA-EMC#3229)
  Reinstate product groups (NOAA-EMC#3208)
  Additional fixes for downstream jobs (NOAA-EMC#3187)
  Turn IAU off during staging job for cold start experiments (NOAA-EMC#3215)
  Update the gdas.cd hash and enable GDASApp to run on WCOSS2 (NOAA-EMC#3220)
  Update upload-artifact to v4 (NOAA-EMC#3216)
  Prevent duplicate case generation in generate_workflows.sh (NOAA-EMC#3217)
  Update g-w to cycle with C1152 ATM (NOAA-EMC#3206)
  Separate use of initial increment/perturbation file from REPLAY/+03 ICs  (NOAA-EMC#3119)
  Update gsi_enkf hash and gsi_ver (NOAA-EMC#3207)
  Remove cpus-per-task from APRUN_OCNANALECEN on WCOSS2 (NOAA-EMC#3212)
  Remove 5WAVH from AWIPS GRIB2 parm files (NOAA-EMC#3146)
  Remove multi-grid wave support (NOAA-EMC#3188)
  Add echgres as a dependency for earc (NOAA-EMC#3202)
  Ensure OCNRES and ICERES have 3 digits in the archive script (NOAA-EMC#3199)
  Set runtime shell requirements within Jenkins Pipeline (NOAA-EMC#3171)
  Add efcs and epos to ufs_hybatm xml (NOAA-EMC#3192) (NOAA-EMC#3193)
  Fix GEFS and SFS compile flags in build_all.sh (NOAA-EMC#3197)
  Remove early-cycle EnKF forecast (NOAA-EMC#3185)
  Fix mod_icec bug in atmos_prod (NOAA-EMC#3167)
  Create compute build option (NOAA-EMC#3186)
  Support global-workflow using Rocky 8 on CSPs (NOAA-EMC#2998)
  Change orog gravity wave drag scheme for grid sizes less than 10km (NOAA-EMC#3175)
  Switch snow DA to use 2DVar for deterministic and ensemble mean (NOAA-EMC#3163)
  Update compression options for GEFS history files (NOAA-EMC#3184)
  Update compression options for high res history files (NOAA-EMC#3178)
  Turn DO_TEST_MODE off (NOAA-EMC#3177)
  Hotfix for gdas_arch div/0 (NOAA-EMC#3169)
  Allow building of the ufs-weather-model, WW3 pre/post execs for GFS, GEFS, SFS in the same clone of global-workflow (NOAA-EMC#3098)
  Switch Aerosol DA to use JCB and Jedi class (NOAA-EMC#3125)
  Update ufs-weather-model to 2024-12-06 commit  (NOAA-EMC#3145)
  Enable traditional threading as an option (NOAA-EMC#3149)
  Update HPC_ACCOUNT on Hercules to fv3-cpu (NOAA-EMC#3164)
  Turn C96C48_ufs_hybatmDA and C48mx500_3DVarAOWCDA into a regression test (NOAA-EMC#3120)
  Update GSI analysis jobs to use COMIN/COMOUT (NOAA-EMC#3092)
  Update HPC Tier Definitions (NOAA-EMC#3138)
  Add marine hybrid envar (NOAA-EMC#3041)
  Archive the experiment directory along with git status/diff output (NOAA-EMC#3105)
  Use stochastic restart patterns on rerun (NOAA-EMC#3077)
  Point Jenkinsfile back to CI/ (NOAA-EMC#3139)
  Fix wave restart for cold start and add ic version file (NOAA-EMC#3112)
  Allow users to override the default account at setup time (NOAA-EMC#3127)
  Refactor gridded wave post (NOAA-EMC#3014)
  Update docs related to NOAA CSPs (NOAA-EMC#3043)
  Allow APP to differ between RUNs (NOAA-EMC#2943)
  Run one executable for soca2cice (instead of two) (NOAA-EMC#3118)
  Speed up GSI analysis jobs in CI testing (NOAA-EMC#3115)
  Make aerosol output frequency variable (NOAA-EMC#2982)
  Add new stations to GFS BUFR sounding products (NOAA-EMC#3107)
  JCB-based obs+bias staging, Jedi class updates, and marine B-matrix refactoring (NOAA-EMC#2992)
  Enable tapering of atm ens perts at the model top (NOAA-EMC#3097)
  Update JGDAS ENKF POST  job  (NOAA-EMC#3090)
  SFS Runs at C96mx100  (NOAA-EMC#2960)
  Move machine-based options from config.base to host files (NOAA-EMC#3053)
  Remove RUNDIRS before running CI cases to cover re-run events (NOAA-EMC#3076)
  CI GitHub pipeline (hotfix) update for fetching repo name (NOAA-EMC#3084)
  Update JGDAS ENKF ECEN job  (NOAA-EMC#3050)
  Update snow obs processing job (NOAA-EMC#3055)
  Update to action workflow pipeline in default repo for development  (NOAA-EMC#3062)
  Update to action workflow pipeline in default repo for development (NOAA-EMC#3061)
  Update workflow pipeline (NOAA-EMC#3060)
  PW CI pipeline update5 ready for review so it can be merged and tested (NOAA-EMC#3059)
  Revert "GitHub CI Pipeline update for debugging forked PR support" (NOAA-EMC#3057)
  GitHub CI Pipeline update for debugging forked PR support (NOAA-EMC#3056)
  Add more ocean variables for post-processing in GEFS (NOAA-EMC#2995)
  Auto provisioning of PW clusters from GitHub CI added (NOAA-EMC#3051)
  Fix the name of the TC tracker filenames in archive.py (NOAA-EMC#3030)
  Make wxflow links static instead of from link_workflow (NOAA-EMC#3008)
  Update global jdas enkf diag job with COMIN/COMOUT for COM prefix (NOAA-EMC#2959)
  Add run and finalize methods to marine LETKF task (NOAA-EMC#2944)
  Fix wave restarts and GEFS FHOUT/FHMAX (NOAA-EMC#3009)
  Disabling hyper-threading (NOAA-EMC#2965)
  GitHub Actions Pipeline Updates for Self-Hosted Runners on PW (NOAA-EMC#3018)
  CI jekninsfile update hotfix (NOAA-EMC#3038)
  Update gdas.cd (NOAA-EMC#2978)
  Add ability to add tag to pslots with generate_workflows (NOAA-EMC#3036)
  CI update to shell environment with HOMEgfs to HOME_GFS for systems that need the path (NOAA-EMC#3013)
  Quick updated to Jenkins (health check) launch script (NOAA-EMC#3033)
  Document the generate_workflows.sh script (NOAA-EMC#3028)
  Replace gfs_cyc with an interval (NOAA-EMC#2928)
  Hotfix: Fix generate_workflows.sh optional build flags (NOAA-EMC#3024)
  Add a tool to run multiple YAML cases locally (NOAA-EMC#3004)
  Hotfix: Correctly set overwrite option when specified (NOAA-EMC#3021)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working triage Issues that are triage
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants