Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

GEFS Staging in exglobal_stage_ic #1892

Merged

Conversation

AnilKumar-NOAA
Copy link
Contributor

@AnilKumar-NOAA AnilKumar-NOAA commented Sep 27, 2023

Description

Adjusts source paths for forecast-only ICs in support of extension to ensembles. New paths omit the resolution (that is captured by the IC id name) and allow for a member level.

Directories with the reorganized ICs have been added to the three tier-1 machines under glopara space.

Building on this, also adds capability to stage GEFS ICs.

Resolves #911

Type of change

  • New feature

Change characteristics

  • Is this a breaking change (a change in existing functionality)? YES
  • Does this change require a documentation update? NO

How has this been tested?

Checklist

  • Any dependent changes have been merged and published
  • My code follows the style guidelines of this project
  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas
  • My changes generate no new warnings
  • New and existing tests pass with my changes
  • I have made corresponding changes to the documentation if necessary

Copy link

@github-advanced-security github-advanced-security bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

shellcheck found more than 10 potential problems in the proposed changes. Check the Files changed tab for more details.

scripts/exglobal_stage_ic.sh Fixed Show fixed Hide fixed
scripts/exglobal_stage_ic.sh Fixed Show fixed Hide fixed
scripts/exglobal_stage_ic.sh Fixed Show fixed Hide fixed
scripts/exglobal_stage_ic.sh Fixed Show fixed Hide fixed
scripts/exglobal_stage_ic.sh Fixed Show fixed Hide fixed
scripts/exglobal_stage_ic.sh Fixed Show fixed Hide fixed
scripts/exglobal_stage_ic.sh Fixed Show fixed Hide fixed
@WalterKolczynski-NOAA
Copy link
Contributor

Please fill in the PR template.

Copy link
Contributor

@WalterKolczynski-NOAA WalterKolczynski-NOAA left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Didn't read a lot of it carefully because it is going to need a lot of rewriting.

BASE_CPLIC will also need to be updated on all the machines once I duplicate what I've done on Hera elsewhere.

scripts/exglobal_stage_ic.sh Outdated Show resolved Hide resolved
scripts/exglobal_stage_ic.sh Outdated Show resolved Hide resolved
scripts/exglobal_stage_ic.sh Outdated Show resolved Hide resolved
scripts/exglobal_stage_ic.sh Outdated Show resolved Hide resolved
scripts/exglobal_stage_ic.sh Outdated Show resolved Hide resolved
scripts/exglobal_stage_ic.sh Fixed Show fixed Hide fixed
scripts/exglobal_stage_ic.sh Fixed Show resolved Hide resolved
@emcbot
Copy link

emcbot commented Oct 27, 2023

Automated global-workflow Testing Results:

Machine: Hera
Start: Thu Oct 26 20:05:00 UTC 2023 on hfe05
---------------------------------------------------
Checkout:                      *SUCCESS*
Checkout: Completed at Thu Oct 26 20:08:46 UTC 2023
Build:                         *SUCCESS*
Build: Completed at Thu Oct 26 21:28:12 UTC 2023
Created experiment:            *SUCCESS*
Case setup: Completed at Thu Oct 26 21:28:24 UTC 2023 for experiment C48_ATM_a2716750
Created experiment:            *SUCCESS*
Case setup: Completed at Thu Oct 26 21:28:28 UTC 2023 for experiment C48_S2SA_gefs_a2716750
Created experiment:            *SUCCESS*
Case setup: Completed at Thu Oct 26 21:28:32 UTC 2023 for experiment C48_S2SW_a2716750
Created experiment:            *SUCCESS*
Case setup: Completed at Thu Oct 26 21:28:38 UTC 2023 for experiment C96C48_hybatmDA_a2716750
Created experiment:            *SUCCESS*
Case setup: Completed at Thu Oct 26 21:28:42 UTC 2023 for experiment C96_atm3DVar_a2716750
Experiment C48_ATM_a2716750 completed: *SUCCESS*
Experiment C48_ATM_a2716750 Completed at Thu Oct 26 22:48:16 UTC 2023
with 29 successfully completed jobs
Experiment C48_S2SA_gefs_a2716750 completed: *SUCCESS*
Experiment C48_S2SA_gefs_a2716750 Completed at Thu Oct 26 22:51:12 UTC 2023
with 4 successfully completed jobs
Experiment C96C48_hybatmDA_a2716750 completed: *SUCCESS*
Experiment C96C48_hybatmDA_a2716750 Completed at Fri Oct 27 01:21:16 UTC 2023
with 136 successfully completed jobs
Experiment C48_S2SW_a2716750 completed: *SUCCESS*
Experiment C48_S2SW_a2716750 Completed at Fri Oct 27 01:57:15 UTC 2023
with 37 successfully completed jobs
Experiment C96_atm3DVar_a2716750 completed: *SUCCESS*
Experiment C96_atm3DVar_a2716750 Completed at Fri Oct 27 03:48:13 UTC 2023
with 72 successfully completed jobs

@emcbot emcbot added CI-Hera-Passed **Bot use only** CI testing on Hera for this PR has completed successfully and removed CI-Hera-Running **Bot use only** CI testing on Hera for this PR is in-progress labels Oct 27, 2023
@emcbot
Copy link

emcbot commented Oct 27, 2023

Automated global-workflow Testing Results:

Machine: Hera
Start: Thu Oct 26 20:05:00 UTC 2023 on hfe05
---------------------------------------------------
Checkout:                      *SUCCESS*
Checkout: Completed at Thu Oct 26 20:08:46 UTC 2023
Build:                         *SUCCESS*
Build: Completed at Thu Oct 26 21:28:12 UTC 2023
Created experiment:            *SUCCESS*
Case setup: Completed at Thu Oct 26 21:28:24 UTC 2023 for experiment C48_ATM_a2716750
Created experiment:            *SUCCESS*
Case setup: Completed at Thu Oct 26 21:28:28 UTC 2023 for experiment C48_S2SA_gefs_a2716750
Created experiment:            *SUCCESS*
Case setup: Completed at Thu Oct 26 21:28:32 UTC 2023 for experiment C48_S2SW_a2716750
Created experiment:            *SUCCESS*
Case setup: Completed at Thu Oct 26 21:28:38 UTC 2023 for experiment C96C48_hybatmDA_a2716750
Created experiment:            *SUCCESS*
Case setup: Completed at Thu Oct 26 21:28:42 UTC 2023 for experiment C96_atm3DVar_a2716750
Experiment C48_ATM_a2716750 completed: *SUCCESS*
Experiment C48_ATM_a2716750 Completed at Thu Oct 26 22:48:16 UTC 2023
with 29 successfully completed jobs
Experiment C48_S2SA_gefs_a2716750 completed: *SUCCESS*
Experiment C48_S2SA_gefs_a2716750 Completed at Thu Oct 26 22:51:12 UTC 2023
with 4 successfully completed jobs
Experiment C96C48_hybatmDA_a2716750 completed: *SUCCESS*
Experiment C96C48_hybatmDA_a2716750 Completed at Fri Oct 27 01:21:16 UTC 2023
with 136 successfully completed jobs
Experiment C48_S2SW_a2716750 completed: *SUCCESS*
Experiment C48_S2SW_a2716750 Completed at Fri Oct 27 01:57:15 UTC 2023
with 37 successfully completed jobs
Experiment C96_atm3DVar_a2716750 completed: *SUCCESS*
Experiment C96_atm3DVar_a2716750 Completed at Fri Oct 27 03:48:13 UTC 2023
with 72 successfully completed jobs

@emcbot
Copy link

emcbot commented Oct 27, 2023

Automated global-workflow Testing Results:

Machine: Orion
Start: Thu Oct 26 15:04:20 CDT 2023 on Orion-login-1.HPC.MsState.Edu
---------------------------------------------------
Checkout:                      *SUCCESS*
Checkout: Completed at Thu Oct 26 15:05:45 CDT 2023
Build:                         *SUCCESS*
Build: Completed at Thu Oct 26 15:34:27 CDT 2023
Created experiment:            *SUCCESS*
Case setup: Completed at Thu Oct 26 15:34:32 CDT 2023 for experiment C48_ATM_a2716750
Created experiment:            *SUCCESS*
Case setup: Completed at Thu Oct 26 15:34:36 CDT 2023 for experiment C48_S2SA_gefs_a2716750
Created experiment:            *SUCCESS*
Case setup: Completed at Thu Oct 26 15:34:39 CDT 2023 for experiment C48_S2SW_a2716750
Created experiment:            *SUCCESS*
Case setup: Completed at Thu Oct 26 15:34:42 CDT 2023 for experiment C96_atm3DVar_a2716750
Created experiment:            *SUCCESS*
Case setup: Completed at Thu Oct 26 15:34:46 CDT 2023 for experiment C96C48_hybatmDA_a2716750
Experiment C48_S2SA_gefs_a2716750 completed: *SUCCESS*
Experiment C48_S2SA_gefs_a2716750 Completed at Fri Oct 27 04:00:00 CDT 2023
with 4 successfully completed jobs

Copy link
Contributor

@aerorahul aerorahul left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

some comments

scripts/exglobal_stage_ic.sh Outdated Show resolved Hide resolved
workflow/rocoto/gefs_tasks.py Show resolved Hide resolved
rc=$?
(( rc != 0 )) && error_message "${source}" "${target}" "${rc}"
[[ ${rc} -ne 0 ]] && error_message "${src}" "${tgt}" "${rc}"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why [[ ${rc} -ne 0 ]] here?
Elsewhere we have (( rc != 0 ))

@@ -0,0 +1,14 @@
#! /usr/bin/env bash
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is this being added? Are we running with waves in GEFS at this point?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we just copy that files from gfs when we were trying to run with wave.

@WalterKolczynski-NOAA WalterKolczynski-NOAA merged commit aff6ca7 into NOAA-EMC:develop Oct 27, 2023
@TerrenceMcGuinness-NOAA
Copy link
Collaborator

TerrenceMcGuinness-NOAA commented Oct 27, 2023

(gfs_workflow) mterry (Orion-login-3) C48_ATM_a2716750 $ pwd
/work2/noaa/stmp/GFS_CI_ROOT/PR/1892/RUNTESTS/EXPDIR/C48_ATM_a2716750
(gfs_workflow) mterry (Orion-login-3) C48_ATM_a2716750 $ rocotocheck -d C48_ATM_a2716750.db -w C48_ATM_a2716750.xml -c 202103231200 -t gfsstage_ic

Task: gfsstage_ic
  account: nems
  command: /work2/noaa/stmp/GFS_CI_ROOT/PR/1892/global-workflow/jobs/rocoto/stage_ic.sh
  cores: 1
  cycledefs: gfs
  final: false
  jobname: C48_ATM_a2716750_gfsstage_ic_12
  join: /work2/noaa/stmp/GFS_CI_ROOT/PR/1892/RUNTESTS/COMROT/C48_ATM_a2716750/logs/2021032312/gfsstage_ic.log
  maxtries: 2
  name: gfsstage_ic
  nodes: 1:ppn=1:tpp=1
  partition: orion
  queue: batch
  throttle: 9999999
  walltime: 00:15:00
  environment
    CDATE ==> 2021032312
    CDUMP ==> gfs
    COMROOT ==> /work/noaa/global/glopara/com
    DATAROOT ==> /work/noaa/stmp/mterry/RUNDIRS/C48_ATM_a2716750
    EXPDIR ==> /work2/noaa/stmp/GFS_CI_ROOT/PR/1892/RUNTESTS/EXPDIR/C48_ATM_a2716750
    HOMEgfs ==> /work2/noaa/stmp/GFS_CI_ROOT/PR/1892/global-workflow
    NET ==> gfs
    PDY ==> 20210323
    RUN ==> gfs
    RUN_ENVIR ==> emc
    cyc ==> 12
  dependencies
    AND is not satisfied
      /work/noaa/global/glopara/data/ICSDIR/prototype_ICs/workflow_C48_refactored/2021032312/atmos/gfs_ctrl.nc does not exist

Cycle: 202103231200
  Valid for this task: YES
  State: active
  Activated: 2023-10-27 18:12:52 UTC
  Completed: -
  Expired: -

Job: This task has not been submitted for this cycle

Task can not be submitted because:
  Dependencies are not satisfied
(gfs_workflow) mterry (Orion-login-3) C48_ATM_a2716750 $ ls -l /work/noaa/global/glopara/data/ICSDIR/prototype_ICs/workflow_C48_refactored/2021032312/atmos/gfs_ctrl.nc
lrwxrwxrwx 1 wkolczyn global 49 Oct 19 14:33 /work/noaa/global/glopara/data/ICSDIR/prototype_ICs/workflow_C48_refactored/2021032312/atmos/gfs_ctrl.nc -> workflowtest/2021032312/gfs/C48/INPUT/gfs_ctrl.nc
(gfs_workflow) mterry (Orion-login-3) C48_ATM_a2716750 $ 

Not sure whey Rocoto is not finding this link to this gfs_crl.nc file.

@emcbot
Copy link

emcbot commented Oct 27, 2023

Automated global-workflow Testing Results:

Machine: Orion
Start: Thu Oct 26 15:04:20 CDT 2023 on Orion-login-1.HPC.MsState.Edu
---------------------------------------------------
Checkout:                      *SUCCESS*
Checkout: Completed at Thu Oct 26 15:05:45 CDT 2023
Build:                         *SUCCESS*
Build: Completed at Thu Oct 26 15:34:27 CDT 2023
Created experiment:            *SUCCESS*
Case setup: Completed at Thu Oct 26 15:34:32 CDT 2023 for experiment C48_ATM_a2716750
Created experiment:            *SUCCESS*
Case setup: Completed at Thu Oct 26 15:34:36 CDT 2023 for experiment C48_S2SA_gefs_a2716750
Created experiment:            *SUCCESS*
Case setup: Completed at Thu Oct 26 15:34:39 CDT 2023 for experiment C48_S2SW_a2716750
Created experiment:            *SUCCESS*
Case setup: Completed at Thu Oct 26 15:34:42 CDT 2023 for experiment C96_atm3DVar_a2716750
Created experiment:            *SUCCESS*
Case setup: Completed at Thu Oct 26 15:34:46 CDT 2023 for experiment C96C48_hybatmDA_a2716750
Experiment C48_S2SA_gefs_a2716750 completed: *SUCCESS*
Experiment C48_S2SA_gefs_a2716750 Completed at Fri Oct 27 04:00:00 CDT 2023
with 4 successfully completed jobs
Experiment C48_ATM_a2716750 completed: *SUCCESS*
Experiment C48_ATM_a2716750 Completed at Fri Oct 27 15:10:12 CDT 2023
with 29 successfully completed jobs

@emcbot emcbot added CI-Orion-Failed **Bot use only** CI testing on Orion for this PR has failed and removed CI-Orion-Running **Bot use only** CI testing on Orion for this PR is in-progress labels Oct 27, 2023
@emcbot
Copy link

emcbot commented Oct 27, 2023

Automated global-workflow Testing Results:

Machine: Orion
Start: Thu Oct 26 15:04:20 CDT 2023 on Orion-login-1.HPC.MsState.Edu
---------------------------------------------------
Checkout:                      *SUCCESS*
Checkout: Completed at Thu Oct 26 15:05:45 CDT 2023
Build:                         *SUCCESS*
Build: Completed at Thu Oct 26 15:34:27 CDT 2023
Created experiment:            *SUCCESS*
Case setup: Completed at Thu Oct 26 15:34:32 CDT 2023 for experiment C48_ATM_a2716750
Created experiment:            *SUCCESS*
Case setup: Completed at Thu Oct 26 15:34:36 CDT 2023 for experiment C48_S2SA_gefs_a2716750
Created experiment:            *SUCCESS*
Case setup: Completed at Thu Oct 26 15:34:39 CDT 2023 for experiment C48_S2SW_a2716750
Created experiment:            *SUCCESS*
Case setup: Completed at Thu Oct 26 15:34:42 CDT 2023 for experiment C96_atm3DVar_a2716750
Created experiment:            *SUCCESS*
Case setup: Completed at Thu Oct 26 15:34:46 CDT 2023 for experiment C96C48_hybatmDA_a2716750
Experiment C48_S2SA_gefs_a2716750 completed: *SUCCESS*
Experiment C48_S2SA_gefs_a2716750 Completed at Fri Oct 27 04:00:00 CDT 2023
with 4 successfully completed jobs
Experiment C48_ATM_a2716750 completed: *SUCCESS*
Experiment C48_ATM_a2716750 Completed at Fri Oct 27 15:10:12 CDT 2023
with 29 successfully completed jobs
Experiment C96_atm3DVar_a2716750 Terminated: *FAILED*
Experiment C96_atm3DVar_a2716750 Terminated with 1 tasks failed at Fri Oct 27 16:50:15 CDT 2023
Error logs:
/work2/noaa/stmp/GFS_CI_ROOT/PR/1892/RUNTESTS/COMROT/C96_atm3DVar_a2716750/logs/2021122100/gfsanal.log

@JessicaMeixner-NOAA
Copy link
Contributor

@aerorahul @WalterKolczynski-NOAA - @jiandewang just pointed out that the IC folders have been moved from the ufscpara and his space on wcoss2. We specifically moved it out of the glopara space on wcoss2 because of our data needs exceeding the space requirements on WCOSS2. Is this intentional? Did you just need us to refactor the space? We'd be happy to work with you on this. We will soon be adding ICs for HR3 and will need to know what the process for this is now.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
CI-Hera-Passed **Bot use only** CI testing on Hera for this PR has completed successfully CI-Orion-Failed **Bot use only** CI testing on Orion for this PR has failed
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Extend coupled IC job to handle ensemble staging
6 participants