Cloud Dependencies for Regression Testing

[zach1221]

Description

Add PR to support running the UFS WM Regression Test script on the three NOAA Cloud HPCs (GCP, Azure and AWS).

To run the UFS WM Regression Test suite on the NOAA Cloud platforms, RT related scripts, module and fv3 slurm job files need to be updated with Cloud platform related dependencies and variables.

Following these updates, rt.sh should run any test from the rt.conf suite.

Top of commit queue on: TBD

Input data additions/changes

• No changes are expected to input data.
• There will be new input data.
• Input data will be updated.

Anticipated changes to regression tests:

• No changes are expected to any regression test.
• Changes are expected to the following tests:

Subcomponents involved:

• AQM
• CDEPS
• CICE
• CMEPS
• CMakeModules
• FV3
• GOCART
• HYCOM
• MOM6
• NOAHMP
• WW3
• stochastic_physics
• none

Combined with PR's (If Applicable):

Commit Queue Checklist:

• Link PR's from all sub-components involved
• Confirm reviews completed in sub-component PR's
• Add all appropriate labels to this PR.
• Run full RT suite on either Hera/Cheyenne with both Intel/GNU compilers
• Add list of any failed regression tests to "Anticipated changes to regression tests" section.

Linked PR's and Issues:

Testing Day Checklist:

• This PR is up-to-date with the top of all sub-component repositories except for those sub-components which are the subject of this PR.
• Move new/updated input data on RDHPCS Hera and propagate input data changes to all supported systems.

Testing Log (for CM's):

• RDHPCS
  • Intel
    • Hera
    • Orion
    • Jet
    • Gaea
    • Cheyenne
  • GNU
    • Hera
    • Cheyenne
• WCOSS2
  • Dogwood/Cactus
  • Acorn
• CI
  • Completed
• opnReqTest
  • N/A
  • Log attached to comment

[jkbk2004]

@ulmononian Can you monitor with this PR in terms of spack stack maintenance strategy on PW clouds? We need to keep a consistency of machine names between weather model and srw as well.

[zach1221]

@SadeghTabas-NOAA Here's the draft cloud PR I was referring to previously. @ulmononian Sadegh was able to build new permanent Spack-Stack modules on Parallel Works AWS, for our use in this PR. Would you be able to assist us in doing the same for GCP and Azure?

[jkbk2004]

Automated RT Failure Notification
Machine: orion
Compiler: intel
Job: RT
[RT] Repo location: /work/noaa/epic-ps/jongkim/autort/pr/1284394645/20230503073018/ufs-weather-model
Please make changes and add the following label back: orion-intel-RT

[epic-cicd-jenkins]

Automated RT Failure Notification
Machine: jet
Compiler: intel
Job: RT
[RT] Repo location: /mnt/lfs4/HFIP/hfv3gfs/role.epic/autort/pr/1284394645/20230503122210/ufs-weather-model
[RT] Error: Test control_c384gdas 028 failed in run_test failed
Please make changes and add the following label back: jet-intel-RT

[BrianCurtis-NOAA]

Didn't wait for Acorn. they passed though. Can't push to this any more.

[jkbk2004]

@BrianCurtis-NOAA I could hold my approval until acorn log come. This pr is only for pw-cloud feature. Sanity check can be enough for validation with other machines. @FernandoAndrade-NOAA FYI

[DusanJovic-NOAA]

I see '-B' option was added back to nccmp. Was that intended?

[zach1221]

@DusanJovic-NOAA Yes, the Cheyenne admin installed nccmp/1.9.1.0 on the HPC, which has the -B option.

[DusanJovic-NOAA]

@DusanJovic-NOAA Yes, the Cheyenne admin installed nccmp/1.9.1.0 on the HPC, which has the -B option.

Excellent. Thank you.