Enable and retest RT cases on Derecho

[zach1221]

Description

Cases that were disabled for Derecho in rt.conf, to match Cheyenne setting, and should be retested/debugged to see if they can be enabled to run on the HPC.

Enable and retest regional_atmaq_debug_intel, cpld_control_p8_faster_intel, cpld_bmark_p8, cpld_restart_bmark_p8 and conus13km_debug_qr on Derecho.

To Reproduce:

1. clone ufs-weather model develop repo
2. enable these tests in rt.conf
3. re-run

Additional context

Output

[DeniseWorthen]

Cheyenne-disabled tests also include cpld_bmark_p8, cpld_restart_bmark_p8 and conus13km_debug_qr

[zach1221]

Cheyenne-disabled tests also include cpld_bmark_p8, cpld_restart_bmark_p8 and conus13km_debug_qr

Yes, true. Ok, noted in description as well.

[natalie-perlin]

Here are the modules that are needed to be loaded on Derecho to enable use of ecflow and rocoto:

module use /glade/work/epicufsrt/contrib/spack-stack/derecho/modulefiles
module load ecflow/5.8.4

module use /glade/work/epicufsrt/contrib/derecho/rocoto/modulefiles
module load rocoto

[zach1221]

@natalie-perlin ok, should variable ECFLOW_START=/glade/p/ral/jntp/tools/miniconda3/4.8.3/envs/ufs-weather-model/bin/ecflow_start.sh be changed?

[BrianCurtis-NOAA]

Derecho has it's own ecflow install through module load ecflow, if it's easier to use that.

Once you module load ecflow it creates the paths to use their ecflow_start.sh. I am not sure though, how much that complicates ecflow package on python.

[natalie-perlin]

There is an ecflow_start.sh script already:
ECFLOW_START=/glade/work/epicufsrt/contrib/spack-stack/derecho/ecflow-5.8.4/bin/ecflow_start.sh

[natalie-perlin]

Derecho has it's own ecflow install through module load ecflow, if it's easier to use that.

Once you module load ecflow it creates the paths to use their ecflow_start.sh. I am not sure though, how much that complicates ecflow package on python.

@BrianCurtis-NOAA - it might be more handy to use the same ecflow version as used during the spack-stack build

[zach1221]

I have this setup for Derecho in rt.sh.
elif [[ $MACHINE_ID = derecho ]]; then

export PATH=/glade/work/epicufsrt/contrib/derecho/rocoto/bin:$PATH
module use /glade/work/epicufsrt/contrib/spack-stack/derecho/modulefiles
module load ecflow/5.8.4
ECF_PORT=$(( $(id -u) + 1500 ))
ECFLOW_START=/glade/work/epicufsrt/contrib/spack-stack/derecho/ecflow-5.8.4/bin/ecflow_start.sh

Getting: ImportError: /glade/u/home/zshrader/miniconda3/bin/../lib/libstdc++.so.6: version `GLIBCXX_3.4.29' not found (required by /glade/work/epicufsrt/contrib/spack-stack/derecho/ecflow-5.8.4/lib/python3.10/site-packages/ecflow/ecflow.so)

[natalie-perlin]

These changes to rt.sh allowed a job to enter the queue (still Queued ):

elif [[ $MACHINE_ID = derecho ]]; then

 module use /glade/work/epicufsrt/contrib/derecho/rocoto/modulefiles
 module load rocoto
 module use /glade/work/epicufsrt/contrib/spack-stack/derecho/modulefiles
 module load ecflow/5.8.4
 module unload ncarcompilers
 module use /glade/work/epicufsrt/contrib/spack-stack/derecho/spack-stack-1.5.1/envs/unified-env/install/modulefiles/Core
 module load stack-intel/2021.10.0
 module load stack-python/3.10.8
 ECFLOW_START=/glade/work/epicufsrt/contrib/spack-stack/derecho/ecflow-5.8.4/bin/ecflow_start.sh

I could also suggest to modify the following in the ./modulefiles/ufs_derecho.intel.lua:

Change the line:
prepend_path("MODULEPATH", "/lustre/desc1/scratch/epicufsrt/contrib/modulefiles")
to
prepend_path("MODULEPATH", "/glade/work/epicufsrt/contrib/spack-stack/derecho/modulefiles")

[natalie-perlin]

@zach1221 - with these changes, a test cpld_control_p8_mixedmode_intel passed successfully (only this test is set in rt.conf).
Regression test log: /glade/derecho/scratch/nperlin/UFS-WM/ufs-weather-model/tests/logs/RegressionTests_derecho.log

[zach1221]

/glade/derecho/scratch/nperlin/UFS-WM/ufs-weather-model/tests

Seems to be working @natalie-perlin . Thank you,

[zach1221]

Full rt.conf has been run on Derecho and logs pushed to PR #2061 as well as the changes Natalie recommended committed to the PR. I have also run regional_atmaq_debug_intel, cpld_control_p8_faster_intel, cpld_bmark_p8, cpld_restart_bmark_p8 and conus13km_debug_qr separately on Derecho to confirm they're working. I'll update the baselines on Derecho and enable those cases in an upcoming WM PR.