Spack-stack moved from /lfs4 to /contrib on Jet

[InnocentSouopgui-NOAA]

Spack stack moved from the storage /lfs4 to a new storage space /contrib, following the failure of /lfs4.

The requires the update of module files for all softwares supported on Jet

Refs NOAA-EMC/global-workflow#2377

[InnocentSouopgui-NOAA]

@InnocentSouopgui-NOAA is working on this issue

[InnocentSouopgui-NOAA]

ice_blend consistency test is failing on segfault while running copygb2

+ /contrib/spack-stack/spack-stack-1.6.0/envs/unified-env-rocky8/install/intel/2021.5.0/grib-util-1.3.0-74mdurc/bin/copygb2 -x -i3 -g '0 0 0 0 0 0 0 0 4320 2160 0 0 89958000 42000 48 -89958000 359958000 83000 83000 0' ims.icec.grib2 ims.icec.5min.grib2
forrtl: severe (174): SIGSEGV, segmentation fault occurred
Image              PC                Routine            Line        Source             
copygb2            00000000004D67BA  Unknown               Unknown  Unknown
libpthread-2.28.s  00007FA269B92CF0  Unknown               Unknown  Unknown
copygb2            000000000040A85E  Unknown               Unknown  Unknown
copygb2            00000000004125EC  Unknown               Unknown  Unknown
copygb2            0000000000408377  Unknown               Unknown  Unknown
copygb2            0000000000407122  Unknown               Unknown  Unknown
libc-2.28.so       00007FA2697F5D85  __libc_start_main     Unknown  Unknown
copygb2            000000000040702E  Unknown               Unknown  Unknown

[GeorgeGayno-NOAA]

ice_blend consistency test is failing on segfault while running copygb2

+ /contrib/spack-stack/spack-stack-1.6.0/envs/unified-env-rocky8/install/intel/2021.5.0/grib-util-1.3.0-74mdurc/bin/copygb2 -x -i3 -g '0 0 0 0 0 0 0 0 4320 2160 0 0 89958000 42000 48 -89958000 359958000 83000 83000 0' ims.icec.grib2 ims.icec.5min.grib2
forrtl: severe (174): SIGSEGV, segmentation fault occurred
Image              PC                Routine            Line        Source             
copygb2            00000000004D67BA  Unknown               Unknown  Unknown
libpthread-2.28.s  00007FA269B92CF0  Unknown               Unknown  Unknown
copygb2            000000000040A85E  Unknown               Unknown  Unknown
copygb2            00000000004125EC  Unknown               Unknown  Unknown
copygb2            0000000000408377  Unknown               Unknown  Unknown
copygb2            0000000000407122  Unknown               Unknown  Unknown
libc-2.28.so       00007FA2697F5D85  __libc_start_main     Unknown  Unknown
copygb2            000000000040702E  Unknown               Unknown  Unknown

Can you check in your changes to your branch so I can run exactly what you are?

How are you invoking the driver script?

[InnocentSouopgui-NOAA]

I pushed all my updates to https://github.com/InnocentSouopgui-NOAA/UFS_UTILS/tree/migration-jet-contrib

To run the ice_blend test

1. set the account for the job inside the script
2. set Work directory export WORK_DIR=/lfs5/NESDIS/nesdis-rdo2/Innocent.Souopgui/stmp
3. cal the script from inside the directory reg_tests/ice_blend as:
   ./driver.jet.sh

All other consistency tests were successful.

[InnocentSouopgui-NOAA]

@GeorgeGayno-NOAA
Have you had the chance to look at this problem again?

In case it takes longer, I suggest we open a separate issue for this problem and move forward to deliver Global workflow on Jet?
People who run global Workflow on Jet are not able to run since the crash of lfs4.

What are your thoughts?

[GeorgeGayno-NOAA]

I pushed all my updates to https://github.com/InnocentSouopgui-NOAA/UFS_UTILS/tree/migration-jet-contrib

To run the ice_blend test

1. set the account for the job inside the script
2. set Work directory export WORK_DIR=/lfs5/NESDIS/nesdis-rdo2/Innocent.Souopgui/stmp
3. cal the script from inside the directory reg_tests/ice_blend as:
   ./driver.jet.sh

All other consistency tests were successful.

Oh. Step 3 might be the problem. Try sbatch driver.jet.sh.

[InnocentSouopgui-NOAA]

Thank you,
I did not pay full attention that this script calling sequence was different from others.

Everything works.
Let's move forward with the pull request #978

[GeorgeGayno-NOAA]

A few minor bugs need to be fixed. Reopening.