GFSv16.2.1 (#899)

RFC 9682 - On WCOSS2, update GFS package to v16.2.1. This includes fixes for two issues:
* Source code and resource configuration changes to address bufr sounding job failures.
* Script changes to address the gfs_forecast job restart issue that happened when the f1 filesystem was degraded.

Refs #399

docs/Release_Notes.gfs.v16.2.1.md

@@ -0,0 +1,129 @@
+GFS V16.2.1 RELEASE NOTES
+
+-------
+PRELUDE
+-------
+
+Several bug fixes for the GFSv16.2 package to resolve issues with the gfs_forecast job (wave restart calculation) and the gfs_atmos_postsnd (bufr sounding) job.
+
+IMPLEMENTATION INSTRUCTIONS
+---------------------------
+
+The NOAA VLab and both the NOAA-EMC and NCAR organization spaces on github.com are used to manage the GFS.v16.2.1 code. The SPA(s) handling the GFS.v16.2.1 implementation need to have permissions to clone VLab gerrit repositories and the private NCAR UPP_GTG repository. All NOAA-EMC organization repositories are publicly readable and do not require access permissions. Please follow the following steps to install the package on WCOSS2:
+
+```bash
+cd $PACKAGEROOT
+mkdir gfs.v16.2.1
+cd gfs.v16.2.1
+git clone -b EMC-v16.2.1 https://github.com/NOAA-EMC/global-workflow.git .
+cd sorc
+./checkout.sh -o
+```
+
+The checkout script extracts the following GFS components:
+
+| Component | Tag         | POC               |
+| --------- | ----------- | ----------------- |
+| MODEL     | GFS.v16.2.0   | Jun.Wang@noaa.gov |
+| GSI       | gfsda.v16.2.0 | Russ.Treadon@noaa.gov |
+| GLDAS     | gldas_gfsv16_release.v.2.0.0 | Helin.Wei@noaa.gov |
+| UFS_UTILS | ops-gfsv16.2.0 | George.Gayno@noaa.gov |
+| POST      | upp_v8.1.2 | Wen.Meng@noaa.gov |
+| WAFS      | gfs_wafs.v6.2.8 | Yali.Mao@noaa.gov |
+
+To build all the GFS components, execute:
+```bash
+./build_all.sh
+```
+The `build_all.sh` script compiles all GFS components. Runtime output from the build for each package is written to log files in directory logs. To build an individual program, for instance, gsi, use `build_gsi.sh`.
+
+Next, link the executables, fix files, parm files etc in their final respective locations by executing:
+```bash
+./link_fv3gfs.sh nco wcoss2
+```
+
+Lastly, link the ecf scripts by moving back up to the ecf folder and executing:
+```bash
+cd ../ecf
+./setup_ecf_links.sh
+```
+
+SORC CHANGES
+------------
+
+* Workflow
+  * `sorc/gfs_bufr.fd/calpreciptype.f`
+  * `sorc/gfs_bufr.fd/meteorg.f`
+
+FIX CHANGES
+-----------
+
+* No changes from GFS v16.2.0
+
+PARM/CONFIG CHANGES
+-------------------
+
+* Workflow
+  * `env/WCOSS2.env` - postsnd adjustments
+  * `parm/config/config.resources.nco.static` - postsnd adjustments
+  * `parm/config/config.resources.emc.dyn` - postsnd adjustments
+  * `parm/transfer/transfer_rdhpcs_gfs_nawips.list`
+
+JOBS CHANGES
+------------
+
+* No changes from GFS v16.2.0
+
+SCRIPT CHANGES
+--------------
+
+* Workflow
+  * `ecf/scripts/gfs/atmos/post_processing/bulletins/jgfs_atmos_fbwind.ecf` - memory adjustment
+  * `ecf/scripts/gfs/atmos/post_processing/bufr_sounding/jgfs_atmos_postsnd.ecf` - resource adjustment
+  * `scripts/exglobal_forecast.sh` - update calculation of starting time of rerun based on if wave restarts exist
+
+MODULE CHANGES
+--------------
+
+* Workflow
+  * `modulefiles/gfs_bufr.wcoss2.lua` - no longer build with -qopenmp
+
+CHANGES TO RESOURCES AND FILE SIZES
+-----------------------------------
+
+* File sizes
+  * No change to GFSv16.2.0.
+* Resource changes
+  * Adjustment to the gfs_atmos_postsnd job resources.
+  * Increase to memory for fbwind job.
+
+PRE-IMPLEMENTATION TESTING REQUIREMENTS
+---------------------------------------
+
+* Which production jobs should be tested as part of this implementation?
+  * The entire GFS v16.2.1 package needs to be installed and tested.
+* Does this change require a 30-day evaluation?
+  * No.
+
+DISSEMINATION INFORMATION
+-------------------------
+
+* Where should this output be sent?
+  * No change from GFS v16.2.0
+* Who are the users?
+  * No change from GFS v16.2.0
+* Which output files should be transferred from PROD WCOSS2 to DEV WCOSS2?
+  * No change from GFS v16.2.0
+* Directory changes
+  * No change from GFS v16.2.0
+* File changes
+  * No change from GFS v16.2.0
+
+HPSS ARCHIVE
+------------
+
+* No change from GFS v16.2.0
+
+JOB DEPENDENCIES AND FLOW DIAGRAM
+---------------------------------
+* No change from GFS v16.2.0

ecf/scripts/gfs/atmos/post_processing/bufr_sounding/jgfs_atmos_postsnd.ecf

@@ -4,7 +4,7 @@
 #PBS -q %QUEUE%
 #PBS -A %PROJ%-%PROJENVIR%
 #PBS -l walltime=02:00:00
-#PBS -l select=2:mpiprocs=20:ompthreads=6:ncpus=120
+#PBS -l select=4:mpiprocs=10:ompthreads=8:ncpus=80
 #PBS -l place=vscatter:exclhost
 #PBS -l debug=true
 
@@ -40,6 +40,8 @@ module list
 export cyc=%CYC%
 export cycle=t%CYC%z
 export USE_CFP=YES
+export MPICH_MPIIO_HINTS_DISPLAY=1
+export OMP_NUM_THREADS=1
 ############################################################
 # CALL executable job script here
 ############################################################

ecf/scripts/gfs/atmos/post_processing/bulletins/jgfs_atmos_fbwind.ecf

@@ -4,7 +4,7 @@
 #PBS -q %QUEUE%
 #PBS -A %PROJ%-%PROJENVIR%
 #PBS -l walltime=00:05:00
-#PBS -l select=1:ncpus=1:mem=2GB
+#PBS -l select=1:ncpus=1:mem=4GB
 #PBS -l place=vscatter
 #PBS -l debug=true
 

env/WCOSS2.env

@@ -181,6 +181,8 @@ elif [ $step = "fv3ic" ]; then
 
 elif [ $step = "postsnd" ]; then
 
+    export MPICH_MPIIO_HINTS_DISPLAY=1
+    export OMP_NUM_THREADS=1
     export NTHREADS_POSTSND=${nth_postsnd:-1}
     export APRUN_POSTSND="$launcher -n $npe_postsnd --depth=$NTHREADS_POSTSND --cpu-bind depth"
     export NTHREADS_POSTSNDCFP=${nth_postsndcfp:-1}

modulefiles/gfs_bufr.wcoss2.lua

@@ -20,6 +20,6 @@ load(pathJoin("w3emc", os.getenv("w3emc_ver")))
 load(pathJoin("bufr", os.getenv("bufr_ver")))
 
 setenv("myFC","ftn")
-setenv("myFCFLAGS","-O3 -convert big_endian -traceback -g -fp-model source -qopenmp")
+setenv("myFCFLAGS","-O3 -convert big_endian -traceback -g -fp-model source")
 setenv("myCPP","/lib/cpp")
 setenv("myCPPFLAGS","-P")

parm/config/config.resources.emc.dyn

@@ -381,8 +381,8 @@ elif [ $step = "postsnd" ]; then
 
     export wtime_postsnd="02:00:00"
     export npe_postsnd=40
-    export nth_postsnd=6
-    export npe_node_postsnd=20
+    export nth_postsnd=8
+    export npe_node_postsnd=10
     export npe_postsndcfp=9
     export npe_node_postsndcfp=1
     if [ $OUTPUT_FILE == "nemsio" ]; then

parm/config/config.resources.nco.static

@@ -318,8 +318,8 @@ elif [ $step = "postsnd" ]; then
 
     export wtime_postsnd="02:00:00"
     export npe_postsnd=40
-    export nth_postsnd=6
-    export npe_node_postsnd=20
+    export nth_postsnd=8
+    export npe_node_postsnd=10
     export npe_postsndcfp=9
     export npe_node_postsndcfp=1
 

parm/transfer/transfer_rdhpcs_gfs_nawips.list

@@ -24,7 +24,7 @@
 # directory are included, so if no exclude patterns match that file, it will be
 # transferred.
 
-_COMROOT_/gfs/_SHORTVER_/gfs._PDY_/  _REMOTEPATH_/com/nawips/_ENVIR_/gfs._PDY_/
+_COMROOT_/gfs/_SHORTVER_/gfs._PDY_/  _REMOTEPATH_/_ENVIR_/com/gfs/_SHORTVER_/gfs._PDY_/
 + /??/
 + /??/atmos/
 + /??/atmos/gempak/

scripts/exglobal_forecast.sh

@@ -181,7 +181,18 @@ if [ $CDUMP = "gfs" -a $rst_invt1 -gt 0 -a $FHMAX -gt $rst_invt1 -a $filecount -
         cycs=$(echo $SDATE | cut -c9-10)
         flag1=$RSTDIR_ATM/${PDYS}.${cycs}0000.coupler.res
         flag2=$RSTDIR_ATM/coupler.res
-        if [ -s $flag1 ]; then
+        #make sure that the wave restart files also exist if cplwav=true
+        waverstok=".true."
+        if [ $cplwav = ".true." ]; then
+          for wavGRD in $waveGRD ; do
+            if [ ! -f ${RSTDIR_WAVE}/${PDYS}.${cycs}0000.restart.${wavGRD} ]; then
+              waverstok=".false."
+            fi
+          done
+        fi
+
+        if [ -s $flag1 -a $waverstok = ".true." ]; then
+	#if [ -s $flag1 ]; then
             CDATE_RST=$SDATE          
             [[ $RERUN = "YES" ]] && break
             mv $flag1 ${flag1}.old
@@ -432,15 +443,15 @@ if [ $cplwav = ".true." ]; then
 
   for wavGRD in $waveGRD ; do
     if [ $RERUN = "NO" ]; then
-      if [ ! -f ${WRDIR}/${sPDY}.${scyc}0000.restart.${wavGRD} ]; then 
-        echo "WARNING: NON-FATAL ERROR wave IC is missing, will start from rest"
+      if [ -f ${WRDIR}/${sPDY}.${scyc}0000.restart.${wavGRD} ]; then
+        $NLN ${WRDIR}/${sPDY}.${scyc}0000.restart.${wavGRD} $DATA/restart.${wavGRD}
+      else
+        echo "WARNING: Wave ICs are absent, will start from rest"
       fi
-      $NLN ${WRDIR}/${sPDY}.${scyc}0000.restart.${wavGRD} $DATA/restart.${wavGRD}
     else
-      if [ ! -f ${RSTDIR_WAVE}/${PDYT}.${cyct}0000.restart.${wavGRD} ]; then
-        echo "WARNING: NON-FATAL ERROR wave IC is missing, will start from rest"
+      if [ -f ${RSTDIR_WAVE}/${PDYT}.${cyct}0000.restart.${wavGRD} ]; then
+        $NLN ${RSTDIR_WAVE}/${PDYT}.${cyct}0000.restart.${wavGRD} $DATA/restart.${wavGRD}
       fi
-      $NLN ${RSTDIR_WAVE}/${PDYT}.${cyct}0000.restart.${wavGRD} $DATA/restart.${wavGRD}
     fi
     eval $NLN $datwave/${wavprfx}.log.${wavGRD}.${PDY}${cyc} log.${wavGRD}
   done

sorc/gfs_bufr.fd/calpreciptype.f

@@ -77,7 +77,7 @@ SUBROUTINE CALPRECIPTYPE(kdt,nrcm,im,ix,lm,lp1,randomno,      &
       ALLOCATE ( RH(LM),TD8(LM),TWET8(LM) )
 
 ! Create look up table
-      call gfuncphys
+!     call gfuncphys
 
       time_vert    = 0.
       time_ncep    = 0.

sorc/gfs_bufr.fd/meteorg.f

@@ -63,6 +63,7 @@ subroutine meteorg(npoint,rlat,rlon,istat,cstat,elevstn,
       use sigio_module 
       use physcons
       use mersenne_twister
+!      use funcphys, only : gfuncphys
       use funcphys
       implicit none 
       include 'mpif.h'
@@ -1109,6 +1110,7 @@ subroutine meteorg(npoint,rlat,rlon,istat,cstat,elevstn,
 !  prepare buffer data
 !
         if(iope == 0) then
+        call gfuncphys
         do np = 1, npoint
           pi3(np,1)=psn(np)*1000
           do k=1,levs