APP=S2S free forecast fails after model tag update.

[aerorahul]

Expected behavior
Free-forecast with APP=S2S should work. Instead the model fails.

Current behavior
Free-forecast with APP=S2S got broken after the model hash was updated as part of PR #946. The corresponding model hash is 9ae617. The previous model hash b97375c worked with APP=S2S.

Machines affected
Likely all.

To Reproduce

# checkout global-workflow
$>  git clone https://github.com/noaa-emc/global-workflow

# build 
$> cd global-workflow/sorc
$> ./checkout.sh
$> ./build_all.sh -a S2S
$> ./link_workflow.sh emc hera

# Load relevant modules
$> module use -a /contrib/anaconda/modulefiles
$> module load anaconda/anaconda3-5.3.1
$> module load rocoto/1.3.3

# Setup experiment
$> cd ../workflow

# This is optional, I am just using this to have shorthand.
$> stmp2=/scratch1/NCEPDEV/stmp2/$USER

# Create experiment directory
$> ./setup_expt.py forecast-only --app S2S --resdet 384 --comrot $stmp2/ROTDIR --icsdir $stmp2/FV3ICS --expdir $stmp2/EXPDIR --idate 2013040100 --edate 2013040100 --pslot cpld_s2s

# Create workflow xml
$> ./setup_xml.py $stmp2/EXPDIR/cpld_s2s

# cd into the experiment directory
$> cd $stmp2/EXPDIR/cpld_s2s

# Setup cron to execute the workflow tasks one by one.
$> crontab cpld_s2s.crontab

An example log file for the above experiment is at:

/scratch1/NCEPDEV/stmp2/Rahul.Mahajan/ROTDIR/cpld_s2s/logs/2013040100/gfsfcst.log.0

An example log file for the model hash b97375c (working)

/scratch1/NCEPDEV/stmp2/Rahul.Mahajan/ROTDIR/cpld_s2sw/logs/2013040100/gfsfcst.log

[WalterKolczynski-NOAA]

My first thought was something with diag_table, but that wasn't changed in that PR.

[aerorahul]

@WalterKolczynski-NOAA you had the correct hunch.

I got a tip from @yangfanglin who got cc-ed on an email from @JessicaMeixner-NOAA to someone who also encountered the same failure, instructing to remove the following lines from diag_table_cpl

"wavocn%4yr%2mo%2dy%2hr",      6,  "hours", 1, "hours", "time", 6, "hours", "1901 1 1 0 0 0"

and

#wave_variables
 "ocean_model", "geolon",             "geolon",             "wavocn%4yr%2mo%2dy%2hr","all", .false., "none", 2
 "ocean_model", "geolat",             "geolat",             "wavocn%4yr%2mo%2dy%2hr","all", .false., "none", 2
 "ocean_model", "MSTAR",              "MSTAR",              "wavocn%4yr%2mo%2dy%2hr","all", .true.,  "none", 2
 "ocean_model", "LA",                 "LA",                 "wavocn%4yr%2mo%2dy%2hr","all", .true.,  "none", 2
 "ocean_model", "LA_MOD",             "LA_MOD",             "wavocn%4yr%2mo%2dy%2hr","all", .true.,  "none", 2
 "ocean_model", "MSTAR_LT",           "MSTAR_LT",           "wavocn%4yr%2mo%2dy%2hr","all", .true.,  "none", 2
 "ocean_model", "ePBL_h_ML",          "ePBL",               "wavocn%4yr%2mo%2dy%2hr","all", .true.,  "none", 2
 "ocean_model", "ustar",              "ustar",              "wavocn%4yr%2mo%2dy%2hr","all", .true.,  "none", 2
 "ocean_model", "ePBL_TKE_wind",      "ePBL_TKE_wind",      "wavocn%4yr%2mo%2dy%2hr","all", .true.,  "none", 2
 "ocean_model", "ePBL_TKE_MKE",       "ePBL_TKE_MKE",       "wavocn%4yr%2mo%2dy%2hr","all", .true.,  "none", 2
 "ocean_model", "ePBL_TKE_conv",      "ePBL_TKE_conv",      "wavocn%4yr%2mo%2dy%2hr","all", .true.,  "none", 2
 "ocean_model", "ePBL_TKE_forcing",   "ePBL_TKE_forcing",   "wavocn%4yr%2mo%2dy%2hr","all", .true.,  "none", 2
 "ocean_model", "ePBL_TKE_mixing",    "ePBL_TKE_mixing",    "wavocn%4yr%2mo%2dy%2hr","all", .true.,  "none", 2
 "ocean_model", "ePBL_TKE_mech_decay","ePBL_TKE_mech_decay","wavocn%4yr%2mo%2dy%2hr","all", .true.,  "none", 2
 "ocean_model", "ePBL_TKE_conv_decay","ePBL_TKE_conv_decay","wavocn%4yr%2mo%2dy%2hr","all", .true.,  "none", 2

Doing so, allowed the model with APP=S2S to proceed past the failure.

I should have scrolled up in the failed log to notice:

408: FATAL from PE     0: NetCDF: Numeric conversion not representable: netcdf_write_data_4d: file:wavocn_2013_04_01_03.nc variable: LA_MOD

Even so, I wouldn't have known what it meant, rather than just peek my curiosity about why the model would be writing out "wave variables" for a non-wave configuration.

Thank you @yangfanglin and @JessicaMeixner-NOAA for the solution.

[JessicaMeixner-NOAA]

These variables are for evaluating wave-ocean coupling in prototypes. These will not be variables output in operations, so perhaps it's time to take these out as P8 has completed. I'll open a PR with this removal and check to make sure there's no other updates that need to be taken out, such as in the archiving.

[JessicaMeixner-NOAA]

A branch with the proposed fix is here: https://github.com/JessicaMeixner-NOAA/global-workflow/tree/bug/s2sapp

I'll submit a PR after a S2S free forecast job test has succeeded.