Land component model configurations that has active atmosphere component are not working in debug mode

[uturuncoglu]

Description

The land component model RT (control_p8_atmlnd) is failing when it is compiled with -DDEBUG=ON option. This is mainly caused by the inconsistencies in the input files (fix files vs. sfc_data file). @barlage did extra checking in the input files and found following,

---

What I mean below by "not consistent" is that there are not valid land characteristics over all grids where land_frac>0. From what I can tell:

• those used in the RT by default are not consistent, e.g.

fixpath = "/scratch1/NCEPDEV/stmp2/Michael.Barlage/FV3_RT/rt_114204/control_p8_atmlnd_intel/INPUT/"
sfcpath = "/scratch1/NCEPDEV/stmp2/Michael.Barlage/FV3_RT/rt_114204/control_p8_atmlnd_intel/INPUT/"

• I believe these are coming from here and they are also not consistent (doesn't matter which oro file is used):

fixpath = "/scratch2/NAGAPE/epic/UFS-WM_RT/NEMSfv3gfs/input-data-20221101/FV3_fix_tiled/C96/"
sfcpath = "/scratch2/NAGAPE/epic/UFS-WM_RT/NEMSfv3gfs/input-data-20221101/FV3_input_data/INPUT_L127/"

Note these are consistent for non-fractional grids.

• these "v2" data are consistent for fractional grids and probably should be used:

fixpath = "/scratch2/NAGAPE/epic/UFS-WM_RT/NEMSfv3gfs/input-data-20221101/FV3_input_data/INPUT_L127_v2_sfc/fix_sfc/"
sfcpath = "/scratch2/NAGAPE/epic/UFS-WM_RT/NEMSfv3gfs/input-data-20221101/FV3_input_data/INPUT_L127_v2_sfc/"

• these also seem consistent, but may be dangerous to use since they are a mix of two dirs and may not be consistent in other ways

fixpath = "/scratch2/NAGAPE/epic/UFS-WM_RT/NEMSfv3gfs/input-data-20221101/FV3_input_data/INPUT_L127_v2_sfc/fix_sfc/"
sfcpath = "/scratch2/NAGAPE/epic/UFS-WM_RT/NEMSfv3gfs/input-data-20221101/FV3_input_data/INPUT_L127_mx100_v2_sfc/"

---

So, the solution is to switch to V2 input files. The PR will be created soon to solve the issue.

To Reproduce:

What compilers/machines are you seeing this with? It is tested on both Derecho and also Orion and result is same. It is tested with Intel compiler.
Give explicit steps to reproduce the behavior.

1. git clone --recursive https://github.com/ufs-community/ufs-weather-model.git ufs-weather-model_dev
2. cd ufs-weather-model_dev/tests
3. define debug test for control_p8_atmlnd under rt.conf like following,

COMPILE | atml_debug | intel | -DAPP=ATML -DCCPP_SUITES=FV3_GFS_v16,FV3_GFS_v16_flake,FV3_GFS_v17_p8,FV3_GFS_v17_p8_rrtmgp,FV3_GFS_v15_thompson_mynn_lam3km,FV3_WoFS_v0,FV3_GFS_v17_p8_mynn -D32BIT=ON -DDEBUG=ON | | fv3 |
RUN | control_p8_atmlnd_debug                           | - noaacloud | baseline |

4. Add test file
   cp tests/control_p8_atmlnd tests/control_p8_atmlnd_debug
5. Make required changes in the tests/control_p8_atmlnd_debug like name of the baseline directory etc.
6. Then, run the test using ./rt.sh -a nems -k -n "control_p8_atmlnd_debug intel" command.
7. The test fill fail with the error message shown below.

Additional context

N/A

Output

output logs

274: forrtl: severe (408): fort: (3): Subscript #1 of the array LAIM_TABLE has value 0 which is less than the lower bound of 1
274:
274: Image              PC                Routine            Line        Source
274: fv3.exe            000000000E4B527F  Unknown               Unknown  Unknown
274: fv3.exe            000000000E374F4E  lnd_comp_types_mp         731  lnd_comp_types.F90
274: fv3.exe            000000000E2FA68B  lnd_comp_driver_m         286  lnd_comp_driver.F90
274: fv3.exe            000000000E2EBE7C  lnd_comp_nuopc_mp         458  lnd_comp_nuopc.F90

[uturuncoglu]

The mismatch is the input files also creates other side effects. More information can be also found in the conversation of NOAA-EMC/noahmp#8.

[uturuncoglu]

The NoahMP cap needs to be fixed and also RTs that uses active atmosphere needs to be updated to use V2 sfc files to solve the issue. Our initial work also shows that there could be a bug in CCPP/physics side while treating t2m and q2m coming from component land model since we are seeing zero values around the coastline for those fields. I am working on it and once everything fixed, I'll create UFS Weather Model PR that also includes debug test for control_p8_atmlnd. We could also add a test for fully coupled configuration since it was requested but at this point I am not sure that test uses V2 files or not.

[uturuncoglu]

This is also related with #1423

[uturuncoglu]

@barlage The land PR that fixes the issue is almost ready. Do you want to use V2 fixed files also for the data atmosphere coupled configurations. That might require to recreate initial conditions for those tests again but not sure. Anyway, let me know if you also want to update those tests.

[DeniseWorthen]

@uturuncoglu I am going to close the related Issue #2178.

[uturuncoglu]

@DeniseWorthen Sure.