Feature/update mom6 20220329 mixed fms

[jiandewang]

PR Checklist

• This PR is up-to-date with the top of all sub-component repositories except for those sub-components which are the subject of this PR. Please consult the ufs-weather-model wiki if you are unsure how to do this.

• This PR has been tested using a branch which is up-to-date with the top of all sub-component repositories except for those sub-components which are the subject of this PR

• An Issue describing the work contained in this PR has been created either in the subcomponent(s) or in the ufs-weather-model. The Issue should be created in the repository that is most relevant to the changes in contained in the PR. The Issue and the dependent sub-component PR
  are specified below.

• Results for one or more of the regression tests change and the reasons for the changes are understood and explained below.

• New or updated input data is required by this PR. If checked, please work with the code managers to update input data sets on all platforms.

Instructions: All subsequent sections of text should be filled in as appropriate.

The information provided below allows the code managers to understand the changes relevant to this PR, whether those changes are in the ufs-weather-model repository or in a subcomponent repository. Ufs-weather-model code managers will use the information provided to add any applicable labels, assign reviewers and place it in the Commit Queue. Once the PR is in the Commit Queue, it is the PR owner's responsiblity to keep the PR up-to-date with the develop branch of ufs-weather-model.

Description

This PR is for the updating of MOM6 with fixing in infra/FMS1 and FMS2 code aimed for UFS FMS mixed mode.

Issue(s) addressed

• fixes update MOM6 20220329 #1148
• fixes MOM6 PR An alternate fix to class(*) issues with FMS 2022-01 NOAA-EMC/MOM6#88

Testing

How were these changes tested? What compilers / HPCs was it tested with? Are the changes covered by regression tests? (If not, why? Do new tests need to be added?) Have regression tests and unit tests (utests) been run? On which platforms and with which compilers? (Note that unit tests can only be run on tier-1 platforms)

• hera.intel
• hera.gnu
• orion.intel
• cheyenne.intel
• cheyenne.gnu
• gaea.intel
• jet.intel
• wcoss_cray
• wcoss_dell_p3
• opnReqTest for newly added/changed feature
• CI

Dependencies

If testing this branch requires non-default branches in other repositories, list them. Those branches should have matching names (ideally).

Do PRs in upstream repositories need to be merged first?
If so add the "waiting for other repos" label and list the upstream PRs

• waiting on noaa-emc/nems/pull/<pr_number>
• waiting on noaa-emc/fv3atm/pull/<pr_number>

[jiandewang]

we need to fully test this PR as it is a critical updating for the upcoming FMS mixed mode. This is my suggestions:

• test it in current UFS to make sure it won't break current system (@jiandewang and @DeniseWorthen )
• test in FMS mixed mode style (@MinsukJi-NOAA and @binli2337 )
• test in hurricane group region setting (John Steffen)
• test in current DA system (@guillaumevernieres)

[jiandewang]

@DeniseWorthen and @jiandewang confirmed that it reproduced current UFS baseline for intel/gnu on all platforms.

[MinsukJi-NOAA]

@DeniseWorthen and @jiandewang confirmed that it reproduced current UFS baseline for intel/gnu on all platforms.

@jiandewang How did you verify against develop-20220328 on Hera? That baseline does not work for me.

[binli2337]

Tests using files from MOM6/config_src/infra/FMS1:

Full RT tests have been run on Gaea.
All tests with 64 bit atmosphere or datm passed.
All tests with 32 bit atmosphere completed the run, but results are different from the baseline.

[jiandewang]

John confirmed it works fine for hurricane regional setting.

[jiandewang]

@DeniseWorthen and @jiandewang confirmed that it reproduced current UFS baseline for intel/gnu on all platforms.

@jiandewang How did you verify against develop-20220328 on Hera? That baseline does not work for me.

@DeniseWorthen my test was done 2 days ago, at that time the BL was develop-20220325. Now BL is 20220329 so I am going to re-run my branch.

[MinsukJi-NOAA]

UFS test branch can be found here. This uses the ufs weather model commit 995fed5.

Full regression tests for Hera/Intel:

• All tests with 64 bit atmosphere or datm passed. These include hafs_regional_datm_cdeps and control_atm_aerosols.
• All tests with 32 bit atmosphere completed the run, but failed due to differing results. This is as expected.

These results are for both FMS1 and FMS2 infra.

[MinsukJi-NOAA]

Full regression tests for Hera/GNU:

• All tests with 64 bit atmosphere or datm passed: These include cpld_control_c96_noaero_p8, cpld_debug_noaero_p8, datm_cdeps_control_cfsr, and control.
• All tests with 32 bit atmosphere completed the run, but failed due to differing results. This is as expected.

These results are for both FMS1 and FMS2 infra.

[jiandewang]

@DeniseWorthen and @jiandewang confirmed that it reproduced current UFS baseline for intel/gnu on all platforms.

@jiandewang How did you verify against develop-20220328 on Hera? That baseline does not work for me.

@DeniseWorthen my test was done 2 days ago, at that time the BL was develop-20220325. Now BL is 20220329 so I am going to re-run my branch.

@DeniseWorthen re-ran on HERA with 0329 BL is fine.

[MinsukJi-NOAA]

Full regression tests for Wcoss Dell/Intel:

• All tests with 64 bit atmosphere or datm passed
• All tests with 32 bit atmosphere completed the run, but failed due to differing results. This is as expected.

These results are for both FMS1 and FMS2 infra.

[binli2337]

Tests using files from MOM6/config_src/infra/FMS2:

Full RT tests have been run on Gaea.
All tests with 64 bit atmosphere or datm passed.
All tests with 32 bit atmosphere completed the run, but results are different from the baseline.

[jiandewang]

thanks @MinsukJi-NOAA and @binli2337 for your careful testing.
@junwang-noaa can you put this PR in the commit queue ?

[jiandewang]

per discussion with Jun, we will combine this PR with #1137