Add fms2-io to MOM restart interfaces

[wrongkindofdoctor]

This PR contains the upgrades to save_restart and restore_state, and their dependencies required to use the fms2-io interfaces. Major changes include:

• making restore_state and save_restart wrapper routines with optional use_fms2 and write_ic arguments that, when present and .true., call the routines that contain the appropriate fms2-io interfaces
• adding the logical argument use_fms2=.true. to save_restart and restore_state calls in MOM6 code
• making create_file an interface with routines that wrap calls to mpp-io and fms2-io routines
• adding the module MOM_axis.F90 with helper routines for defining, querying, reading, and writing axes using fms2-io
• adding the module MOM_read_data_fms2.F90 with MOM_read_data routines that wrap fms2-io read_data calls
• adding the module MOM_write_field_fms2.F90 with write_field routines that wrap fms2-io write_data calls
• note that the MOM_read_data and MOM_write_field interfaces that are called outside of the framework modules will be updated in a separate PR.

[wrongkindofdoctor]

@marshallward Yes, this is ready for review. If you feel some changes are unnecessary with the configuration file, you can modify them as you see fit; my intent is to sync with FMS beta releases. I have implemented the fms2 selection in save_restart using the wrapper routine instead of an interface as requested, so it would help if you provided an example.

[marshallward]

There should not be any changes outside of framework. Removing the use_fms2 argument and controlling this inside of save_restart (hard-coded for now) should work, yes?

[wrongkindofdoctor]

@marshallward Would use_fms2 =.true. set as module variable in MOM_restart rather than a flag suffice?

[marshallward]

Yes, I think a module parameter would be fine. We can work out something more elegant on our end if needed.

[adcroft]

We don't use module variables (https://github.com/NOAA-GFDL/MOM6/wiki/Code-style-guide#global--module-data). What's wrong with a run-time parameter?

[wrongkindofdoctor]

@marshallward @adcroft I think we need to meet offline to flesh out the requirements.

[Hallberg-NOAA]

It seems to me that get_time_index should include proper error handling for the case when the time being requested is outside of the range of times spanned by the time index of the input file.

[sanAkel]

@marshallward , @wrongkindofdoctor a question from myself and @mathomp4

Once this PR is merged, does it

• require FMS 2020.03?
• or it can use FMS 2020.03?

Knowing this or any other requirements would help us update since we the same FMS in MOM6 and FV3. Thanks a lot!

[adcroft]

I think it does require the new FMS (for single column mode there's a change in MOM6 required). We've been sitting on this because downstream users are still recovering from the last time we changed the version of FMS. There is an answer change associated with i/o due to a bug fix in FMS which affects a few setups. While we know how to recover answers with .03 there's another release in which it will be easier to disable the fix (if needed, via the namelist), coming in .04. Since these require changes to either build or run scripts, as well as MOM6, we figured it better to bundle the next two changes together and do so in a focused PR with nothing else in it. This is happening and it's in our hands now - it's been in the pipe for a long time but we're cognizant of how painful it is downstream and are trying minimize impact by combining the changes.

[mathomp4]

@adcroft I'd been avoiding it only because syncing up forks is so annoying. But if all I need to do is target getting, say, 2020.04 in, that might make things easier (I have to make sure we change the constants file in all our releases).

And @sanAkel Did we make a PR for changing all the call flush() to flush() in FMS? If not, we/I might want to

[sanAkel]

I think it does require the new FMS (for single column mode there's a change in MOM6 required). We've been sitting on this because downstream users are still recovering from the last time we changed the version of FMS. There is an answer change associated with i/o due to a bug fix in FMS which affects a few setups. While we know how to recover answers with .03 there's another release in which it will be easier to disable the fix (if needed, via the namelist), coming in .04. Since these require changes to either build or run scripts, as well as MOM6, we figured it better to bundle the next two changes together and do so in a focused PR with nothing else in it. This is happening and it's in our hands now - it's been in the pipe for a long time but we're cognizant of how painful it is downstream and are trying minimize impact by combining the changes.

@adcroft Thanks for that heads up! I do use the single column mode, but straight off of MOM6-examples, @mathomp4 , that has nothing to do with anything in GEOS.

[sanAkel]

And @sanAkel Did we make a PR for changing all the call flush() to flush() in FMS? If not, we/I might want to

@mathomp4 I don't recall that you did. As much as I know, it is in our: GEOS-ESM, that's it.

[mathomp4]

And @sanAkel Did we make a PR for changing all the call flush() to flush() in FMS? If not, we/I might want to

@mathomp4 I don't recall that you did. As much as I know, it is in our: GEOS-ESM, that's it.

Okay.

So fair warning to @adcroft and @marshallward and all the GFDL folks. I'll probably make an Issue/PR to FMS and MOM5 MOM6 about this. Essentially we found that when using Intel Fortran on macOS that for some reason, Intel was getting confused by call FLUSH() calls. This is probably because the call FLUSH() subroutine is a Fortran extension. So we've been internally in GEOS and in our forks, converting all the call FLUSH() to use the Fortran Standard FLUSH() intrinsic (which I think every good Fortran compiler now should support) and that seems to be helping.

I'm going to ping Tom Clune to help me get the wording right for the Issue/PR so it sounds better than my "it made things work" explanation :)

[sanAkel]

@mathomp4 you meant MOM6?

[mathomp4]

@mathomp4 you meant MOM6?

Sigh. Yeah. Too many years typing MOM5. :)

[marshallward]

I agree with this, flush() should be used over the call flush() vendor extension.

---

These are the instances I can find:

$ grep -ir "call \+flush" ../../{src,config_src}
../../src/diagnostics/MOM_sum_output.F90:  call flush_file(CS%fileenergy_nc)
../../src/diagnostics/MOM_PointAccel.F90:    call flush(file)
../../src/diagnostics/MOM_PointAccel.F90:    call flush(file)
../../src/framework/MOM_write_cputime.F90:    call flush(CS%fileCPU_ascii)
../../src/framework/MOM_write_cputime.F90:    call flush(CS%fileCPU_ascii)
../../src/framework/MOM_domains.F90:!$   call flush(6)
../../config_src/solo_driver/MOM_driver.F90:!$  call flush(6)

[mathomp4]

I agree with this, flush() should be used over the call flush() vendor extension.

These are the instances I can find:

$ grep -ir "call \+flush" ../../{src,config_src}
../../src/diagnostics/MOM_sum_output.F90:  call flush_file(CS%fileenergy_nc)
../../src/diagnostics/MOM_PointAccel.F90:    call flush(file)
../../src/diagnostics/MOM_PointAccel.F90:    call flush(file)
../../src/framework/MOM_write_cputime.F90:    call flush(CS%fileCPU_ascii)
../../src/framework/MOM_write_cputime.F90:    call flush(CS%fileCPU_ascii)
../../src/framework/MOM_domains.F90:!$   call flush(6)
../../config_src/solo_driver/MOM_driver.F90:!$  call flush(6)

This looks right to me:

❯ git diff dev/gfdl --name-only
config_src/solo_driver/MOM_driver.F90
src/diagnostics/MOM_PointAccel.F90
src/framework/MOM_domains.F90
src/framework/MOM_write_cputime.F90

Would you like me to make a PR here and with FMS?

Okay. Here's the MOM6 PR: https://github.com/NOAA-GFDL/MOM6/pull/1264 and here is the FMS: NOAA-GFDL/FMS#628

[adcroft]

FMS 2020.04 was just released (NOAA-GFDL/FMS#632) so now we move forward on this PR.

Recall we said on a previous dev call that we'd let everyone sync up their dev branches to main and then we'd send this in isolation to main. So start preparing tags/PRs to main. Definitely a good thing to try to do during AGU, and a hackathon! 😁 At GFDL, we need to discuss which of the outstanding PRs make it into our own PR to main (#1268 might slow things down for everyone). We'll certainly not consider new ones until this is done. Looks like we should have a dev call Monday...