* 'dev/gfdl' of github.com:NOAA-GFDL/MOM6:
  Fix to out-of-bounds error
  Slight improvement in writeMessageAndDesc
  Corrected recent bug in writeMessageAndDesc
  Included H_subroundoff in a denominator
  +Added code to handle tab lengths in documentation
  Added explicit form of cpu_clock_id
  (*)Improve ePBL when EPBL_ORIGINAL_PE_CALC=False
  Alternate fix for divide-by-zero.
  Clean up some divide-by-zero issues in OBCs.
  Added a space to a description.
  Codecov: Set base as parent (target) branch
  Fixes an integer-kind mismatch in MOM_random, seed_from_time() (#1113)

Merge in dev/gfdl updates

Sync with MOM6 dev/gfdl

Sync with NOAA-GFDL dev/gfdl

sync with dev/gfdl

Sync with NOAA-GFDL dev/gfdl

Sync with NOAA-GFDL dev/gfdl

Sync with NOAA-GFDL

Sync with NOAA-GFDL dev/gfdl

…ons of the routines with the fms-io or fms2-io interfaces

added module use statments for fms2_io and MOM_io helper routines to MOM_restart
added use_fms2=.true. arguments to save_restart and restore_state calls
added write_ic=.true. to the save_restart call in MOM.F90

added module MOM_axis with routines to define and register axes and their metadata
added module MOM_read_data_fms2.F90 with wrappers for fms2_io read_data interfaces and required routines
added module MOM_write_field_fms2.F90 with wrappers for fms2_io write_data interfaces
updated module use statments in MOM_io and MOM_restart to reference routines in MOM_read_data_fms2, MOM_write_field_fms2, and MOM_axis
made write_field and create_file interfaces in MOM_io
added create_file routines to MOM_io that accept file names or file objects to create/overwrite netcdf files that will be written to via write_field calls

fixed compile-time errors
added new MOM_read_data routines to MOM_read_data interface in MOM_io
added placeholder call for new write_field routines to MOM_io

changed use_fms2 to a required first argument in save_restart_fms2 and restore_state_fms2
changed write_ic to a required argument in write_initial_conditions

fixed the layer and interface checks in MOM_get_diagnostic_axis_data

commented out manual checksum registration in save_restart_fms2 so that internal fms2-io checksum computation is used

added checks for time units to restore_state and save_restart
added logic to make the restart time 1 to save_restart_fms2 if there is an abnormally large value passed to the routine
added interface routine file_exists_FMS2 that uses the fms2_io file_exists call

added subroutine get_num_restart_files to MOM_restart that searches for known variants of the input file names and returns then number of restart files available for querying, and the optional list of filepaths
added loop to search the files for all mandatory variables in the list of file paths returned to by cal to get_num_restart_files
added calls to get the variable dimension names and pass them as arguments to register_restart_field in restore_state_fms2

removed exit from inner variable loop in restore_state_fms2
code cleanup

moved missing_fields=0 outside of the CS loop in restore_state_fms2

moved missing_fields=0 outside of the CS loop in restore_state_fms2

converted save_restart and restore_state to interface that call versions of the routines with the fms-io or fms2-io interfaces
added module use statments for fms2_io and MOM_io helper routines to MOM_restart
added use_fms2=.true. arguments to save_restart and restore_state calls
added write_ic=.true. to the save_restart call in MOM.F90

added module MOM_axis with routines to define and register axes and their metadata
added module MOM_read_data_fms2.F90 with wrappers for fms2_io read_data interfaces and required routines
added module MOM_write_field_fms2.F90 with wrappers for fms2_io write_data interfaces
updated module use statments in MOM_io and MOM_restart to reference routines in MOM_read_data_fms2, MOM_write_field_fms2, and MOM_axis
made write_field and create_file interfaces in MOM_io
added create_file routines to MOM_io that accept file names or file objects to create/overwrite netcdf files that will be written to via write_field calls

fixed compile-time errors
added new MOM_read_data routines to MOM_read_data interface in MOM_io
added placeholder call for new write_field routines to MOM_io

changed use_fms2 to a required first argument in save_restart_fms2 and restore_state_fms2
changed write_ic to a required argument in write_initial_conditions

fixed the layer and interface checks in MOM_get_diagnostic_axis_data

commented out manual checksum registration in save_restart_fms2 so that internal fms2-io checksum computation is used

added checks for time units to restore_state and save_restart
added logic to make the restart time 1 to save_restart_fms2 if there is an abnormally large value passed to the routine
added interface routine file_exists_FMS2 that uses the fms2_io file_exists call

added subroutine get_num_restart_files to MOM_restart that searches for known variants of the input file names and returns then number of restart files available for querying, and the optional list of filepaths
added loop to search the files for all mandatory variables in the list of file paths returned to by cal to get_num_restart_files
added calls to get the variable dimension names and pass them as arguments to register_restart_field in restore_state_fms2

removed exit from inner variable loop in restore_state_fms2
code cleanup

removed test workflow directory

moved missing_fields=0 outside of the CS loop in restore_state_fms2

converted save_restart and restore_state to interface that call versions of the routines with the fms-io or fms2-io interfaces
added module use statments for fms2_io and MOM_io helper routines to MOM_restart
added use_fms2=.true. arguments to save_restart and restore_state calls
added write_ic=.true. to the save_restart call in MOM.F90

added module MOM_axis with routines to define and register axes and their metadata
added module MOM_read_data_fms2.F90 with wrappers for fms2_io read_data interfaces and required routines
added module MOM_write_field_fms2.F90 with wrappers for fms2_io write_data interfaces
updated module use statments in MOM_io and MOM_restart to reference routines in MOM_read_data_fms2, MOM_write_field_fms2, and MOM_axis
made write_field and create_file interfaces in MOM_io
added create_file routines to MOM_io that accept file names or file objects to create/overwrite netcdf files that will be written to via write_field calls

fixed compile-time errors
added new MOM_read_data routines to MOM_read_data interface in MOM_io
added placeholder call for new write_field routines to MOM_io

changed use_fms2 to a required first argument in save_restart_fms2 and restore_state_fms2
changed write_ic to a required argument in write_initial_conditions

fixed the layer and interface checks in MOM_get_diagnostic_axis_data

commented out manual checksum registration in save_restart_fms2 so that internal fms2-io checksum computation is used

added checks for time units to restore_state and save_restart
added logic to make the restart time 1 to save_restart_fms2 if there is an abnormally large value passed to the routine
added interface routine file_exists_FMS2 that uses the fms2_io file_exists call

added subroutine get_num_restart_files to MOM_restart that searches for known variants of the input file names and returns then number of restart files available for querying, and the optional list of filepaths
added loop to search the files for all mandatory variables in the list of file paths returned to by cal to get_num_restart_files
added calls to get the variable dimension names and pass them as arguments to register_restart_field in restore_state_fms2

removed exit from inner variable loop in restore_state_fms2
code cleanup

removed test workflow directory

removed white space

updated FMS tag in .testing Makefile
fixed argument comments to use doxygen style in MOM_write_field_fms2

code cleanup

removed \TODO from MOM_restart
fixed documentation for module variables in MOM_read_data_fms2 and MOM_write_field_fms2

removed doxygenized TODO statements

removed new MOM_read_data routines from interface, and commented out calls in MOM_io
fixed doxygen definitions

more doxygen fixes

changed save_restart and restore_state to wrapper routines with opitional use_fms2 and write_ic arguments
added use_fms2=.true. and write_ic=.true. to save_restart and restore_state calls

removed the error messages from append_substring
tried reordering the write_ic and use_fms2 checks in save_restart to fix error with invalid memory reference in the MOM.F90 call to save_restart-write_initial_conditions

…ing the use_fms2 and write_ic flags if present to avoid invalid memory reference error

added str_len argument to register_variable_attribute calls

added support to for rotated fields to write_initial_conditions and save_restart_fms2

removed whitespace

removed use_fms2 optional arguments in save_restart, restore_state, and create_file
made use_fms2 a MOM_restart module variable

Update MOM_state_initialization.F90

Remove space.

Update MOM_ice_shelf.F90

Add space.

Update MOM_driver.F90

Add space

Update mom_surface_forcing_mct.F90

remove whitespace

Update ocean_model_MOM.F90

Add space

Update mom_ocean_model_nuopc.F90

Add space

Update MOM_surface_forcing.F90

Add space

Changes needed work ESM4 to run with new io

  Corrected three module use statements in infra/FMS1 to eliminate any
dependencies of code in config_src/infra/FMS1 on code in src/framework.  These
use statements eventually point to the same place as before, but with less
indirection.  This change should facilitate later steps to compile everything
in and under the config_src/infra directories as libraries.  All answers are
bitwise identical.

  Duplicated all of the modules in infra/FMS1 into new copies in infra/FMS2, in
preparation for merging in the FMS2 i/o changes to the new infra/FMS2 directory.
All answers are bitwise identical, regardless of which config_src/infra
directory is used.

  Added code to explicitly set a (1x1) io_domain when no other io_layout is
specified, complying with changing requirements for 2020 and later versions of
FMS, and following the default behavior of previous versions.  All answers are
bitwise identical.

…om/wrongkindofdoctor/MOM6 into FMS2_io

  Merged in PR #1165, "Add fms2-io to MOM restart interfaces", but with
modifications to move the new files to config_src/infra/FMS2, where they belong,
and to avoid inappropriate changes to code in the src/framework directory.  All
existing tests are passing with config_src/infra/FMS1, but the new changes
require that the version of FMS used with config_src/infra/FMS2 to at least
2020.03, and this disables all of our AM2-based coupled test cases (due to the
deliberate decision from the GFDL Modeling Systems group to drop support for
this older atmospheric model).

  Although the new code reproduces the previous answers in ocean_only and
ice_ocean_SIS2 test cases with the intel and gnu compilers, with the pgi
(19.10.0) compiler there are fatal errors in all ice_ocean_SIS2 cases and in
ocean_only cases driven by data override.  The last version of FMS where the pgi
compiler works in all cases is 2019.01. The problem in some ocean_only cases
appears to be related to the use of FMS2 i/o calls in the data_override
routines.  With the ice_ocean_SIS2 cases, the error message in with the pgi
compiler is "mpp_broadcast_text: broadcasting from invalid PE.".

 (Note that with the newer versions of FMS (2020.03 and later), OM_1deg requires
that the namelist file for data_override_nml has the added line
{reproduce_null_char_bug = .true.} to reproduce the previous answers, but it is
a fatal error to add this setting with FMS versions before 2020.03.)

  Eliminated scale_data from MOM_read_data_fms2, as it was duplicative of
rescale_comp_data from MOM_domain_infra.F90.  Also eliminated the optional scale
argument from the fms2 write_field routines, because this scaling is already
handled in the framework/MOM_io.F90 routines, and because the way the scaling
was implemented with a pointer to defeat the intent(in) of the data array to
the write routines was a deviously terrible idea that would cause endless
problems.  Because the mpp_domain in the MOM_domain_types have been modified to
always have an io_domain set, the test and calls setting an io_domain in all of
the fms2 read_data and fms2 write_data routines.  In addition, there is no
longer a need for a routine to read specifically from the supergrid, so the fms2
MOM_read_data_2d_supergrid routine was eliminated.  The code compiles with these
changes, but none of this fms2 I/O code is exercised yet in MOM6, so of course
all answers are bitwise identical.

  Replaced the two optional arguments xPosition and yPosition to the two
routines wrapped by the interface MOM_register_variable_axes in MOM_axis.F90
with a single optional argument, position.  Also dramatically refactored the
code in read_data_fms2.F90 to eliminate unused variants and adding more granular
subroutines to dramatically reduce duplicative code, eliminating several
(unexercised) bugs in the process; that file went from 1441 lines long to 924.
The code compiles with these changes, but none of this fms2 I/O code is
exercised yet in MOM6, so of course all answers are bitwise identical.

  Cleaned up MOM_restart.F90 in the wake of the recent merge of FMS2 I/O related
changes into MOM6.  The changes include calling the recently added functions
get_variable_byte_size and get_num_restart_files, and revising the names of
several variables in open_restart_units and the order in which the count of
files is incremented for greater clarity.  There are also a number of
duplicative comments that were removed while others were reformatted.  All
answers and output are bitwise identical.

  Added the option to use FMS2 interfaces to read variables in the FMS2 version
of MOM_io_infra.F90.  For now, this is controlled by a hard-coded module
variable, FMS2_reads, in MOM_io.F90.  Also extensively revised MOM_read_fms2.F90
to add the new public subroutine prepare_to_read_var that is used from
MOM_io_infra.F90, to systematically determine the directions that axes in files
using the cartesian_axis attribute before resorting to using the axis names or
units.  All of the subroutines from MOM_read_fms2.F90 whose functionality has
been merged into MOM_io_infra.F90 have now been removed.  All answers are
bitwise identical and reproduce the FMS1 test cases, but there is a new publicly
visible routine.

  Modified the routines that write out the checksums of various types related to
the surface ocean state so that only the root PE writes the checksums.  The
modified routines are ocn_ice_bnd_type_chksum and ocean_public_type_chksum in
config_src/infra/FMS_cap.  The analogous routines with the nuopc_cap and mct_cap
have not been modified; although doing so would be a very good idea, it should
be done by someone who is actively testing them.  In some large-PE tests cases
this reduces the volume of output to stdout by over 90% without any loss of
information.  All solutions are bitwise identical, but there may be minor
white-space reformatting of the output in some cases.

  Use axistype for MOM_interp_infra directly from mpp_io_mod and add a copy of
get_axis_data to both copies of infra/FMS[12]/MOM_interp_infra.F90, and then use
these in framework/MOM_horizontal_regridding.F90, to permit the MOM6 I/O calls
to use the FMS2 interfaces without simultaneously requiring changes to the
horizontal interpolation code.  All answers are bitwise identical, but there
are changes to the interfaces offered by a public module.

  Added a new optional logical argument to write_metadata_axis to indicate when
an axis is staggered at the edges of the tracer grid, and changed calls to
get_file_info to stop requesting the number of global attributes.  Also
eliminated some unused optional arguments to the FMS1 version of write_metadata
field.  All answers are bitwise identical, but there are minor changes to
some I/O related interfaces.

 Corrected the logic of a warning message in categorize_axis by adding
parentheses.  All answers are bitwise identical, and spurious warnings are no
longer being issued.

  Added a large number of calls to handle all of the writes via the FMS2
interfaces to infra/FMS2/MOM_io_infra.F90.  There are newly defined private
types in MOM_io_infra to wrap the axistype and fieldtype that had previously
been offered from mpp_io_mod.  All answers are bitwise identical and it has been
verified that output files do not change and the restarts are still working.

  Deleted the unused MOM_axis.F90 and MOM_write_field_fms2.F90 modules, now
that MOM_io_infra.F90 has been updated to be able to use FMS2 interfaces for
both reading and writing to files.  As these modules had never been used, they
do not change any answers.

  Added variants of MOM_read_data_0d, MOM_read_data_0d_int, MOM_read_data_1d,
MOM_read_data_1d_int, and MOM_read_data_2d_region that use the FMS2 interfaces
to read data.  Also altered prepare_to_read_var so that it does not open a file
(which had been an option before); one argument to prepare_to_read_var was no
longer needed and so it was removed.  Also added a public interface to
find_varname_in_file that does the same as prepare_to_read_var but works on the
other FMS2 file type without domain decomposition.  Several unused and now
redundant routines were removed from MOM_read_data_fms2.F90. Comments describing
a number of variables were also added.  All of these changes are confined to
config_src/infra/FMS2, and all answers are bitwise identical.

Only write checksums from root PE MCT/NUOPC caps

  Moved the routines prepare_to_read_var and find_varname_in_file, along with
four other routines that prepare_to_read_var calls from MOM_read_data_fms2 to
MOM_io_infra, and eliminated the file MOM_read_data_fms2.F90.  All answers are
bitwise identical, but this rearrangement of identical code does eliminate one
public module, so that the file structure of infra/FMS2 is identical to that of
infra/FMS1.

…before getting currents in A- or B-grid staggering. Now added.

  Add a missing ".nc" suffix to the output filename with FMS2_io, while also
issuing a warning, following the practice of FMS1.  Also reordered the calls to
add the longname and axis attributes to FMS2 files, to follow the order used in
MOM6 calls to FMS1.  All answers are bitwise identical, but there are some
changes to output filenames and orders of attributes in files (to revert to
traditional behavior).

dev/gfdl updates

FMS2 restart routines expect axes to be domain-decomposed.  However, the
domain_write_1d function does not apply this decomposition and instead
routes this operation to compressed_write_1d.

In order to accommodate this, we explicitly slice the 1d arrays of any
axes into its domain-decomposed segment before passing to write_data.
We have also introduced a control flag to MOM's FMS2 axistype to direct
MOM_write_axis when this needs to be applied.

We currently apply the domain decomposition flag to all horizontal axes
regardless of circumstances.  For now this is probably sufficient, but
may need further testing (e.g. cube sphere).

Explicit domain decomposition of horizontal axes

  Corrects reads of hexadecimal checksum attributes from distributed files when
using the FMS2 IO interfaces.  All answers are bitwise identical, and reads and
writes of distributed sets of restart files are now working with the FMS2 IO
interfaces.

+Enable FMS2 interface reads (includes merge of PR #1165)

Only write checksums from root PE for FMS_cap

…urrents

Bugfix in getting surface currents [Additions from GMAO (3)]

  Remove timelevel arguments from MOM_read_data calls for variables that should
never have multiple time-levels.  All answers are bitwise identical, and this
will eliminate some warning messages in output.

New diagnostics to horizontal_viscosity were causing issues with stack
memory on some platforms, causing the runtime to more than double.

Two of the diagnostics were allocatables and the other two were local
variables.  By redefining the two allocatables as locals (and presumably
moving to stack), the faster performance was restored.

While the underlying cause is unclear, this is almost certainly due to
stack spill in this function, which happens to have a large number of
local arrays - including many 3d arrays used to gather diagnostics - and
any new variable is going to have volatile consequences.

This should be seen as a short term fix.  In the future, we need better
tools to detect this problem and better guidance on how to responsibly
use stack.

Also note that two variables were removed: `max_diss_rate_[qh]`.
Neither variable was used in the function.

Merge branch 'ice_dynamics' of https://github.com/OlgaSergienko/MOM6 into ice_dynamics

+Enable FMS2 interface writes

  Added additional wrappers to the MOM6 framework and MOM_couplertype_infra for
coupler_types routines and types that are needed by SIS2.  These include support
for the use of a coupler_3d_bc_type, including as overloads to the existing
coupler_type_spawn, coupler_type_copy_data, coupler_type_increment_data,
coupler_type_initialized and coupler_type_write_chksums.  There are also new
overloaded wrappers in all three files for coupler_type_redistribute_data,
coupler_type_data_override, coupler_type_rescale_data.  All answers are bitwise
identical.

  Added additional domain routine interfaces that are needed by SIS2, including
the new function same_domain, which tests whether two domains use the same
layout and conforming computational domain sizes, and a new 4d-array variant of
redistribute_array because SIS2 uses thickness categories as a 4th dimension.
All answers are bitwise identical.

Remove inappropriate timelevel arguments

- previous code had averaging instead of summation for
  SPP (x:sum,y:point,z:point) diagnostics
- corrects an issue where these diagnostics were incorrect
  by approximately a factor of 2.
- Orginially found when analyzing the depth-integrated
  temperature advection diagnostic (T_ady_2d)

MOM_hor_visc: Variables moved to stack

This patch restructures the CorAdCalc function so that the loops are
more easily vectorized on a broader range of systems. The number of
memory access has also been slightly reduced.

We observed a 1.75x speedup on a modern consumer AMD processor (Ryzen 5
2600) and a 1.24x speedup on Gaea's Intel Xeons (E5-2697 v4).
Description

There are two major changes:

- An if-block testing for `Area_q` was removed, and the `h_neglect *
  Area_q` term was replaced with a new `vol_neglect` term.

  This term is intended to prevent division by zero when the hArea_q is
  zero.  Otherwise, it is meant to be below roundoff and have no impact
  on the calculation.

  Previously, a zero value of Area_q would force a division by zero.
  Using vol_neglect ensures that the denominator is always nonzero.

  The value is set to use `H_subroundoff` times an area of 0.1 mm2,
  suggested by Robert Hallberg as a hypothetical Kolmogorov scale.
  Numerical results are intended to be independent of this choice.

- Two separated loops associated with the bounded Coriolis term were
  combined into a single loop, which reduced both the number of internal
  if-blocks and avoided redundant memory load/stores.

Other if-blocks inside of do-loops were moved outside of the loops.

I can provide two potential explanations for the difference in Intel and
AMD performance:

* Masking instructions have a lower latency on Intel CPUs, which permit
  limited vectorization of if-blocks. Similar vectorization is not
  possible on AMD CPUs. So Intel is less likely to benefit from if-block
  re-ordering.

* Our Intel nodes on Gaea have a lower RAM bandwidth, and see a smaller
  benefit from vectorization, which must required greater bandwidth.
  This speedup may be greater on a more modern Intel platform.

Although the code has been vectorized on both Intel and AMD platforms,
there are still many memory accesses per operation, which is limiting
performance.

The changes below are not expected to change any answers, and none were
detected. But since we are changing a core component, I'd suggest
reviewing this carefully.

Sample timings are provided below.

Runtime measurements
--------------------

AMD Before:

(Ocean Coriolis & mom advection)      1.091571
(Ocean Coriolis & mom advection)      1.086183
(Ocean Coriolis & mom advection)      1.091197
(Ocean Coriolis & mom advection)      1.087709
(Ocean Coriolis & mom advection)      1.086990

AMD After:

(Ocean Coriolis & mom advection)      0.619346
(Ocean Coriolis & mom advection)      0.624106
(Ocean Coriolis & mom advection)      0.625438
(Ocean Coriolis & mom advection)      0.630169
(Ocean Coriolis & mom advection)      0.621736

----

Intel Before:

(Ocean Coriolis & mom advection)      0.981367
(Ocean Coriolis & mom advection)      0.982316
(Ocean Coriolis & mom advection)      0.986633
(Ocean Coriolis & mom advection)      0.981260
(Ocean Coriolis & mom advection)      0.982810

Intel After:

(Ocean Coriolis & mom advection)      0.788747
(Ocean Coriolis & mom advection)      0.797684
(Ocean Coriolis & mom advection)      0.786874
(Ocean Coriolis & mom advection)      0.792120
(Ocean Coriolis & mom advection)      0.795373

- Fixes for SSP (x:sum;y:sum,z:point) and
  PSP (x:point,y:sum,z:point) diagnostics
- Removed unused `total_weight` arrays in these cases

  Corrected comments describing the various CT_copy_data routines, following
suggestions in a review by Keith Lindsay.  All answers are bitwise identical.

…to SIS2_infra_support

+Add infrastructure interfaces needed by SIS2

Fixed downsampling for x:sum y:point z:point diags

Coriolis: Improved coradcalc vectorization

- The perils of cherry-picking commits to try and create clean PRs: these
  units were in the first draft of the #1351 but were lost in the "clean up".

- We had an unused scalar variable that used to hold the value of slope
  within the various logic branches. Not sure why it was changed but I
  am about to add a re-use of that scalar variable so am bringing it
  back to life in this commit.

- The Viscbeck scheme needs a vertical average of S*N, the Eady growth
  rate. N can become infinite for vanished layers but since
  dz * S * N = dz * sqrt(S^2 N^2) = sqrt(S^2 N^2 dz^2) = sqrt(S^2 g' dz)
  it will be more robust to vertically sum the last expression rather
  than use the first in an integral.

- The dz, rather than h, at u- and v-points is needed in the Visbeck
  calclation of Eady growth rate. Since dz can be expensive in the
  non-Boussinesq mode, and it is already available where the slopes
  are bein calculated, it makes sense to store them, even though
  using "h" in Boussinesq mode

- Rather than take a sqrt() of dz^2 S^2 N^2 while summing dz*Sx*N, it's
  easier, and proves to be less troublesome, to pass dz*Sx*N between
  calc_isoneutral_sloeps() and Visbeck.
- Added new s/r to calculate the Eady Growth rate using the dz*Sx*N
  array rather than from S^2 and N^2 separate arrays.
- Added run-time parameter "USE_NEW_EADY_GROWTH_RATE" to use new method.

  Added support for new IO capabilities that are needed by SIS2 to use the MOM6
framework and infrastructure code, but should also be useful within MOM6
itself.  These new capabilities include writing global attributes to files,
using create_file named axes that are not derived from a MOM6 grid type, and new
options and elements in the vardesc type to support a wider array of axes and to
provide the position of the grid staggering via an integer position variable
instead of the short character strings that had been used.

  As a part of this commit, there are the new opaques type axis_info and
attribute_info, and the new routines set_axis_info, delete_axis_info,
set_attribute_info and delete_attribute_info to facilitate these new
capabilities, as well as the publicly visible function position_from_horgrid to
translate the vardesc%hor_grid character strings into the integer position flag
used elsewhere in the MOM6 and FMS codes.  Within the MOM_io_infra, there is a
new variant of the overloaded interface write_meta to handle writing global
attributes. There are also two new optional arguments to create_file and
reopen_file, and two new optional arguments to var_desc, modify_vardesc, and
query_vardesc.  All answers and output are bitwise identical.

  Restructured the code slightly so that the output files that are generated
when input_filename = 'F' is exactly the same as if it is 'n' if there are no
restart files in the restart_input_dir, or as if it is 'r' if the restart files
are there.  Previously, the solutions with 'F' worked this way, but no
ocean_geometry.nc or Vertical_grid.nc files were written when WRITE_GEOM=1,
regardless of the presence or absence of the restart files, and the
MOM_parameter_doc.all files differed slightly between the 'n' and 'F' or 'r'
cases.  As a part of these changes, the determination of whether this is a new
run is moved earlier in the algorithm, and now sits outside of
MOM_initialize_state.  All solutions are bitwise identical, but there are
changes in the position of the PARALLEL_RESTARTFILES and REFERENCE_HEIGHT
entries in most MOM_parameter_doc.all files.

Feature/update to gfdl 20210402

+Enhanced support for novel axes in MOM_io

  Change the types returned from the 5 rotated_field_chksum from integer to
integer(kind=int64), so that the full 64-byte checksums are returned.  Without
this change, the checksums that are written to MOM6 restart files or interpreted
from them are truncated to the latter half of their length.  This changes the
checksums that are written to the restart files, but both before and after this
change the values that are written are the same as those that are generated
after reading the restart with the same version of the code.  The code can run
across this change by setting RESTART_CHECKSUMS_REQUIRED = False for the run
segment where the transition occurs.  The solutions themselves are bitwise
identical.

  Changed the declarations of the vardesc and fields arrays to allocatable in
write_ocean_geometry_files, primarily to get one of the TC test cases to run
properly with the gcc compiler by shifting the memory for these arrays from
stack to heap.  The reason why this change works is not clear.  Some comments
describing these variables were also added.  All answers are bitwise identical.

…e_file_output

+More consistent treatment of input_filename = 'F'

  Added internal branches of the infra/FMS1 version of MOM_read_data that work
when there are 4-d arrays with a fifth time dimension in the file that is being
read, which was previously failing even if the variable that is being read has
fewer dimensions, due to code limitations within the fms_io read_data routines.
These new branches are selected via the new optional argument file_may_be_4d.
The infra/FMS2 versions also have the same new optional argument, although in
that case it does nothing.  All answers are bitwise identical in the cases that
worked previously, but SIS2 restart files can now be read via the MOM6
infrastructure interfaces.

(*)Write full checksums to restarts

- We found it hard to figure out the exact state of the pipeline
  because it updates to the latest of MOM6-examples at the time
  of running, and that might change between the run and when you
  examine the logs.
- This adds a few git commands to show the hash of every submodule
  we care about.

  Added the optional argument global_file to all of the overloaded MOM_read_data
routines, to indicate whether the file being read is global or is domain-
decomposed using the IO-layout.  With FMS2, this change does nothing, but with
FMS1 it is needed to correctly read from decomposed files that might include 4-d
arrays.  With this change there are versions of SIS2 that reproduce across
restarts without using the FMS1 restart routines.  All answers are bitwise
identical, but there are new optional arguments.

- @Hallberg-NOAA suspected I'd had a few too many. Give that US pint
  glasses don't hold a real pint I can assure hom that the total volume
  most suredly would not have been too much, if any fake pints had been
  involved at all, which I deny.

- Following @Hallberg-NOAA's suggestion, changed runtime parameter
  name from USE_NEW_EADY_GROWTH_RATE to USE_SIMPLER_EADY_GROWTH_RATE.

Updates read_attribute_str() so that if the optional argument found is provided
and the attribute is not defined, then the function returns with found=.false.
instead of calling MOM_error(FATAL).

I did a little bit of testing, and it looks like read_attribute_real and
read_attribute_int (not sure if the default is int32 or int64) already behave
correctly when found is present so they do not need a similar update