Latest MOM6 updates

merge in latest MOM updates
* GFDL_MOM6/dev/gfdl: (220 commits)
  Update cache dir for /lustre/f2
  Update cache dir for /lustre/f2
  Calculate height-related diagnostics in Z
  Set coord_adapt and coord_slight parameters in H
  Corrected comments in build_zlike_column
  +Pass max_depth to initialize_regridding in Z
  Corrected comments in build_sigma_column
  Combined scaling factors in build_adapt_column
  Clarified comments in Idealized_Hurricane
  +Recast MOM_diag_to_Z to work in units of Z
  +Recast MOM_ALE_sponge to work in units of Z
  +Added m_to_Z arg to horiz_interp_and_extrap_tracer
  Find energetic_PBL column height changes in Z
  +Find diapyc_energy_req column height changes in Z
  Use local variables to rescale in MOM_Point_Accel
  Recast MOM_sum_output to work in units of Z
  Simplify MOM_diagnostics code
  +Rescale depth inside of MEKE_lengthScales_0d
  +Add conversion argument to register_static_field
  Rescale values reported by PointAccel
  ...

Latest updates from MOM6

Merge in latest dev/gfdl updates

Merge in latest MOM6 dev/gfdl updates

Merge in latest commits

Merge in latest MOM dev/gfdl updates

Merge in latest MOM6 updates

Merge in latest dev/gfdl updates

Merge in latest dev/gfdl updates

merge latest updates into dev/gfdl

Merge in latest dev/gfdl commits

Merge in dev/gfdl updates

Merge in latest dev/gfdl updates

Merge in latest dev/gfdl updates

merge in latest dev/gfdl changes

merge in latest dev/gfdl updates

Merge in latest MOM6 updates

Merge in latest dev/gfdl updates

Merge in latest round of dev/gfdl updates

merge in latest dev/gfdl updates

Merge in latest dev/gfdl updates

Merge in latest dev/gfdl updates

Merge in latest dev/gfdl updates

Merge in latest updates from dev/gfdl

Merge in latest dev/gfdl updates

Merge in updates to remap_all_state_vars

Merge in latest updates

Merge in dev/gfdl updates

* TC4 integration into test suite

This patch renames the tc4 test to activate it in the test suite.  It
also modifies the Makefile to build the input field test scripts.

It also modifies the Python build scripts to be PEP8-conformant.

We temporarily disable tc4 in the restart tests, since they currently
fail.  This needs to be addressed before we can merge this into the main
branch.

The patch does not enable the necessary Python modules for running on
Travis, that will also be addressed later.

* Travis python support; tc4 Makefile

The custom TC4 Makefile has been added (oops), and the presumed Python
Ubuntu packages have been added for Travis.

* Verify ENABLE_THERMODYNAMICS is True before posting C_p diagnostic

* Make tc4 faster

* remove trailing whitespace

* add unit scaling

* fix restart fail for tc4 and some cleanup

* remove trailiny ws

* Enable tc4.restart test

* +Pass timeesteps to tracer diagnostics in [T]

  Pass timeesteps to the tracer diagnistics routines post_tracer_diagnostics and
postALE_tracer_diagnostics and to adiabatic in units of [T}.  All answers are
bitwise identical.

* +Rescaled tracer advective flux diagnostics

  Rescaled the internal units of the tracer advective flux diagnostics to units
of [conc H L2 T-1] for code simplicity and dimensional consistency testing.
Also corrected the units of some tracer fluxes as documented in comments and
commented out unused elements of the tracer_type.  All answers are bitwise
identical.

* +Pass timesteps to ALE_main in [T]

  Pass the timesteps to ALE_main, ALE_main_offline, and ALE_main_accelerated in
units of [T] for code simplicity and dimensional consistency testing.  This also
includes the rescaling of remapping-driven tracer tendencies.  All answers and
diagnostics are bitwise identical.

* +Pass timesteps to tracer column_physics in [T]

  Pass timesteps to the various tracer column_physics routines in [T] for
dimensional consistency testing.  Also added a new unit_scale_type argument to
these routines.  All answers are bitwise identical, but there are minor
interface changes to 13 subroutines.

* +Pass timesteps to applyTracerBoundaryFluxesInOut in [T]

  Pass timesteps to applyTracerBoundaryFluxesInOut in [T], and use units of
[T-1] for internal source and decay rates for the oil tracer and in fluxes of
CFCs.  Also modified extract_offline_main to return timesteps as real values
with units of [T].  Also there is a new unit_scale_type argument to
register_oil_tracer.   All answers in the MOM6_examples test cases and
regression tests are bitwise identical.

* Simplified expressions in MOM_PointAccel

  Simplified expressions inside of MOM_PointAccel, taking into account that all
velocities use the same units of [L T-1].  All answers are bitwise identical.

* Corrected dimensional epsilons in downscaling

  Added distinct negligible volumes, face areas, horizonal areas and lengths
with proper dimensional rescaling in the downsample field routines.  With these
changes, downscaled diagnostics should now pass the dimensional rescaling tests,
whereas previously there would have been a problem when the numbers used to
represent lengths are smaller than about 1e-8 times their MKS values.  All
answers are bitwise identical without dimensional rescaling.

* Simplified expressions in MOM_offline_aux

  Simplified expressions in distribute_residual_uh_barotropic.  All answers are
bitwise identical.

* Revised wave_speed to return speed in [L T-1]

  Revised wave_speed to return the internal wave speed in units of [L T-1] and
to use mono_N2_depth in units of [Z] for code simplification and expanded
dimensional consistency testing.  Also revised the internal units of some
related diagnostics in calculate_diagnostic_fields.  All answers and diagnostics
are bitwise identical.

* Rescaled internal variables in wave_speed

  Rescale internal calculations in wave_speed and wave_speeds for greater
robustness and dimensional consistency testing.  All answers are bitwise
identical and pass dimensional scaling tests.

* +Changed the units of minimum_forcing_depth to [H]

  Changed the units of minimum_forcing_depth passed to applyBoundaryFluxesInOut
and applyTracerBoundaryFluxesInOut to [H].  All answers are bitwise identical.

* Correction of documented units in comments

  Corrected some units in comments and eliminated some unused variables.
All answers are bitwise identical.

* Adiabatic clock ID bugfix

This patch fixes an initialization bug of the diabatic timer, which was
being used to measure adiabatic time but was never initialized if the
experiment was configured as adiabatic.

We fix this by introducing a separate timer for the adiabatic solver.
Although we could have reused the diabatic timer, the addition of a new
variable should not add any overhead on modern compilers.

* Corrected an OMP declaration

  Added a variable to an OMP declaration.  All answers are bitwise identical,
and a recently added compile-time error with openMP was fixed.

* Update MOM.F90

Fixed Alistair's embarrassing error.

* Dimensional rescaling in MOM_open_boundary.F90

  Added rescaling for dimensional consistency testing in MOM_open_boundary.F90,
including splitting variables with different units that had previously shared
the same variable and adding more extensive documentation of variables.  Also
changed the dimensions of the timesteps passed to radiation_open_bdry_conds and
update_segment_tracer_reservoirs to [T] and added vertical_grid_type and
unit_scale_type arguments to open_boundary_init and open_boundary_test_extern_h.
All answers are bitwise identical, although some probably bugs have been noted
in comments and there are new or altered arguments to several routines.

* (*)Fixed invariance bugs in MOM_open_boundary.F90

  Corrected dimensional consistency bugs in update_segment_tracer_reservoirs and
horizontal indexing and related bugs in gradient_at_q_points with oblique_grad
OBCs.  These will both change answers in test cases that use some open boundary
condition options, but not in any of the MOM6-examples test cases.

* TC4 integration into test suite

This patch renames the tc4 test to activate it in the test suite.  It
also modifies the Makefile to build the input field test scripts.

It also modifies the Python build scripts to be PEP8-conformant.

We temporarily disable tc4 in the restart tests, since they currently
fail.  This needs to be addressed before we can merge this into the main
branch.

The patch does not enable the necessary Python modules for running on
Travis, that will also be addressed later.

* Travis python support; tc4 Makefile

The custom TC4 Makefile has been added (oops), and the presumed Python
Ubuntu packages have been added for Travis.

* Verify ENABLE_THERMODYNAMICS is True before posting C_p diagnostic

* Make tc4 faster

* remove trailing whitespace

* add unit scaling

* fix restart fail for tc4 and some cleanup

* remove trailiny ws

* Enable tc4.restart test

* +Pass timeesteps to tracer diagnostics in [T]

  Pass timeesteps to the tracer diagnistics routines post_tracer_diagnostics and
postALE_tracer_diagnostics and to adiabatic in units of [T}.  All answers are
bitwise identical.

* +Rescaled tracer advective flux diagnostics

  Rescaled the internal units of the tracer advective flux diagnostics to units
of [conc H L2 T-1] for code simplicity and dimensional consistency testing.
Also corrected the units of some tracer fluxes as documented in comments and
commented out unused elements of the tracer_type.  All answers are bitwise
identical.

* +Pass timesteps to ALE_main in [T]

  Pass the timesteps to ALE_main, ALE_main_offline, and ALE_main_accelerated in
units of [T] for code simplicity and dimensional consistency testing.  This also
includes the rescaling of remapping-driven tracer tendencies.  All answers and
diagnostics are bitwise identical.

* +Pass timesteps to tracer column_physics in [T]

  Pass timesteps to the various tracer column_physics routines in [T] for
dimensional consistency testing.  Also added a new unit_scale_type argument to
these routines.  All answers are bitwise identical, but there are minor
interface changes to 13 subroutines.

* +Pass timesteps to applyTracerBoundaryFluxesInOut in [T]

  Pass timesteps to applyTracerBoundaryFluxesInOut in [T], and use units of
[T-1] for internal source and decay rates for the oil tracer and in fluxes of
CFCs.  Also modified extract_offline_main to return timesteps as real values
with units of [T].  Also there is a new unit_scale_type argument to
register_oil_tracer.   All answers in the MOM6_examples test cases and
regression tests are bitwise identical.

* Simplified expressions in MOM_PointAccel

  Simplified expressions inside of MOM_PointAccel, taking into account that all
velocities use the same units of [L T-1].  All answers are bitwise identical.

* Corrected dimensional epsilons in downscaling

  Added distinct negligible volumes, face areas, horizonal areas and lengths
with proper dimensional rescaling in the downsample field routines.  With these
changes, downscaled diagnostics should now pass the dimensional rescaling tests,
whereas previously there would have been a problem when the numbers used to
represent lengths are smaller than about 1e-8 times their MKS values.  All
answers are bitwise identical without dimensional rescaling.

* Simplified expressions in MOM_offline_aux

  Simplified expressions in distribute_residual_uh_barotropic.  All answers are
bitwise identical.

* Revised wave_speed to return speed in [L T-1]

  Revised wave_speed to return the internal wave speed in units of [L T-1] and
to use mono_N2_depth in units of [Z] for code simplification and expanded
dimensional consistency testing.  Also revised the internal units of some
related diagnostics in calculate_diagnostic_fields.  All answers and diagnostics
are bitwise identical.

* Rescaled internal variables in wave_speed

  Rescale internal calculations in wave_speed and wave_speeds for greater
robustness and dimensional consistency testing.  All answers are bitwise
identical and pass dimensional scaling tests.

* +Changed the units of minimum_forcing_depth to [H]

  Changed the units of minimum_forcing_depth passed to applyBoundaryFluxesInOut
and applyTracerBoundaryFluxesInOut to [H].  All answers are bitwise identical.

* Correction of documented units in comments

  Corrected some units in comments and eliminated some unused variables.
All answers are bitwise identical.

* Adiabatic clock ID bugfix

This patch fixes an initialization bug of the diabatic timer, which was
being used to measure adiabatic time but was never initialized if the
experiment was configured as adiabatic.

We fix this by introducing a separate timer for the adiabatic solver.
Although we could have reused the diabatic timer, the addition of a new
variable should not add any overhead on modern compilers.

* Corrected an OMP declaration

  Added a variable to an OMP declaration.  All answers are bitwise identical,
and a recently added compile-time error with openMP was fixed.

* Update MOM.F90

Fixed Alistair's embarrassing error.

* Dimensional rescaling in MOM_open_boundary.F90

  Added rescaling for dimensional consistency testing in MOM_open_boundary.F90,
including splitting variables with different units that had previously shared
the same variable and adding more extensive documentation of variables.  Also
changed the dimensions of the timesteps passed to radiation_open_bdry_conds and
update_segment_tracer_reservoirs to [T] and added vertical_grid_type and
unit_scale_type arguments to open_boundary_init and open_boundary_test_extern_h.
All answers are bitwise identical, although some probably bugs have been noted
in comments and there are new or altered arguments to several routines.

* (*)Fixed invariance bugs in MOM_open_boundary.F90

  Corrected dimensional consistency bugs in update_segment_tracer_reservoirs and
horizontal indexing and related bugs in gradient_at_q_points with oblique_grad
OBCs.  These will both change answers in test cases that use some open boundary
condition options, but not in any of the MOM6-examples test cases.

This reverts commit 855d706.

Revert "Merge in dev/gfdl updates"

…GFDL-dev/gfdl

Conflicts:
	src/parameterizations/lateral/MOM_MEKE.F90

…/gfdl

* (*)Fixed dimensional inconsistency in P3M_functions

  Corrected dimensionally inconsistent expressions in P3M_functions.F90,
notably in P3M_limiter and monotonize_cubic and a complete rewrite and
simplification of is_cubic_monotonic.  Also added comments documenting the
units of all real variables in this module, and changed the code to use logical
variables in place of integer "booleans", including in the return value from
is_cubic_monotonic.  These changes will change (fix) the answers when remapping
variables with small numerical values, but no answers change in the
MOM6-examples test cases.

* +Added REMAPPING_2018 runtime option

  Added a new runtime option, REMAPPING_2018, which if set to false triggers the
use of new, more accurate expressions in various parts of the ALE remapping
code.  By default, the older expressions are used, and all answers are bitwise
identical, but there are new optional arguments to various routines related to
remapping to trigger the use of new mathematically equivalent expressions.  By
default all answers are bitwise identical, but there are new and reordered
entries in the MOM6_parameter_doc files.

* Corrected the formatting of a doxygen comment

* Added conversion factors to forcing diagnostics

  Added conversion factors to 4 mass-flux diagnostics and comments to 4 others
on why no conversion factors are needed.  All answers are bitwise identical.

* Added correct scaling factors to chksum calls

  Added scale arguments to 5 chksum calls and grouped another two chksum calls
while also adding the right scaling argument. All answers are bitwise identical.

* +Unscales area before taking global sum

  Undoes the dimensional scaling of the cell areas before taking their global
sum, so that the reproducing sum does not overflow when there is dimensional
rescaling.  All answers are bitwise identical when there is no rescaling, but
this eliminates a source of inadvertent overflows or underflows in the global
sums, and there is a new optional argument to compute_global_grid_integrals.

* (*)Correct dimensionally inconsistent advective CFL

  Corrects the dimensionally inconsistent expressions for the CFL number in
the tracer advection code, in which a negligible thickness had been added to
the cell volume to avoid division by zero.  This change does not alter the
solutions in the MOM6-examples test cases, but now it permits dimensional
rescaling of lengths over a much larger range, and it could change answers if
the minimum layer thicknesses are small enough.

* Unscale sea level before averaging

  Unscale interface heights before taking a global average via a reproducing sum
in non-Boussinesq mode global diagnostics to permit dimensional consistency
testing over a larger range.  All answers are bitwise identical.

* +Added an optional tmp_scale arg to global_i_mean

  Added an optional tmp_scale argument to global_i_mean and global_j_mean to
specify an internal rescaling of variables being averaged before the reproducing
sum.  All answers are bitwise identical, but there are new optional arguments
to two public interfaces.

* Expand consistency testing with i-mean sponges

  Use tmp_scale when taking the i-mean interface heights for i-mean sponges, to
give a greatly expanded range of dimensional consistency testing.  All answers
are bitwise identical.

Merge in dev/gfdl updates

Merge in latest dev/gfdl updates

Merge in latest dev/gfdl updates

Merge in latest dev/gfdl updates

Merge in latest dev/gfdl updates

merge in latest dev/gfdl updates

merge in latest dev/gfdl updates

Merge in updates from dev/gfdl

Merge in latest dev/gfdl updates

Merge in dev/gfdl updates

Merge in latest dev/gfdl updates

Merge in dev/gfdl updates

merge in dev/gfdl updates

Merge in latest updates from dev/gfdl

Merge in latest dev/gfdl updates

merge in dev/gfdl updates

Sync with NOAA-GFDL dev/gfdl branch

* 'dev/gfdl' of github.com:NOAA-GFDL/MOM6:
  Fix to out-of-bounds error
  Slight improvement in writeMessageAndDesc
  Corrected recent bug in writeMessageAndDesc
  Included H_subroundoff in a denominator
  +Added code to handle tab lengths in documentation
  Added explicit form of cpu_clock_id
  (*)Improve ePBL when EPBL_ORIGINAL_PE_CALC=False
  Alternate fix for divide-by-zero.
  Clean up some divide-by-zero issues in OBCs.
  Added a space to a description.
  Codecov: Set base as parent (target) branch
  Fixes an integer-kind mismatch in MOM_random, seed_from_time() (#1113)

Merge in dev/gfdl updates

Sync with MOM6 dev/gfdl

Sync with NOAA-GFDL dev/gfdl

sync with dev/gfdl

Sync with NOAA-GFDL dev/gfdl

Sync with NOAA-GFDL dev/gfdl

Sync with NOAA-GFDL

Sync with NOAA-GFDL dev/gfdl

…ons of the routines with the fms-io or fms2-io interfaces

added module use statments for fms2_io and MOM_io helper routines to MOM_restart
added use_fms2=.true. arguments to save_restart and restore_state calls
added write_ic=.true. to the save_restart call in MOM.F90

added module MOM_axis with routines to define and register axes and their metadata
added module MOM_read_data_fms2.F90 with wrappers for fms2_io read_data interfaces and required routines
added module MOM_write_field_fms2.F90 with wrappers for fms2_io write_data interfaces
updated module use statments in MOM_io and MOM_restart to reference routines in MOM_read_data_fms2, MOM_write_field_fms2, and MOM_axis
made write_field and create_file interfaces in MOM_io
added create_file routines to MOM_io that accept file names or file objects to create/overwrite netcdf files that will be written to via write_field calls

fixed compile-time errors
added new MOM_read_data routines to MOM_read_data interface in MOM_io
added placeholder call for new write_field routines to MOM_io

changed use_fms2 to a required first argument in save_restart_fms2 and restore_state_fms2
changed write_ic to a required argument in write_initial_conditions

fixed the layer and interface checks in MOM_get_diagnostic_axis_data

commented out manual checksum registration in save_restart_fms2 so that internal fms2-io checksum computation is used

added checks for time units to restore_state and save_restart
added logic to make the restart time 1 to save_restart_fms2 if there is an abnormally large value passed to the routine
added interface routine file_exists_FMS2 that uses the fms2_io file_exists call

added subroutine get_num_restart_files to MOM_restart that searches for known variants of the input file names and returns then number of restart files available for querying, and the optional list of filepaths
added loop to search the files for all mandatory variables in the list of file paths returned to by cal to get_num_restart_files
added calls to get the variable dimension names and pass them as arguments to register_restart_field in restore_state_fms2

removed exit from inner variable loop in restore_state_fms2
code cleanup

moved missing_fields=0 outside of the CS loop in restore_state_fms2

moved missing_fields=0 outside of the CS loop in restore_state_fms2

converted save_restart and restore_state to interface that call versions of the routines with the fms-io or fms2-io interfaces
added module use statments for fms2_io and MOM_io helper routines to MOM_restart
added use_fms2=.true. arguments to save_restart and restore_state calls
added write_ic=.true. to the save_restart call in MOM.F90

added module MOM_axis with routines to define and register axes and their metadata
added module MOM_read_data_fms2.F90 with wrappers for fms2_io read_data interfaces and required routines
added module MOM_write_field_fms2.F90 with wrappers for fms2_io write_data interfaces
updated module use statments in MOM_io and MOM_restart to reference routines in MOM_read_data_fms2, MOM_write_field_fms2, and MOM_axis
made write_field and create_file interfaces in MOM_io
added create_file routines to MOM_io that accept file names or file objects to create/overwrite netcdf files that will be written to via write_field calls

fixed compile-time errors
added new MOM_read_data routines to MOM_read_data interface in MOM_io
added placeholder call for new write_field routines to MOM_io

changed use_fms2 to a required first argument in save_restart_fms2 and restore_state_fms2
changed write_ic to a required argument in write_initial_conditions

fixed the layer and interface checks in MOM_get_diagnostic_axis_data

commented out manual checksum registration in save_restart_fms2 so that internal fms2-io checksum computation is used

added checks for time units to restore_state and save_restart
added logic to make the restart time 1 to save_restart_fms2 if there is an abnormally large value passed to the routine
added interface routine file_exists_FMS2 that uses the fms2_io file_exists call

added subroutine get_num_restart_files to MOM_restart that searches for known variants of the input file names and returns then number of restart files available for querying, and the optional list of filepaths
added loop to search the files for all mandatory variables in the list of file paths returned to by cal to get_num_restart_files
added calls to get the variable dimension names and pass them as arguments to register_restart_field in restore_state_fms2

removed exit from inner variable loop in restore_state_fms2
code cleanup

removed test workflow directory

moved missing_fields=0 outside of the CS loop in restore_state_fms2

converted save_restart and restore_state to interface that call versions of the routines with the fms-io or fms2-io interfaces
added module use statments for fms2_io and MOM_io helper routines to MOM_restart
added use_fms2=.true. arguments to save_restart and restore_state calls
added write_ic=.true. to the save_restart call in MOM.F90

added module MOM_axis with routines to define and register axes and their metadata
added module MOM_read_data_fms2.F90 with wrappers for fms2_io read_data interfaces and required routines
added module MOM_write_field_fms2.F90 with wrappers for fms2_io write_data interfaces
updated module use statments in MOM_io and MOM_restart to reference routines in MOM_read_data_fms2, MOM_write_field_fms2, and MOM_axis
made write_field and create_file interfaces in MOM_io
added create_file routines to MOM_io that accept file names or file objects to create/overwrite netcdf files that will be written to via write_field calls

fixed compile-time errors
added new MOM_read_data routines to MOM_read_data interface in MOM_io
added placeholder call for new write_field routines to MOM_io

changed use_fms2 to a required first argument in save_restart_fms2 and restore_state_fms2
changed write_ic to a required argument in write_initial_conditions

fixed the layer and interface checks in MOM_get_diagnostic_axis_data

commented out manual checksum registration in save_restart_fms2 so that internal fms2-io checksum computation is used

added checks for time units to restore_state and save_restart
added logic to make the restart time 1 to save_restart_fms2 if there is an abnormally large value passed to the routine
added interface routine file_exists_FMS2 that uses the fms2_io file_exists call

added subroutine get_num_restart_files to MOM_restart that searches for known variants of the input file names and returns then number of restart files available for querying, and the optional list of filepaths
added loop to search the files for all mandatory variables in the list of file paths returned to by cal to get_num_restart_files
added calls to get the variable dimension names and pass them as arguments to register_restart_field in restore_state_fms2

removed exit from inner variable loop in restore_state_fms2
code cleanup

removed test workflow directory

removed white space

updated FMS tag in .testing Makefile
fixed argument comments to use doxygen style in MOM_write_field_fms2

code cleanup

removed \TODO from MOM_restart
fixed documentation for module variables in MOM_read_data_fms2 and MOM_write_field_fms2

removed doxygenized TODO statements

removed new MOM_read_data routines from interface, and commented out calls in MOM_io
fixed doxygen definitions

more doxygen fixes

changed save_restart and restore_state to wrapper routines with opitional use_fms2 and write_ic arguments
added use_fms2=.true. and write_ic=.true. to save_restart and restore_state calls

removed the error messages from append_substring
tried reordering the write_ic and use_fms2 checks in save_restart to fix error with invalid memory reference in the MOM.F90 call to save_restart-write_initial_conditions

…ing the use_fms2 and write_ic flags if present to avoid invalid memory reference error

added str_len argument to register_variable_attribute calls

added support to for rotated fields to write_initial_conditions and save_restart_fms2

removed whitespace

removed use_fms2 optional arguments in save_restart, restore_state, and create_file
made use_fms2 a MOM_restart module variable

Update MOM_state_initialization.F90

Remove space.

Update MOM_ice_shelf.F90

Add space.

Update MOM_driver.F90

Add space

Update mom_surface_forcing_mct.F90

remove whitespace

Update ocean_model_MOM.F90

Add space

Update mom_ocean_model_nuopc.F90

Add space

Update MOM_surface_forcing.F90

Add space

Changes needed work ESM4 to run with new io