Add flexible timining feature in modern diag_manager #1077

uramirez8707 · 2022-11-23T18:17:39Z

Description

Fixes need flexible output average capability #956 Adds use_clock_average namelist to diag_manager_nml
If .true. the averaging is done based on the clock. For example, if doing daily averages and your start the simulation in day1_hour3, it will do the average between day1_hour3 to day2_hour 0. The default behavior will do the average between day1 hour3 to day2 hour3
Adds the capability to allow for different file frequencies for the same file.
For example, for this diag_file yaml entry

- file_name: flexible_timing%4yr%2mo%2dy%2hr
  freq: 1 1 1
  freq_units: hours hours hours
  time_units: hours
  unlimdim: time
  new_file_freq: 6 3 1
  new_file_freq_units: hours hours hours
  start_time: 2 1 1 0 0 0
  file_duration: 12 3 9
  file_duration_units: hours hours hours
  varlist:
  - module: ocn_mod
    var_name: var1
    reduction: average
    kind: r4

It will create a file every 6 hours for 12 hours
flexible_timing_0002_01_01_01.nc - using data from hour 0 to hour 6
flexible_timing_0002_01_01_07.nc - using data from hour 6 to hour 12

Then it will create a file every 3 hours for 3 hours
flexible_timing_0002_01_01_13.nc - using data from hour 12 to hour 15

Then it will create a file every 1 hour for 9 hours.
flexible_timing_0002_01_01_16.nc - using data from hour 15 to hour 16
flexible_timing_0002_01_01_17.nc - using data from hour 17 to hour 18
etc
etc

The filenames still need some more work.

Adds code to write the time averaging relating variables (time_bounds, T1, T2, DT)
Adds test to test these new features

How Has This Been Tested?
CI including added tests

Checklist:

My code follows the style guidelines of this project
I have performed a self-review of my own code
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
My changes generate no new warnings
Any dependent changes have been merged and published in downstream modules
New check tests, if applicable, are included
make distcheck passes

…the axis writes, set up time related variables in the file object

…mUpdate Conflicts: diag_manager/fms_diag_file_object.F90 diag_manager/fms_diag_object.F90 diag_manager/fms_diag_time_utils.F90

thomas-robinson · 2022-11-28T14:48:30Z

diag_manager/fms_diag_file_object.F90

+  character(len=*)      , intent(in)    :: long_name      !< The long_name of the variable
+  character(len=*)      , intent(in)    :: units          !< The units of the variable
+
+  !TODO harcodded double


Is this for reproducibility?

I hardcoded the double to make things easy for me.

I think in the old diag manager if you compile with r8 all of the time related variables were r8 and if you compile with r4 all of the time related variables were r4.

I see there is a TODO for this, so we can move on. We are supporting all types, and netcdf files can have all types in them.

thomas-robinson · 2022-11-28T14:54:55Z

diag_manager/fms_diag_file_object.F90

+  !< Register the "nv" dimension that it needed for the time_bounds
+  call register_axis(fileobj, "nv", 2)
+  !TODO hardcodded double
+  call register_field(fileobj, "nv", "double", (/"nv"/))
+  call register_variable_attribute(fileobj, "nv", "long_name", "vertex number", str_len=13)
+
+  !< Register the "time_bounds" as a variable
+  dimensions(1) = "nv"
+  dimensions(2) = trim(time_var_name)
+  bounds_name = trim(time_var_name)//"_bounds"


I wonder if we can think of something better than just repeating this

thomas-robinson

Combine the repetitive subroutines.

Add a namelist check for your new variable to make sure it's running with the modern diag

thomas-robinson · 2022-11-28T15:39:42Z

diag_manager/fms_diag_file_object.F90

-  class(fmsDiagFileContainer_type), intent(in), target   :: this            !< The file object
-  TYPE(time_type),                  intent(in)           :: time_step       !< Current model step time
+subroutine write_time_data(this)
+  class(fmsDiagFileContainer_type), intent(in), target   :: this !< The file object


This isn't the file object. This is the diag file object container. Why does this have to take the whole container and not just the diag file object?

It's only 1 container object being sent in, not the whole thing.

thomas-robinson · 2022-12-08T13:47:55Z

diag_manager/fms_diag_file_object.F90

+  character(len=*)      , intent(in)    :: long_name      !< The long_name of the variable
+  character(len=*)      , intent(in)    :: units          !< The units of the variable
+
+  !TODO harcodded double


I see there is a TODO for this, so we can move on. We are supporting all types, and netcdf files can have all types in them.

thomas-robinson · 2022-12-08T13:51:34Z

diag_manager/fms_diag_file_object.F90

  domain => null()
  diag_file => null()
 end subroutine open_diag_file

+!< @brief Writes the time average variables (*_T1, *_T2, *_DT) in the netcdf file
+subroutine write_avg_time_metadata(fileobj, variable_name, dimensions, long_name, units)


Please combine the repetitive routines for maintainability

…g use_clock_average = .true. with the old diag_manager

jiandewang · 2022-12-20T15:56:39Z

@uramirez8707 can we use this feature using diag_table ? is yes can you provide a sample ? thanks

uramirez8707 · 2022-12-21T21:41:08Z

@jiandewang unfortunately this feature is only going to be available for diag_table.yaml and the new diag_manager, which won't be production for at least a couple of months.

jiandewang · 2022-12-21T21:52:24Z

@jiandewang unfortunately this feature is only going to be available for diag_table.yaml and the new diag_manager, which won't be production for at least a couple of months.

@uramirez8707 thanks for the heads up

jiandewang · 2023-09-14T02:29:44Z

@uramirez8707 can you tell me whether diag_table.yaml and the new diag_manager is available in FMS now ? I need to use diag_yaml to control the output frequency. I checked the commit log in current FMS but can't find b0334fc commit

thomas-robinson · 2023-09-15T11:41:38Z

@uramirez8707 can you tell me whether diag_table.yaml and the new diag_manager is available in FMS now ? I need to use diag_yaml to control the output frequency. I checked the commit log in current FMS but can't find b0334fc commit

@jiandewang The updated diag manager is almost ready. We are coding the updated reduction routines which will be the last step before testing can begin. The reduction routines are needed so that the diag manager can be run through. The time frame we are looking at is October/November. When it's ready for testing, we would appreciate the help testing the flexible timing feature because we at GFDL don't have a use case and it appears you do.

jiandewang · 2023-09-15T14:31:15Z

@thomas-robinson thank you very much for the headup. Please let me know when it's ready. In EMC we are looking for this feature in our coupled system end-to-end global-worlkflow due to special requirement from DA group

jiandewang · 2024-04-30T13:55:52Z

@thomas-robinson can you tell me whether the new diag_manage and diag_table.yaml feature is in FMA main repo. ?

thomas-robinson · 2024-04-30T16:38:08Z

@jiandewang we are currently testing the tag 2024.01-beta5 which is on the dmUpdate branch. If the tests all pass (which it seems like they finally will), we will be merging the code into main and releasing it at the end of this week as 2024.01. There is likely to be a follow on patch next week or the week after.

)

jiandewang · 2024-05-15T03:25:52Z

@thomas-robinson I just started my first try but without success.
The FMS I am using is cloned from main couple of days ago:
commit 42f8506 (HEAD -> main, tag: 2024.01.01-beta1, origin/main, origin/HEAD)
Author: Alex Huth huthalexandere@gmail.com
Date: Thu May 9 13:33:56 2024 -0400

for the ocean model I tested, using the following original diag_table works fine:
GOLD_SIS
1 1 1 0 0 0
#output files
"ocean", 3, "hours", 1, "hours", "time",
"ocean_model", "geolon", "geolon", "ocean", "all", .false., "none", 2
"ocean_model", "SSH", "SSH", "ocean","all",.false.,"none",2
"ocean_model", "SST", "SST", "ocean","all",.false.,"none",2

now I switched to diag_table.yaml but failed:
title: GOLD_SIS
base_date: 1 1 1 0 0 0
diag_files:

file_name: ocean
time_units: hours
unlimdim: time
is_ocean: true
freq: 3 hours
varlist:
- module: ocean_model
  var_name: geolon
  reduction: none
  kind: r4
- module: ocean_model
  var_name: SSH
  reduction: none
  kind: r4
- module: ocean_model
  var_name: SST
  reduction: none
  kind: r4

error message I got:
diag_manager_mod::diag_manager_init: Error parsing diag_table. Error reading the base date from the diagnostic table.

what's wrong in my yaml file ? Is there a namelist/flag to tell the system that we are using yaml style instead of traditioanl style ?

uramirez8707 · 2024-05-15T14:58:28Z

I think you are missing setting the use_modern_diag namelist to .True.
https://github.com/NOAA-GFDL/FMS/blob/main/diag_manager/diag_data.F90#L391

jiandewang · 2024-05-15T15:15:24Z

I think you are missing setting the use_modern_diag namelist to .True. https://github.com/NOAA-GFDL/FMS/blob/main/diag_manager/diag_data.F90#L391

thanks @uramirez8707. Now I got "You must compile with -Duse_yaml to use the option use_modern_diag". Let me recompile it and try again

jiandewang · 2024-05-15T18:06:21Z

I add "Duse_yaml" to compile FMS and that worked fine. But when I tried to compile MOM6 I got error:
parser/yaml_output_functions.c:372: undefined reference to yaml_emitter_initialize parser/yaml_output_functions.c:373: undefined reference to yaml_emitter_set_output_file

do I need to also add "Duse_yaml" to compile MOM6 ?

uramirez8707 · 2024-05-15T18:14:39Z

Are you compiling FMS with cmake?

You shouldn't need to add that MOM6.

jiandewang · 2024-05-15T18:19:53Z

Are you compiling FMS with cmake?

You shouldn't need to add that MOM6.

no, I followed the compiling steps from MOM6-examples. See https://github.com/NOAA-GFDL/MOM6-examples/wiki/Getting-started#compiling-the-models

uramirez8707 · 2024-05-15T20:04:19Z

What system are you using?

You do need to install libyaml: https://github.com/yaml/libyaml
and include the -I include directory: https://github.com/NOAA-GFDL/mkmf/blob/master/templates/ncrc5-intel-classic.mk#L51 and the -L library directory and -llibyaml https://github.com/NOAA-GFDL/mkmf/blob/master/templates/ncrc5-intel-classic.mk#L146

jiandewang · 2024-05-15T21:38:49Z

I am using intel. Let me try to follow your information. Thanks @uramirez8707

uramirez8707 added 12 commits November 8, 2022 13:26

Add the io for the time_axis, properly close the file, bug fixes for …

70596f0

…the axis writes, set up time related variables in the file object

removes get_date_diff from diag_utils and cleans up get_global_io_domain

3555257

replaces error_mesg with mpp_error, and fixes error messages

f1ed86a

add more time metadata

7101ae9

Add the use_clock_average namelist flag for flexible timings

38233e4

Fix bug with the new_file_freq implementation

8b18aab

implement flexible output timings

50ef3ed

Bug fixes for the flexible timings

1dda963

fix line count limit

5c125eb

Add missing documentation

5549252

Allow frequency to be an array

e88f01b

Merge branch 'flexible_timinig' of github.com:uramirez8707/FMS into d…

5c87aef

…mUpdate Conflicts: diag_manager/fms_diag_file_object.F90 diag_manager/fms_diag_object.F90 diag_manager/fms_diag_time_utils.F90

uramirez8707 changed the title ~~Flexible timinig~~ Add flexible timining feature in modern diag_manager Nov 23, 2022

believe in magic you muggle

125de03

thomas-robinson reviewed Nov 28, 2022

View reviewed changes

uramirez8707 marked this pull request as ready for review December 2, 2022 19:35

Fix to stop writing data after file_duration

e2bfe66

thomas-robinson requested changes Dec 8, 2022

View reviewed changes

uramirez8707 added 2 commits December 9, 2022 16:33

Cleans up the writing of the time metadata and adds a failure if usin…

2d24bcf

…g use_clock_average = .true. with the old diag_manager

add uniform filenames

5c772a6

thomas-robinson approved these changes Dec 16, 2022

View reviewed changes

thomas-robinson merged commit b0334fc into NOAA-GFDL:dmUpdate Dec 16, 2022

This was referenced Dec 16, 2022

need more flexible capability in saving averaged outputs mom-ocean/MOM6#1564

Open

Need flexible output times for ocean model ufs-community/ufs-weather-model#960

Open

rem1776 pushed a commit to rem1776/FMS that referenced this pull request May 1, 2024

feat: Add flexible timing feature in modern diag_manager (NOAA-GFDL#1077

03198b3

)

rem1776 pushed a commit to rem1776/FMS that referenced this pull request May 1, 2024

feat: Add flexible timing feature in modern diag_manager (NOAA-GFDL#1077

e1b5063

)

rem1776 pushed a commit to rem1776/FMS that referenced this pull request May 1, 2024

feat: Add flexible timing feature in modern diag_manager (NOAA-GFDL#1077

e7fd8f0

)

rem1776 pushed a commit to rem1776/FMS that referenced this pull request May 1, 2024

feat: Add flexible timing feature in modern diag_manager (NOAA-GFDL#1077

0a0faba

)

rem1776 pushed a commit to rem1776/FMS that referenced this pull request May 1, 2024

feat: Add flexible timing feature in modern diag_manager (NOAA-GFDL#1077

169f094

)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add flexible timining feature in modern diag_manager #1077

Add flexible timining feature in modern diag_manager #1077

uramirez8707 commented Nov 23, 2022 •

edited

Loading

thomas-robinson Nov 28, 2022

uramirez8707 Nov 28, 2022

thomas-robinson Dec 8, 2022

thomas-robinson Nov 28, 2022

thomas-robinson left a comment

thomas-robinson Nov 28, 2022

thomas-robinson Dec 8, 2022

thomas-robinson Dec 8, 2022

thomas-robinson Dec 8, 2022

jiandewang commented Dec 20, 2022

uramirez8707 commented Dec 21, 2022

jiandewang commented Dec 21, 2022

jiandewang commented Sep 14, 2023 •

edited

Loading

thomas-robinson commented Sep 15, 2023

jiandewang commented Sep 15, 2023

jiandewang commented Apr 30, 2024

thomas-robinson commented Apr 30, 2024

jiandewang commented May 15, 2024

uramirez8707 commented May 15, 2024

jiandewang commented May 15, 2024

jiandewang commented May 15, 2024 •

edited

Loading

uramirez8707 commented May 15, 2024

jiandewang commented May 15, 2024

uramirez8707 commented May 15, 2024

jiandewang commented May 15, 2024

Add flexible timining feature in modern diag_manager #1077

Add flexible timining feature in modern diag_manager #1077

Conversation

uramirez8707 commented Nov 23, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

thomas-robinson left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jiandewang commented Dec 20, 2022

uramirez8707 commented Dec 21, 2022

jiandewang commented Dec 21, 2022

jiandewang commented Sep 14, 2023 • edited Loading

thomas-robinson commented Sep 15, 2023

jiandewang commented Sep 15, 2023

jiandewang commented Apr 30, 2024

thomas-robinson commented Apr 30, 2024

jiandewang commented May 15, 2024

uramirez8707 commented May 15, 2024

jiandewang commented May 15, 2024

jiandewang commented May 15, 2024 • edited Loading

uramirez8707 commented May 15, 2024

jiandewang commented May 15, 2024

uramirez8707 commented May 15, 2024

jiandewang commented May 15, 2024

uramirez8707 commented Nov 23, 2022 •

edited

Loading

jiandewang commented Sep 14, 2023 •

edited

Loading

jiandewang commented May 15, 2024 •

edited

Loading