PET.ESMF_LogFiles with errors produced at end of successful run

[Twize]

Name and Institution (Required)

Name: Tyler Wizenberg
Institution: University of Toronto

Confirm you have reviewed the following documentation

• Support guidelines
• Current and past Github issues

Description of your issue or question

Hi, I just completed a 1-year segment GCHP v14.1.1 simulation (split into two 6-month blocks), and at the end of each run, the model produced many PETXXX.ESMF_LogFile (where XXX is the physical core #, so 600 total) files in the run directory. Each contains the following errors:

20230317 203158.968 ERROR PET000 ESMF_Clock.F90:887 ESMF_ClockGetAlarm()
Failure - Internal subroutine call returned Error
20230318 070014.992 INFO PET000 Finalizing ESMF
20230318 070142.750 ERROR PET000 ESMF_Clock.F90:887 ESMF_ClockGetAlarm()
Failure - Internal subroutine call returned Error
20230318 174408.815 INFO PET000 Finalizing ESMF

The times on the errors look to be from the start and end of the run. However, no errors showed up in the actual run logs (slurm output or gchp.log), nor were there any errors in allPEs.log. The output files and restart were all produced successfully, and cap_restart was updated with the new start time. From a surface-level perspective, it does not look like the run was negatively impacted.

Should I be concerned or is this relatively benign?

Relevant files

Example PET.ESMF_LogFile PET000.ESMF_LogFile.zip
gchp.log: gchp.20020701_0000z.log
Slurm output: slurm_output_9070488.txt
allPEs.log: allPEs.log

Software versions

GCHP: v14.1.1
Compiler: GCC 10.3.0
OpenMPI: 4.1.1
Netcdf-Fortran: 4.6.0
ESMF: 8.3.1

[lizziel]

Hi @Twize, this is nothing to be concerned about. It seems to have to do with the monthly diagnostic setting and it is on my to do list to further investigate. If the files are bothersome you can disable them by commenting this line in file src/GCHPctm.F90 and rebuilding:
cap_options%esmf_logging_mode = ESMF_LOGKIND_MULTI_ON_ERROR

[lizziel]

I will leave this issue open as a reminder to investigate the problem further.

[Twize]

Hi @lizziel, thanks for the quick reply, and I am glad that it's nothing concerned about. They aren't too bothersome, I can just delete them after the run completes, so its not too big of an issue!

[lizziel]

I have not been able to readily reproduce this issue so am putting off introducing a fix until 14.3, or a Z-version of 14.2.

[Twize]

Hi @lizziel, I can provide my GCHP input files for the run which is giving these errors if you think it would help in reproducing the issue.

[lizziel]

Sure, thanks!

[Twize]

Hi @lizziel, sorry for the delay. Attached (as a .zip) are the main input files used for my run which produces these errors. Please let me know if any important ones are missing!

GCHP_input_files_for_Lizzie.zip

[stale]

This issue has been automatically marked as stale because it has not had recent activity. If there are no updates within 7 days it will be closed. You can add the "never stale" tag to prevent the Stale bot from closing this issue.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. If there are no updates within 7 days it will be closed. You can add the "never stale" tag to prevent the Stale bot from closing this issue.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. If there are no updates within 7 days it will be closed. You can add the "never stale" tag to prevent the Stale bot from closing this issue.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. If there are no updates within 7 days it will be closed. You can add the "never stale" tag to prevent the Stale bot from closing this issue.

[lizziel]

@atrayano is looking into if this is also an issue in GEOS.

[lizziel]

We now suppress these files by default starting in version 14.2.1 (see #330). I am closing this issue since (1) it has been passed to MAPL developers as a MAPL History bug, and (2) it does not impact GCHP runs beyond creating lots of log files if run directory config file ESMF.rc is changed to output ESMF error logs.