Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add simulation with all diagnostics turned to integration testing #2399

Merged
merged 20 commits into from
Aug 1, 2024

Conversation

msulprizio
Copy link
Contributor

@msulprizio msulprizio commented Jul 24, 2024

Name and Institution (Required)

Name: Melissa Sulprizio
Institution: Harvard

Describe the update

In order to evaluate all diagnostic collections, we now include a simulation with all collections activated in HISTORY.rc. This is currently a copy of the gc_4x5_merra2_fullchem_benchmark simulation with the comment character removed from all collection names at the top of HISTORY.rc with the name gc_4x5_merra2_fullchem_alldiags.

Still to do:

  • Add all diagnostics simulation to GCHP integration tests
  • Turn on ObsPack and Planeflight in geoschem_config.yml for GCClassic test
  • Fix display of superproject and submodule commit hashes in log files and to stdout

Expected changes

This is a zero diff update with respect to benchmark simulations.

Related Github Issue

…RY.rc to GCClassic integration tests

In order to evaluate all diagnostic collections, we now include a simulation
with all collections activated in HISTORY.rc. This is a copy of the
gc_4x5_merra2_fullchem_benchmark simulation with the comment character
removed from all collection names at the top of HISTORY.rc.

Signed-off-by: Melissa Sulprizio <mpayer@seas.harvard.edu>
@msulprizio msulprizio linked an issue Jul 24, 2024 that may be closed by this pull request
1 task
@msulprizio
Copy link
Contributor Author

The all diagnostics run is currently failing integration tests:

==============================================================================
GEOS-Chem Classic: Execution Test Results

GCClassic #c49fcec Submod updates: Merge GEOS-Chem PR #2353 and Cloud-J PR #19
GEOS-Chem #2c8910869 Add ModelE2.1 (GCAP 2.0) simulation to GCClassic integration tests
HEMCO     #2192e0e HEMCO 3.9.1 release

Using 24 OpenMP threads
Number of execution tests: 30

Submitted as SLURM job: 41196464
==============================================================================
 
Execution tests:
------------------------------------------------------------------------------
gc_05x0625_NA_47L_merra2_CH4........................Execute Simulation....PASS
gc_05x0625_NA_47L_merra2_fullchem...................Execute Simulation....PASS
gc_2x25_ModelE2.1_fullchem..........................Execute Simulation....PASS
gc_4x5_47L_merra2_fullchem..........................Execute Simulation....PASS
gc_4x5_47L_merra2_fullchem_TOMAS15..................Execute Simulation....PASS
gc_4x5_merra2_aerosol...............................Execute Simulation....PASS
gc_4x5_merra2_carbon................................Execute Simulation....PASS
gc_4x5_merra2_carbon_CH4............................Execute Simulation....PASS
gc_4x5_merra2_carbon_CO.............................Execute Simulation....PASS
gc_4x5_merra2_carbon_CO2............................Execute Simulation....PASS
gc_4x5_merra2_carbon_OCS............................Execute Simulation....PASS
gc_4x5_merra2_CH4...................................Execute Simulation....PASS
gc_4x5_merra2_CO2...................................Execute Simulation....PASS
gc_4x5_merra2_fullchem..............................Execute Simulation....PASS
gc_4x5_merra2_fullchem_aciduptake...................Execute Simulation....PASS
gc_4x5_merra2_fullchem_alldiags.....................Execute Simulation....FAIL
...

The error in the log file is:

===============================================================================
GEOS-Chem ERROR: The value of attribute "UVFlux.template:" must end with a 
single quote character, followed by a comma. Please check the HISTORY.rc file.
 -> at Init_DiagList (Headers/diaglist_mod.F90)

 -> ERROR occurred at (or near) line    397 of the HISTORY.rc file
===============================================================================

===============================================================================
GEOS-CHEM ERROR: Error encountered in "Init_DiagList"!
STOP at  -> at GEOS-Chem (in GeosCore/main.F90)
===============================================================================
 Timer forced to stop due to error: GEOS-Chem                     
 Timer forced to stop due to error: Diagnostics                   

We will investigate and add fixes to this branch.

@msulprizio msulprizio added topic: Benchmarking and Testing Related to CI, integration tests, or scientific benchmarking no-diff-to-benchmark This update will not change the results of fullchem benchmark simulations labels Jul 24, 2024
yantosca added 11 commits July 26, 2024 10:19
run/GCClassic/HISTORY.rc.fullchem
- Added missing fields for the UVFlux collection (frequency,
  duration, mode, fields).

CHANGELOG.md
- Updated accordingly

Signed-off-by: Bob Yantosca <yantosca@seas.harvard.edu>
test/shared/commonFunctionsForTests.sh
- Update SED_HISTORY_RC_1 and SED_HISTORY_RC_N to use regex wildcard
  characters for certain digits
- Update logic so that we first test on the resolution name before
  changing frequency & duration values to 20 mins (for nested-grid
  tests) or 1-hr (for other tests)

run/GCClassic/HISTORY.rc.templates/HISTORY.rc.fullchem
- Update frequency and duration of the TOMAS collection from "010000"
  to "00000100 000000", to be consistent with other collections

Signed-off-by: Bob Yantosca <yantosca@seas.harvard.edu>
test/shared/commonFunctionsForTests.sh
- Add "print_submodule_head_commits" function to print the commit
  hash & description at the head of each submodule

test/integration/GCClassic/integrationTestCreate.sh
test/integration/GCClassic/integrationTestCompile.sh
test/integration/GCClassic/integrationTestExecute.sh
test/integration/GCHP/integrationTestCreate.sh
test/integration/GCHP/integrationTestCompile.sh
test/integration/GCHP/integrationTestExecute.sh
- Now call print_submodule_head_commits instead of using inlined code

Signed-off-by: Bob Yantosca <yantosca@seas.harvard.edu>
test/shared/commonFunctionsForTests.sh
- Remove "_GridComp" from the "FVdycoreCubed" submodule display name
  in function print_submodule_head_commits

test/integration/GCHP/integrationTestCompile.sh
test/integration/GCHP/integrationTestCreate.sh
test/integration/GCHP/integrationTestExecute.sh
- Now use an offset of 14 characters for print_submodule_head_commits

Signed-off-by: Bob Yantosca <yantosca@seas.harvard.edu>
test/integration/GCHP/integrationTestCreate.sh
- Removed "exit 1" statement that was leftover from debugging.

Signed-off-by: Bob Yantosca <yantosca@seas.harvard.edu>
test/integration/GCHP/integrationTestCreate.sh
- Create a gchp_merra2_fullchem_alldiags by copying the
  gchp_merra2_fullchem_benchmark run directory and:
- Disable RRTMG, Tomas, and DynHeating collections, as these are
  not enabled for fullchem_benchmark
- Print the path of the gchp_merra2_fullchem_alldiags directory

test/integration/GCClassic/integrationTestCreate.sh
- Print the path of the gc_4x5_merra2_fullchem_alldiags directory
- Also disable the DynHeating collection (if present)

Signed-off-by: Bob Yantosca <yantosca@seas.harvard.edu>
GeosCore/planeflight_mod.F90
- Replace calls to GEOS_CHEM_STOP (which stops the run upon error)
  with calls to GC_Error + RETURN (to pass the error back to the main
  program).

run/GCClassic/geoschem_config.yml.templates/geoschem_config.yml.*
- Send planeflight diagnostic "plane.log" output to OutputDir
- Now use dummy input file for ObsPack -- facilitates testing

test/integration/GCClassic/integrationTestCreate.sh
- Define bash variables to hold paths to configuration files
  (and unset them at the end of the script)
- Navigate into the alldiags run directory to simplify bash commands
- Copy the dummy planeflight and obspack input files to the run
  directory for the "alldiags" integration test
- Call "toggle_geoschem_config_option" to turn on the planeflight
  and obspack diagnostics

test/shared/alldiags/Planeflight.dat.20190701
- Dummy input file for the planeflight diagnostic used in the
  "alldiags" integration tests

test/shared/commonFunctionsForTests.sh
- Add comments to "print_submodule_head_commits" function
- Add new function "toggle_geoschem_config_option" to toggle the
  "activate" value in the planeflight and obspack diagnostic menus
  in geoschem_config.yml

CHANGELOG.md
- Updated accordingly

Signed-off-by: Bob Yantosca <yantosca@seas.harvard.edu>
run/GCClassic/geoschem_config.yml.templates/geoschem_config.yml.*
- Change "obspack_dummy_input_for_testing.20190701.nc"
  to "obspack_input_for_testing.20190701.nc" in all GEOS-Chem
  Classic geoschem_config.yml templates.  This will avoid an issue
  where "mm" is confused with minutes.  Fix this issue later.

Signed-off-by: Bob Yantosca <yantosca@seas.harvard.edu>
test/shared/commonFunctionsForTests.sh
- Add the current directory (.) to the list of submodules for which
  we will obtain Git commit hashes
- In the display name (y), replace "." with the basename of
  argument ${2}.  This is the name of the superproject.

Signed-off-by: Bob Yantosca <yantosca@seas.harvard.edu>
test/integration/GCClassic/integrationTestCreate.sh
- Bug fix: We forgot to adjust the name of the sample ObsPack data file
  stored in the $obsPk variable.  This has now been corrected.

Signed-off-by: Bob Yantosca <yantosca@seas.harvard.edu>
test/integration/GCClassic/integrationTestCreate.sh
- In the grep command used to define the extDataDir variable, we must
  use ${HOME} instead of ~.  Otherwise the grep fails.  This was
  causing the sample ObsPack data file not to be copied to the alldiags
  test folder.

Signed-off-by: Bob Yantosca <yantosca@seas.harvard.edu>
@yantosca
Copy link
Contributor

All GEOS-Chem Classic integration tests passed:

==============================================================================
GEOS-Chem Classic: Execution Test Results

CodeDir   : 67b58e6 GEOS-Chem submod update: Merge PR #2394 (Add GCAP int test)
GEOS-Chem : 03a25c37d Bug fix: Use ${HOME} instead of ~ in grep command
HEMCO     : e68ebae HEMCO 3.9.2 release
Cloud-J   : 37a4ece Update release version number and date
HETP      : 2a99b24 Merge pull request #2 from geoschem/bugfix/initialize_local_variables

Using 24 OpenMP threads
Number of execution tests: 29

Submitted as SLURM job: 41764773
==============================================================================
 
Execution tests:
------------------------------------------------------------------------------
...
gc_4x5_merra2_fullchem_alldiags.....................Execute Simulation....PASS
...
 
Summary of test results:
------------------------------------------------------------------------------
Execution tests passed: 29
Execution tests failed: 0
Execution tests not yet completed: 0

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%  All execution tests passed!  %%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

The alldiags test created these files:

GEOSChem.AdvFluxVert.20190701_0000z.nc4         GEOSChem.ProdLoss.20190701_0000z.nc4
GEOSChem.AerosolMass.20190701_0000z.nc4         GEOSChem.RxnConst.20190701_0000z.nc4
GEOSChem.Aerosols.20190701_0000z.nc4            GEOSChem.RxnRates.20190701_0000z.nc4
GEOSChem.BoundaryConditions.20190701_0000z.nc4  GEOSChem.SatDiagn.20190701_0000z.nc4
GEOSChem.BoundaryConditions.20190701_0100z.nc4  GEOSChem.SatDiagnEdge.20190701_0000z.nc4
GEOSChem.Budget.20190701_0000z.nc4              GEOSChem.SpeciesConc.20190701_0000z.nc4
GEOSChem.CloudConvFlux.20190701_0000z.nc4       geoschem_species_metadata.yml
GEOSChem.ConcAboveSfc.20190701_0100z.nc4        GEOSChem.StateChm.20190701_0000z.nc4
GEOSChem.ConcAfterChem.20190701_0000z.nc4       GEOSChem.StateMet.20190701_0000z.nc4
GEOSChem.DryDep.20190701_0000z.nc4              GEOSChem.StratBM.20190701_0000z.nc4
GEOSChem.JValues.20190701_0000z.nc4             GEOSChem.UVFlux.20190701_0000z.nc4
GEOSChem.KppARDiags.20190701_0000z.nc4          GEOSChem.WetLossConv.20190701_0000z.nc4
GEOSChem.KppDiags.20190701_0000z.nc4            GEOSChem.WetLossLS.20190701_0000z.nc4
GEOSChem.LevelEdgeDiags.20190701_0000z.nc4      HEMCO_diagnostics.201907010000.nc
GEOSChem.Metrics.20190701_0000z.nc4             plane.log.20190701

The ObsPack diagnostic output is:

netcdf GEOSChem.ObsPack.20190701_0000z {
dimensions:
	obs = UNLIMITED ; // (48 currently)
	species = 3 ;
	char_len_obs = 200 ;
variables:
	char obspack_id(obs, char_len_obs) ;
		obspack_id:long_name = "obspack_id" ;
		obspack_id:units = "1" ;
		obspack_id:_Storage = "chunked" ;
		obspack_id:_ChunkSizes = 1, 200 ;
		obspack_id:_NoFill = "true" ;
	int nsamples(obs) ;
		nsamples:long_name = "no. of model samples" ;
		nsamples:units = "1" ;
		nsamples:comment = "Number of discrete model samples in average" ;
		nsamples:_Storage = "chunked" ;
		nsamples:_ChunkSizes = 1024 ;
		nsamples:_Endianness = "little" ;
		nsamples:_NoFill = "true" ;
	int averaging_interval(obs) ;
		averaging_interval:long_name = "Amount of model time over which this observation is averaged" ;
		averaging_interval:units = "seconds" ;
		averaging_interval:_Storage = "chunked" ;
		averaging_interval:_ChunkSizes = 1024 ;
		averaging_interval:_Endianness = "little" ;
		averaging_interval:_NoFill = "true" ;
	int averaging_interval_start(obs) ;
		averaging_interval_start:long_name = "Start of averaging interval" ;
		averaging_interval_start:units = "seconds since 1970-01-01 00:00:00 UTC" ;
		averaging_interval_start:calendar = "standard" ;
		averaging_interval_start:_Storage = "chunked" ;
		averaging_interval_start:_ChunkSizes = 1024 ;
		averaging_interval_start:_Endianness = "little" ;
		averaging_interval_start:_NoFill = "true" ;
	int averaging_interval_end(obs) ;
		averaging_interval_end:long_name = "End of averaging interval" ;
		averaging_interval_end:units = "seconds since 1970-01-01 00:00:00 UTC" ;
		averaging_interval_end:calendar = "standard" ;
		averaging_interval_end:_Storage = "chunked" ;
		averaging_interval_end:_ChunkSizes = 1024 ;
		averaging_interval_end:_Endianness = "little" ;
		averaging_interval_end:_NoFill = "true" ;
	float lon(obs) ;
		lon:long_name = "longitude" ;
		lon:units = "degrees_east" ;
		lon:_Storage = "chunked" ;
		lon:_ChunkSizes = 1024 ;
		lon:_Endianness = "little" ;
		lon:_NoFill = "true" ;
	float lat(obs) ;
		lat:long_name = "latitude" ;
		lat:units = "degrees_north" ;
		lat:_Storage = "chunked" ;
		lat:_ChunkSizes = 1024 ;
		lat:_Endianness = "little" ;
		lat:_NoFill = "true" ;
	float u(obs) ;
		u:long_name = "Zonal component of wind" ;
		u:units = "m s^-1" ;
		u:_Storage = "chunked" ;
		u:_ChunkSizes = 1024 ;
		u:_Endianness = "little" ;
		u:_NoFill = "true" ;
	float v(obs) ;
		v:long_name = "Meridional component of wind" ;
		v:units = "m s^-1" ;
		v:_Storage = "chunked" ;
		v:_ChunkSizes = 1024 ;
		v:_Endianness = "little" ;
		v:_NoFill = "true" ;
	float blh(obs) ;
		blh:long_name = "Boundary layer height" ;
		blh:units = "m" ;
		blh:_Storage = "chunked" ;
		blh:_ChunkSizes = 1024 ;
		blh:_Endianness = "little" ;
		blh:_NoFill = "true" ;
	float q(obs) ;
		q:long_name = "mass_fraction_of_water_inair" ;
		q:units = "g water (kg air)^-1" ;
		q:_Storage = "chunked" ;
		q:_ChunkSizes = 1024 ;
		q:_Endianness = "little" ;
		q:_NoFill = "true" ;
	float pressure(obs) ;
		pressure:long_name = "pressure" ;
		pressure:units = "hPa" ;
		pressure:_Storage = "chunked" ;
		pressure:_ChunkSizes = 1024 ;
		pressure:_Endianness = "little" ;
		pressure:_NoFill = "true" ;
	float temperature(obs) ;
		temperature:long_name = "temperature" ;
		temperature:units = "K" ;
		temperature:_Storage = "chunked" ;
		temperature:_ChunkSizes = 1024 ;
		temperature:_Endianness = "little" ;
		temperature:_NoFill = "true" ;
	float CO(obs) ;
		CO:long_name = "Carbon monoxide" ;
		CO:units = "mol mol-1" ;
		CO:_FillValue = -1.e+34f ;
		CO:_Storage = "chunked" ;
		CO:_ChunkSizes = 1024 ;
		CO:_Endianness = "little" ;
		CO:_NoFill = "true" ;
	float NO(obs) ;
		NO:long_name = "Nitrogen oxide" ;
		NO:units = "mol mol-1" ;
		NO:_FillValue = -1.e+34f ;
		NO:_Storage = "chunked" ;
		NO:_ChunkSizes = 1024 ;
		NO:_Endianness = "little" ;
		NO:_NoFill = "true" ;
	float O3(obs) ;
		O3:long_name = "Ozone" ;
		O3:units = "mol mol-1" ;
		O3:_FillValue = -1.e+34f ;
		O3:_Storage = "chunked" ;
		O3:_ChunkSizes = 1024 ;
		O3:_Endianness = "little" ;
		O3:_NoFill = "true" ;

// global attributes:
		:history = "GEOS-Chem simulation at 2024/07/29 18:41" ;
		:conventions = "CF-1.4" ;
		:references = "www.geos-chem.org; wiki.geos-chem.org" ;
		:model_start_date = "2019/07/01 00:00:00 UTC" ;
		:model_end_date = "2019/07/01 01:00:00 UTC" ;
		:_NCProperties = "version=2,netcdf=4.8.0,hdf5=1.10.7" ;
		:_SuperblockVersion = 2 ;
		:_IsNetcdf4 = 0 ;
		:_Format = "netCDF-4 classic model" ;
data:
}

and the Planeflight output is:

POINT    TYPE YYYYMMDD HHMM     LAT     LON   PRESS        OBS     T-IND P-I I-IND J-IND  TRA_001     TRA_002     O3          OH          RO2         AN          NOy         GMAO_TEMP   AODC_SULF   AODB_SULF   TIME_LT     AQAER_RAD   AQAER_SURF  JVL_001   
    1    DC-8 20190701 0005   34.39  -95.00  757.33   9999.000 000000001 015 00018 00032  1.130E-009  1.711E-010  1.239E+012  8.618E+005  2.172E-011  1.117E-011  8.873E-010  2.855E+002  2.976E-002  2.788E-002  1.772E+001  1.162E-005  7.701E-008  3.905E-027
    2    DC-8 20190701 0007   34.53  -95.08  668.89   9999.000 000000001 019 00018 00032  9.849E-010  8.714E-011  1.150E+012  1.015E+006  2.103E-011  8.928E-012  9.583E-010  2.789E+002  2.976E-002  2.839E-002  1.774E+001  8.989E-006  2.940E-008  1.609E-025
    3    DC-8 20190701 0009   34.70  -95.16  574.40   9999.000 000000001 021 00018 00032  9.753E-010  6.904E-011  1.101E+012  1.038E+006  2.020E-011  8.802E-012  1.039E-009  2.731E+002  2.976E-002  2.852E-002  1.776E+001  8.067E-006  1.770E-008  2.169E-024
    4    DC-8 20190701 0010   34.86  -95.26  484.89   9999.000 000000001 024 00018 00032  9.726E-010  5.282E-011  1.032E+012  1.302E+006  1.744E-011  8.728E-012  1.216E-009  2.626E+002  2.976E-002  2.878E-002  1.777E+001  7.396E-006  4.076E-008  1.207E-022
    5    DC-8 20190701 0012   35.03  -95.35  433.15   9999.000 000000003 025 00018 00032  9.695E-010  4.951E-011  9.837E+011  9.055E+005  1.462E-011  8.671E-012  1.255E-009  2.582E+002  2.902E-002  2.821E-002  1.779E+001  7.279E-006  5.059E-008  3.449E-022
    6    DC-8 20190701 0014   35.21  -95.46  428.34   9999.000 000000003 025 00018 00032  9.695E-010  4.951E-011  9.837E+011  9.055E+005  1.462E-011  8.671E-012  1.255E-009  2.582E+002  2.902E-002  2.821E-002  1.781E+001  7.279E-006  5.059E-008  3.449E-022
    7    DC-8 20190701 0016   35.39  -95.56  428.38   9999.000 000000003 025 00018 00032  9.695E-010  4.951E-011  9.837E+011  9.055E+005  1.462E-011  8.671E-012  1.255E-009  2.582E+002  2.902E-002  2.821E-002  1.783E+001  7.279E-006  5.059E-008  3.449E-022
    8    DC-8 20190701 0017   35.57  -95.67  428.47   9999.000 000000003 025 00018 00032  9.695E-010  4.951E-011  9.837E+011  9.055E+005  1.462E-011  8.671E-012  1.255E-009  2.582E+002  2.902E-002  2.821E-002  1.784E+001  7.279E-006  5.059E-008  3.449E-022
    9    DC-8 20190701 0019   35.75  -95.78  428.47   9999.000 000000003 025 00018 00032  9.695E-010  4.951E-011  9.837E+011  9.055E+005  1.462E-011  8.671E-012  1.255E-009  2.582E+002  2.902E-002  2.821E-002  1.786E+001  7.279E-006  5.059E-008  3.449E-022
   10    DC-8 20190701 0024   36.34  -95.95  428.52   9999.000 000000003 025 00018 00033  6.711E-010  1.786E-011  1.125E+012  1.258E+006  6.137E-012  3.256E-012  2.211E-009  2.578E+002  1.675E-002  1.319E-002  1.791E+001  0.000E+000  0.000E+000  5.270E-022
   11    DC-8 20190701 0026   36.53  -95.99  428.31   9999.000 000000003 025 00018 00033  6.711E-010  1.786E-011  1.125E+012  1.258E+006  6.137E-012  3.256E-012  2.211E-009  2.578E+002  1.675E-002  1.319E-002  1.793E+001  0.000E+000  0.000E+000  5.270E-022
   12    DC-8 20190701 0028   36.73  -96.01  428.36   9999.000 000000003 025 00018 00033  6.711E-010  1.786E-011  1.125E+012  1.258E+006  6.137E-012  3.256E-012  2.211E-009  2.578E+002  1.675E-002  1.319E-002  1.795E+001  0.000E+000  0.000E+000  5.270E-022
   13    DC-8 20190701 0030   36.93  -96.00  428.75   9999.000 000000003 025 00018 00033  6.711E-010  1.786E-011  1.125E+012  1.258E+006  6.137E-012  3.256E-012  2.211E-009  2.578E+002  1.675E-002  1.319E-002  1.797E+001  0.000E+000  0.000E+000  5.270E-022
   14    DC-8 20190701 0031   37.09  -96.14  428.42   9999.000 000000005 025 00018 00033  6.701E-010  1.793E-011  1.124E+012  7.785E+005  5.736E-012  3.251E-012  2.218E-009  2.579E+002  1.668E-002  1.313E-002  1.798E+001  0.000E+000  0.000E+000  3.766E-022
   15    DC-8 20190701 0033   37.25  -96.28  428.45   9999.000 000000005 025 00018 00033  6.701E-010  1.793E-011  1.124E+012  7.785E+005  5.736E-012  3.251E-012  2.218E-009  2.579E+002  1.668E-002  1.313E-002  1.800E+001  0.000E+000  0.000E+000  3.766E-022
   16    DC-8 20190701 0035   37.41  -96.41  428.55   9999.000 000000005 025 00018 00033  6.701E-010  1.793E-011  1.124E+012  7.785E+005  5.736E-012  3.251E-012  2.218E-009  2.579E+002  1.668E-002  1.313E-002  1.802E+001  0.000E+000  0.000E+000  3.766E-022
   17    DC-8 20190701 0037   37.57  -96.55  428.55   9999.000 000000005 025 00018 00033  6.701E-010  1.793E-011  1.124E+012  7.785E+005  5.736E-012  3.251E-012  2.218E-009  2.579E+002  1.668E-002  1.313E-002  1.804E+001  0.000E+000  0.000E+000  3.766E-022
   18    DC-8 20190701 0038   37.72  -96.69  428.50   9999.000 000000005 025 00018 00033  6.701E-010  1.793E-011  1.124E+012  7.785E+005  5.736E-012  3.251E-012  2.218E-009  2.579E+002  1.668E-002  1.313E-002  1.805E+001  0.000E+000  0.000E+000  3.766E-022
   19    DC-8 20190701 0040   37.88  -96.83  433.37   9999.000 000000005 025 00018 00033  6.701E-010  1.793E-011  1.124E+012  7.785E+005  5.736E-012  3.251E-012  2.218E-009  2.579E+002  1.668E-002  1.313E-002  1.807E+001  0.000E+000  0.000E+000  3.766E-022
   20    DC-8 20190701 0042   38.02  -96.96  508.21   9999.000 000000005 023 00018 00033  6.319E-010  1.467E-011  1.273E+012  6.847E+005  6.338E-012  2.529E-012  2.246E-009  2.662E+002  1.668E-002  1.281E-002  1.809E+001  0.000E+000  0.000E+000  2.577E-023
   21    DC-8 20190701 0044   38.16  -97.08  547.75   9999.000 000000005 022 00018 00033  6.072E-010  1.436E-011  1.330E+012  6.580E+005  7.058E-012  2.275E-012  2.055E-009  2.699E+002  1.668E-002  1.267E-002  1.811E+001  0.000E+000  0.000E+000  6.806E-024
   22    DC-8 20190701 0045   38.29  -97.21  592.62   9999.000 000000005 020 00018 00033  5.642E-010  1.855E-011  1.366E+012  6.438E+005  8.849E-012  2.039E-012  1.892E-009  2.769E+002  1.668E-002  1.228E-002  1.812E+001  7.399E-006  3.245E-008  4.927E-025
   23    DC-8 20190701 0047   38.38  -97.36  678.79   9999.000 000000005 018 00018 00033  5.716E-010  2.860E-011  1.389E+012  6.032E+005  9.921E-012  1.989E-012  1.705E-009  2.821E+002  1.668E-002  1.210E-002  1.814E+001  7.897E-006  3.401E-008  4.343E-026
   24    DC-8 20190701 0049   38.46  -97.50  754.43   9999.000 000000005 015 00018 00033  6.411E-010  5.666E-011  1.450E+012  5.037E+005  1.155E-011  3.501E-012  1.595E-009  2.874E+002  1.668E-002  1.189E-002  1.816E+001  8.736E-006  4.730E-008  3.184E-027

All GCClassic integration tests were zero-diff w/r/t PR #2394, except:

  • TOMAS (parallelization error?)
  • APM (parallelization error?)
  • ModelE2.1 (not present in this PR, as it is based off of an earlier commit)

@yantosca
Copy link
Contributor

The GCHP "alldiags" integration test fails with this error:

GEOS-Chem ERROR [0006]: The mapData%slot2Id array corresponding to collection "BudgetWetDepFull" contains missing values! This can indicate that this collection is either undefined or turned off.  Please check the HISTORY.rc configuration file in your run directory.
 --> LOCATION:  -> at Get_Mapping (in module Headers/state_diag_mod.F90)
 
GEOS-Chem ERROR [0006]: Error encountered in "Get_Mapping": BudgetWetDepFull
 --> LOCATION:  -> at Get_MapData_and_NumSlots (in module Headers/state_diag_mod.F90)
 
GEOS-Chem ERROR [0006]: Error encountered in "Get_MapData_and_NumSlots"!
 --> LOCATION:  -> at Init_and_Register_R8_3D (in module Headers/state_diag_mod.F90)
 
GEOS-Chem ERROR [0006]: Error encountered in "Init_and_Register", diagID =BudgetWetDepFull
 --> LOCATION:  -> at Init_State_Diag (in Headers/state_diag_mod.F90)
 
GEOS-Chem ERROR [0006]: Error encountered within call to "Init_State_Diag"!
 --> LOCATION:  -> at GC_Init_StateObj (in GeosCore/gc_environment_mod.F90)
pe=00006 FAIL at line=00255    gchp_chunk_mod.F90                       <Error calling GC_Init_StateObj>
pe=00006 FAIL at line=01498    Chem_GridCompMod.F90                     <status=1>
pe=00006 FAIL at line=01807    MAPL_Generic.F90                         <status=1>

We are investigating further.

yantosca added 2 commits July 30, 2024 10:51
run/GCHP/HISTORY.rc.templates/HISTORY.rc.fullchem
- Now use SO4 instead of O3 for the Budget collection diagnostic fields.
  This will allow us to test The BudgetWetDep diagnostic, as SO4 is
  soluble and O3 is not.  Using O3 caused a runtime error, which should
  now be fixed.

CHANGELOG.md
- Updated accordingly

Signed-off-by: Bob Yantosca <yantosca@seas.harvard.edu>
test/integration/GCHP/integrationTestExecute.sh
- Add an sed command to reset the cap_restart time to 000000 UTC.
  This will make sure that we restart the run from 0h UTC when we
  manually restart tests with utils/redoIntegrationTestExecute.sh.

CHANGELOG.md
- Updated accordingly

Signed-off-by: Bob Yantosca <yantosca@seas.harvard.edu>
@yantosca
Copy link
Contributor

yantosca commented Jul 30, 2024

The GCHP error is due to the fact that the budget diagnostic used O3 as the sample species in the HISTORY.rc.fullchem template file:

 Budget.fields:          'BudgetWetDepFull_O3          ', 'GCHPchem',
                          'BudgetChemistryFull_O3       ', 'GCHPchem',
                          'BudgetConvectionFull_O3      ', 'GCHPchem',
                          'BudgetMixingFull_O3          ', 'GCHPchem',
                          'BudgetEmisDryDepFull_O3      ', 'GCHPchem',
                          #
                          'BudgetWetDepLevs1to35_O3     ', 'GCHPchem',
                          'BudgetChemistryLevs1to35_O3  ', 'GCHPchem',
                          'BudgetConvectionLevs1to35_O3 ', 'GCHPchem',
                          'BudgetMixingLevs1to35_O3     ', 'GCHPchem',
                          'BudgetEmisDryDepLevs1to35_O3 ', 'GCHPchem',
                           #
                          'BudgetWetDepTrop_O3          ', 'GCHPchem',
                          'BudgetChemistryTrop_O3       ', 'GCHPchem',
                          'BudgetConvectionTrop_O3      ', 'GCHPchem',
                          'BudgetMixingTrop_O3          ', 'GCHPchem',
                          'BudgetEmisDryDepTrop_O3      ', 'GCHPchem',
                           #
                          'BudgetWetDepPBL_O3           ', 'GCHPchem',
                          'BudgetChemistryPBL_O3        ', 'GCHPchem',
                          'BudgetConvectionPBL_O3       ', 'GCHPchem',
                          'BudgetMixingPBL_O3           ', 'GCHPchem',
                          'BudgetEmisDryDepPBL_O3  

But O3 is not water-soluble, and thus does not have a wet deposition ID. The BudgetWetDep*_O3 fields are therefore undefined, and caused GCHP to exit with an error.

We have now changed the sample species from O3 to SO4, which is water soluble.

@yantosca
Copy link
Contributor

yantosca commented Jul 30, 2024

Another GCHP error was revealed by the "alldiags" test:

GEOS-Chem ERROR [0010]: Could not find bin index for tag name: 574NM
 --> LOCATION:  -> at Get_UVFlux_Index (in module Headers/state_diag_mod.F90)
 
GEOS-Chem ERROR [0010]: Error encountered in routine "Get_UVFlux_Bin"!
 --> LOCATION:  -> at Get_Mapping (in module Headers/state_diag_mod.F90)

@lizziel and I have traced it to routine Get_UVFlux_Bin in Headers/state_diag_mod.F90:

    ! Get the index for the tagname                                                       
    DO N = 1, 18
       IF ( TRIM( tagName ) == TRIM( UVFlux_Tag_Names(N) ) ) THEN
          bin = N
          EXIT
       ENDIF
    ENDDO

    ! Trap potential errros                                                               
    IF ( bin < 0 ) THEN
       errMsg = 'Could not find bin index for tag name: ' // TRIM( tagName )
       CALL GC_Error( errMsg, RC, thisLoc )
       RETURN
    ENDIF

The error is that tagName is has NM in uppercase whereas TRIM( UVFlux_Tag_Names(N) has nm in lowercase. We will make sure that both are in uppercase when we do the comparison and rerun the tests.

Headers/state_diag_mod.F90
- In routine "Get_UVFLux_Bin", make sure that tagName and
  UVFlux_Tag_Names(N) are both in uppercase before comparing them
  for equality.

CHANGELOG.md
- Updated accordingly

Signed-off-by: Bob Yantosca <yantosca@seas.harvard.edu>
@yantosca
Copy link
Contributor

Encountered yet another error in the GCHP alldiags test. I think the job ran out of memory. I bumped up the requested memory to 450000 GB and will see if that goes

Interfaces/GCHP/gchp_historyexportsmod.F90
- Replace "informative message here" with a detailed error message.

run/GCHP/HISTORY.rc.templates/HISTORY.rc.fullchem
- Uncomment the StratBM diagnostic entries

test/integration/GCHP/integrationTestExecute.sh
- Add a sed command to comment out the StratBM collection in the
  collections list of HISTORY.rc for the GCHP "alldiags" test

CHANGELOG.md
- Updated accordingly

Signed-off-by: Bob Yantosca <yantosca@seas.harvard.edu>
@yantosca
Copy link
Contributor

The issue was not the job running out of memory, but the StratBM diagnostic entries were commented out. I've uncommented those entries but have disabled the StratBM collection for the GCHP alldiags test, as that is only needed
for 10-yr benchmarks.

run/GCHP/HISTORY.rc.templates/HISTORY.rc.fullchem
- Fix typo (i.e. "DryDepACET" -> "DryDep_ACET") in the DryDep collection
  entries.  This was causing a GCHP runtime error.

test/integration/GCHP/integrationTestCreate.sh
- Reactivate the StratBM collection, this should be OK.

CHANGELOG.md
- Updated accordingly

Signed-off-by: Bob Yantosca <yantosca@seas.harvard.edu>
@yantosca
Copy link
Contributor

Found that the GCHP DryDep collection entries had a bunch of missing _ characters, so entries were e.g. DryDepACET instead of DryDep_ACET. This was causing the GCHP alldiags test to fail. A fix has been pushed in commit dcc8809 and the GCHP integration tests are re-running.

run/GCHP/HISTORY.rc.fullchem
- Comment out (with `###` the Emissions diagnostic entries that are
  are also commented out in the corresponding HEMCO_Diagn.rc.fullchem
  template.  This will disable them in the GCHP "alldiags" test.

test/integration/GCClassic/integrationTestCreate.sh
test/integration/GCHP/integrationTestCreate.sh
- Disable the UVFlux collection in the "alldiags" integration test,
  as this is currently problematic.  Look to fix this later.

CHANGELOG.md
- Updated accordingly

Signed-off-by: Bob Yantosca <yantosca@seas.harvard.edu>
@yantosca
Copy link
Contributor

Also pushed the following updates:

  1. Fixed several inconsistencies in the emissions diagnostics between HISTORY.rc and HEMCO_Diagn.rc for GCHP.
  2. Disable the UVFlux diagnostic in the "alldiags" tests. Turning this on in GCHP results in a floating-point error, see: Turning on UVFlux diagnostic collection results in a floating point error GCHP#436

run/GCHP/HISTORY.rc.templates/HISTORY.rc.fullchem
- Added an extra # (now '####') when commenting out the unused
  diagnostic entries in the Emissions collection.  This is needed
  for the "alldiags" test, which strips the first 3 '###' chars.

Signed-off-by: Bob Yantosca <yantosca@seas.harvard.edu>
@yantosca yantosca requested a review from lizziel July 31, 2024 18:41
@yantosca yantosca self-assigned this Jul 31, 2024
@yantosca
Copy link
Contributor

After adding all of the above-mentioned fixes, all GCHP integration tests now pass:

==============================================================================
GCHP: Execution Test Results

CodeDir       : 2d1bc44 GEOS-Chem submod update: Merge PR #2394 (Add GCAP int test)
MAPL          : 231d53c Merge pull request #36 from geoschem/feature/improve_hflux_regridding
GMAO_Shared   : 4ddb3ec Merge pull request #2 from geoschem/feature/mapl-upgrade
ESMA_cmake    : ad5deba Added ecbuild as a submodule of ESMA_cmake
gFTL-shared   : 4b82492 Merge branch 'upstream_v1.5.0' into feature/v1.5.0
FMS           : 259759d Merge pull request #3 from geoschem/feature/update_gmao_libs
FVdycoreCubed : af42462 Merge PR #8 (Add PLEadv diagnostic for offline advection in GCHP)
geos-chem     : ee5a477e6 Add an extra "#" to commented-out emission diagnostics
HEMCO         : e68ebae HEMCO 3.9.2 release
yaFyaml       : 19afe50 Merge branch 'upstream_v1.0.4' into feature/v1.0.4
pFlogger      : 2c4b724 Merge branch 'upstream_v1.9.1' into feature/v1.9.1
Cloud-J       : 37a4ece Update release version number and date
HETP          : 2a99b24 Merge pull request #2 from geoschem/bugfix/initialize_local_variables

Number of execution tests: 12

Submitted as SLURM job: 42064039
==============================================================================
...
gchp_merra2_fullchem_alldiags.......................Execute Simulation....PASS
...
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%  All execution tests passed!  %%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

All tests were zero-diff w/r/t PR #2394.

@yantosca yantosca marked this pull request as ready for review July 31, 2024 18:44
@yantosca yantosca added this to the 14.4.3 milestone Jul 31, 2024
@yantosca
Copy link
Contributor

@msulprizio: GitHub wouldn't let me add you as a reviewer since this was originally your PR. But please do feel free to submit a review.

Copy link
Contributor Author

@msulprizio msulprizio left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks good to me @yantosca. Thanks for resolving these issues! I have one minor comment in the CHANGELOG but pending that this is good for merging.

CHANGELOG.md Outdated Show resolved Hide resolved
CHANGELOG.md
- Added "and GCHP" to the "Added simulation with all diagnostics..."
  sentence.

Signed-off-by: Bob Yantosca <yantosca@seas.harvard.edu>
@yantosca yantosca self-requested a review August 1, 2024 17:09
@yantosca
Copy link
Contributor

yantosca commented Aug 1, 2024

@lizziel @msulprizio: I merged (locally) into dev/no-diff-to-benchmark and reran integration tests. All tests passed and were zero-diff w/r/t the prior PR (#2394).

@lizziel lizziel removed their request for review August 1, 2024 20:02
@yantosca yantosca merged commit 3e0f6ce into dev/no-diff-to-benchmark Aug 1, 2024
@yantosca yantosca deleted the feature/all-diags-inttest branch August 1, 2024 20:12
yantosca pushed a commit that referenced this pull request Aug 5, 2024
This rebase brings the "feature/trop-prs-in-satdiagn" branch (which contains
an update by @eamarais to add tropopause pressure to the SatDiagn collection)
into the "dev/no-diff-to-benchmark" branch atop PR #2399.  The original update
was based off version 14.1.0.

This merge brings PR # ( , by @) into

Signed-off-by: Bob Yantosca <yantosca@seas.harvard.edu>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
no-diff-to-benchmark This update will not change the results of fullchem benchmark simulations topic: Benchmarking and Testing Related to CI, integration tests, or scientific benchmarking
Projects
None yet
2 participants