Possible parallelization or initialization error in RRTMG diagnostic output

[yantosca]

Name and Institution (Required)

Name: Bob Yantosca
Institution: Harvard + GCST

Confirm you have reviewed the following documentation

• Support guidelines
• User manuals
• Debugging GEOS-Chem and HEMCO errors
• Current and past Github issues

Description of your issue or question

There seems to be a parallelization (or perhaps an initialization) issue in the RRTMG diagnostic output. When performing a difference test on the integration test output between PR #1808 (Ref) and PR #1682 (Dev), we get this output:

Checking gc_4x5_merra2_fullchem_RRTMG
   -> 1 difference found in OutputDir
      * GCC_14.2.1_r8/rundirs/gc_4x5_merra2_fullchem_RRTMG/OutputDir/GEOSChem.RRTMG.20190701_0000z.nc4 
        GCC_14.2.1_r9/rundirs/gc_4x5_merra2_fullchem_RRTMG/OutputDir/GEOSChem.RRTMG.20190701_0000z.nc4 
   -> No differences in Restarts

This only affects the GEOS-Chem RRTMG diagnostic output. The following variable show differences. In the table are sum(Ref), sum(Dev), and absolute difference. Identical output would show Dev - Ref = 0.

Variable               Ref=PR #1808             Dev=PR #1682             Dev - Ref
RadAllSkySWSurf_AM   : -534.46027             | -534.56793             | -0.10762405395507812 
RadAllSkySWSurf_BASE : 591751.94              | 591755.5               | 3.584364175796509 
RadAllSkySWSurf_BC   : -773.7683              | -770.18396             | 3.584364175796509 
RadAllSkySWSurf_DU   : -762.5253              | -761.77686             | 0.748519778251648 
RadAllSkySWSurf_ME   : -602.89856             | -602.8953              | 0.0032442808151245117 
RadAllSkySWSurf_NI   : -519.60486             | -519.5393              | 0.0655827522277832 
RadAllSkySWSurf_O3   : -1234.2449             | -1234.2555             | -0.010667502880096436 
RadAllSkySWSurf_OA   : -2484.1523             | -2484.2134             | -0.061360105872154236 
RadAllSkySWSurf_PM   : -10682.398             | -10678.814             | 3.584364175796509 
RadAllSkySWSurf_SS   : -2945.451              | -2946.6897             | -1.2389745712280273 
RadAllSkySWSurf_SU   : -1855.739              | -1856.278              | -0.538988471031189 
RadAllSkySWTOA_AM    : -363.19217             | -363.29504             | -0.1028774082660675 
RadAllSkySWTOA_BASE  : -257835.12             | -257838.97             | -3.8440985679626465 
RadAllSkySWTOA_BC    : 257.54788              | 253.70377              | -3.8440985679626465 
RadAllSkySWTOA_DU    : -456.84576             | -457.04156             | -0.19583594799041748 
RadAllSkySWTOA_ME    : 191.23547              | 191.23697              | 0.0015055537223815918 
RadAllSkySWTOA_NI    : -348.15295             | -348.02716             | 0.1257724165916443 
RadAllSkySWTOA_O3    : 676.74066              | 676.784                | 0.04335308074951172 
RadAllSkySWTOA_OA    : -1441.3467             | -1441.0868             | 0.2600269317626953 
RadAllSkySWTOA_PM    : -7118.54               | -7122.384              | -3.8440985679626465 
RadAllSkySWTOA_SS    : -2665.7197             | -2665.6082             | 0.11187988519668579 
RadAllSkySWTOA_SU    : -1300.0856             | -1299.6527             | 0.43287360668182373 
RadAsym550nm_BC      : 653.76587              | 661.4774               | 7.711620330810547 
RadAsym550nm_PM      : 2343.625               | 2343.521               | -0.10405570268630981 
RadClrSkySWSurf_AM   : -785.2866              | -785.2456              | 0.0409880131483078 
RadClrSkySWSurf_BASE : 724746.75              | 724750.4               | 3.639738082885742 
RadClrSkySWSurf_BC   : -926.70483             | -923.0651              | 3.639738082885742 
RadClrSkySWSurf_DU   : -1072.2739             | -1071.5387             | 0.7352849841117859 
RadClrSkySWSurf_ME   : -968.12524             | -968.12555             | -0.000293537974357605 
RadClrSkySWSurf_NI   : -726.4045              | -726.44495             | -0.04040423035621643 
RadClrSkySWSurf_O3   : -1449.5626             | -1449.618              | -0.055350154638290405 
RadClrSkySWSurf_OA   : -3502.014              | -3501.9746             | 0.039353832602500916 
RadClrSkySWSurf_PM   : -15184.318             | -15180.678             | 3.639738082885742 
RadClrSkySWSurf_SS   : -4121.166              | -4122.234              | -1.068190097808838 
RadClrSkySWSurf_SU   : -2729.2139             | -2730.0894             | -0.8756707906723022 
RadClrSkySWTOA_AM    : -576.6275              | -576.6278              | -0.00037920475006103516 
RadClrSkySWTOA_BASE  : -146972.0              | -146976.34             | -4.335667133331299 
RadClrSkySWTOA_BC    : 222.26451              | 217.92886              | -4.335667133331299 
RadClrSkySWTOA_DU    : -727.47003             | -727.66864             | -0.1985204815864563 
RadClrSkySWTOA_ME    : 166.84091              | 166.84097              | 7.396936416625977e-05 
RadClrSkySWTOA_NI    : -537.06134             | -537.0442              | 0.017227470874786377 
RadClrSkySWTOA_O3    : 485.25955              | 485.28024              | 0.020655155181884766 
RadClrSkySWTOA_OA    : -2275.7937             | -2275.5698             | 0.2238166332244873 
RadClrSkySWTOA_PM    : -10961.783             | -10966.118             | -4.335667133331299 
RadClrSkySWTOA_SS    : -3754.903              | -3754.7585             | 0.1446855664253235 
RadClrSkySWTOA_SU    : -2040.9421             | -2040.5842             | 0.3578859567642212 
RadSSA550nm_BC       : 759.38245              | 629.96185              | -129.42063903808594 
RadSSA550nm_PM       : 3249.4487              | 3249.6033              | 0.1546829342842102 

[yantosca]

A typical pattern of difference shows "checkerboarding", which is typical of a parallelization error:

[yantosca]

I think the differences are caused by the RRTMG diagnostics in State_Diag not being reset to zero before each call to RRTMG. I am working on a fix that will do this and will run a parallel test again.