[BUG/ISSUE] GEOS-FP Convection Change (June 2020-onward) can have large impacts on surface concentrations

[chfite]

Description of the problem

First off, this message was also emailed to the Transport Working Group to get their input on the issue. However, I wanted to post this on here as well so that other users could be aware of the issue since it could potentially have large impacts on their results.

This is an issue we found with convection in the GEOS-FP files that begins in June 2020.

I had initially noticed in my GCClassic v13.3.4 nested simulation (0.25x0.3125, complexSOA_SVPOA) that my surface CO and PM2.5 concentrations began dropping off very drastically, WELL below observations, beginning around June 2020. Initially we assumed this could be a nested model or boundary condition issue but we saw the same drop-off in concentrations when using my 2x2.5 global simulation. So, we concluded the issue was likely in GEOS-FP.

I then ran a global transport tracer simulation at 2x2.5 resolution (non-local PBL turned on) to further diagnose the issue. Please see the attached PDF file with some figures of the model results. The COUniformEmis25dayTracer output was saved out as 7-day means for the 13 month period of August 2019 - August 2020.

Results are here: TransportTracerSim_Results.pdf

The budget diagnostic time series of CO confirms a sudden increased loss in CO (in the PBL) via convection, which also drives changes in the transport and mixing+emissions terms.

The plots showing the upward moist convective mass flux (CMFMC) show a near doubling that begins on June 1st, 2020. This can be seen in the monthly mean maps, as well as the global time series. For the CMFMC plots I used the raw 2x25 GEOS-FP that are 3-hourly. I have not explored CMFMC past August 2020, so I am not sure whether this issue gets resolved further on, but I suspect it doesn’t.

While looking at the GMAO website (https://gmao.gsfc.nasa.gov/GMAO_products/NRT_products.php#), we suspect that the convection update that occurred on March 30, 2020 (updated on April 7th, 2020) could maybe be the culprit, although that does not exactly correspond with when we see the doubling in CMFMC on June 1st.

We want users to be aware of this issue with GEOS-FP since it can have large impacts on the convection, mixing, and wet deposition terms for species.

[yantosca]

Adding some discussion to this thread:

@ltmurray wrote:

Whereas GMAO updated convection at the end of March, they performed a few months overlap with the old version and the new version. The GCSC decided we would provide the old stream until it was no longer available in order to avoid having such large transport discontinuities in the middle of the COVID shutdown period for anyone studying air quality during that time. That would explain the delay you see.

@aschuh wrote:

Unless I'm missing something, this has been a known change in FP for a long time, right? Going from RAS to GrellFreitas convection. I made plots of the conv mass flux diffs months ago and made several comments to folks on it, as I was drafting a paper on GC convection, submitted to acpd maybe a month ago. This is why most folks would be safer using a reanalysis. In fact, this is why reanalysis is created. For some reason most people (in the CO2 world at least) use FP despite these risks, I guess assuming 0.25deg must be viewed as better than 0.5deg for some reason (?) or FP has more data being assimilated (?). I've explained the exact risk associated with this transition many times. Honestly, happier to see it pop up in this context then as an inaccurate inference in a nature paper : ). Again, this isn't a bug, its exactly what is to be expected in the forward stream of a real time DA system, unless I'm missing something.

I made initial comparisons over 3-4 months early in the transition (shown in the figure attached) and discussed with Hongyu I believe. I was hoping to get a full year of the new FP and then run that year climatologically against a year of the old. Maybe 4-5 years, that should give good indication of expected diffs, at least for long lived tracers. Can't speak for short lived tracers but I'd be shocked if it didn't have a big effect on CO2 meridional gradients.

I've attached a quick and dirty image I made of the diffs back in May in the doc. This is zonally averaged CMFMC, i.e. a curtain, vertical by latitude. Forget the units, but they are the same across panels. Sorry, just indices, south pole to left, north pole to right, vertical model levels in y-axis. Top panel is RAS, middle is GF and bottom is difference. 3-4 months in avg, don't recall which. As was mentioned, huge changes. Plus there is this little pocket of zero conv mass flux around the 20th model level you'll notice, that was a concern as well since I'm not positive how tracers will move across it.

GF_vs_RAS.docx

As far I understand it, there really isn't much to do here, just wait for the next full reanalysis run? Feel free to correct me if I'm seeing this wrong.

Andrea Molod wrote:

i am going to make a comment about this that is from memory.....when the switch to the FP system with the new model took place in 2020, a few months in to the run the issue with the (incorrect) mass flux was
detected. this is not a ras-gf change only, it was an outright bug. there is of course a change from RAS to GF here (the GF ones should be larger as the “scale aware” “squelching” does not kick in for GF until higher resolution), but this was a bug that was discovered.

now here is where it is.... foggy. i thought they ended up doing a re-wind with the correct output and re-ran that first few month time period, and replaced the FP on our servers. if the output was grabbed before
that replacement happened, this could be what you are seeing.

if they decided not to do a re-run but just “live with the bug” and fix it going forward, there may be a
short time period with bad fluxes that ends at some point and then the fluxes look ok.

the best person to ask about all this and get the details (dates, etc..) from is rob lucchessi.

also, here is the web site that you can click around in to see more info:
https://gmao.gsfc.nasa.gov/GMAO_products/NRT_products.php#

one last “point” here – if the re-run was NOT done, it is also not a big deal to request that it be done. we have a mechanism called “exact replay” that minimizes the cost of the re-run. either christoph or emma knowland can make this happen if the request comes from the GC community....

[stale]

This issue has been automatically marked as stale because it has not had recent activity. If there are no updates within 7 days it will be closed. You can add the "never stale" tag to prevent the Stale bot from closing this issue.

[cdholmes]

According to the timeline below, none of the GEOS-FP data contained in the GEOS-Chem archive should be affected by the convection bug mentioned by Andrea Molod. The issue still remains that GEOS-Chem Classic running with GEOS-FP meteorology after 1 June 2020 produces unrealistically low surface concentrations of CO, among other things.

Timeline based on GMAO release notes.

• 30 Jan 2020: GEOS-FP updated to GEOS-5.25.1. The update adopted the Grell-Freitas convection parameterization, among other things. GMAO continued running the old GEOS-FP in parallel, which is what is contained in the GEOS-Chem met archive.
• 7 Apr 2020: GEOS-FP updated to GEOS-5.25.1-p5, which fixed an error in the Grell-Freitas convection code. The fix affected convective mass flux and transport.
• 1 June 2020: GEOS-Chem met data archive switched to the then-current version of GEOS-FP.

[cdholmes]

If any other GMAO products (e.g. GEOS-CF, GEOS-IT) are based on GEOS-5.25.1 and later they likely also produce unrealistic surface CO concentrations when used in GEOS-Chem Classic.

[lizziel]

After discussion with Randall and Daniel, we have decided to do 1-year transport tracer benchmarks for 2021 comparing MERRA2, GEOS-FP, and GEOS-IT, for both GCHP and GC-Classic. I will follow up here with results when they are available.

[aschuh]

STRONGLY recommend, I have not had the time to do this myself but I feel it is pretty critical. I believe 1 year is probably sufficient w/ passive tracers (I assume?).    With 2-3 years, you really see the seasonal patterns emerge clearly but 1 will give you great info.  What tracers will be run?  For tracers tied to NH emissions (CO2, fossil/anthro, etc), it is sort of important to see these effects.  Will you have some decay tracers, full passive, a few zonal, etc?  Not sure what the full suite looks like I guess. andrew

[lizziel]

@aschuh, the benchmarks will include the full set of transport tracers. Plots/tables generated will be everything typically included in the transport tracer benchnmark. The restart file will be the same one used for the MERRA2 2019 benchmariks.

The full set of transport tracers is listed on the wiki here. You can see what types of plots and tables will be generated from the transport tracer benchmark section of the 14.2.0 wiki page here.

[lizziel]

@cdholmes, @aschuh, @chfite - The 2021 transport tracer benchmark comparison for MERRA2, GEOS-FP, and GEOS-IT is now available to view here.

Initial thoughts and observations are included in a summary document here. Overall we do not see any show-stopping differences between MERRA2 and GEOS-FP. As expected, GEOS-FP and GEOS-IT are similar in their differences to MERRA2. Please let us know if you have any comments.

[aschuh]

These are great, thanks Bob. What I'm looking for in these diagnostics are signs that the convective movement of tracers is different, I think that is one of the big changes, RAS vs GF. The convective mass fluxes from the driving met appear to point to more midlat low level conv mixing in FP and maybe stronger vertical motion across the globe in FP, still w/ a few odd differences (I recall?).

My first glance at these, looking at stuff like SF6 which should proxy CO2 fossil which I'm familiar with, seems to indicate that GEOS-FP has stronger mixing but not nearly enough to mimic what we saw in ERA-i/TM5 sims. I'm going to pass these around to a few folks I'm working with to get impressions. Will likely pass back questions and additional impressions...

[yantosca]

Thank @lizziel, she did all the runs!

[viral211]

A difference between RAS and GF output is that DQRCU in GF is the net precip formation rate (precip production- evaporation), but in RAS it was just the production rate (#2469). DQRCU is used in GEOS-Chem convection to diagnose the cloud base height, which is set at the lowermost level where DQRCU>0. In addition, the convection module instantly mixes the entire column below the cloud base. The lowermost layer where DQRCU turns positive in GF is usually at around 2 km (higher than in RAS). Uniform mixing to higher altitudes may be one reason we see these large differences in surface concentrations after switching to GF. Tagging @yuanjianz.

[lizziel]

For another reference of impact of the switch see @msulprizio's full chemistry benchmark for January 2020 located here:
https://wiki.seas.harvard.edu/geos-chem/index.php/Benchmark/GEOS-FP_5.25#Major_update_to_GEOS-FP_meteorology.
That benchmark was done at the time we did the change in GEOS-FP version used to generate the met-fields.

[yuanjianz]

Hi @viral211, thanks for the information! Since DQRCU by definition is changed to net precipitation (MERRA-2 definition DQRCU minus layer-wise re-evaporation), do you think we should add layer-wise REEVAP back to match the DQRCU for determining cloud base height? It may lower the cloud base height somehow, given it seems some gas tracer is substantially below observation constraint? (For surface aerosol, since we always overestimate comparing to ground-based measurement, it seems desirable though.)

[viral211]

Hi @yuanjianz. You are right. We will at least then be consistent with the current approach based on the RAS. Good point about the surface aerosol concentrations though.