Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CMAQ-WRF coupling: cherry pick #1609

Merged
merged 16 commits into from
Jan 26, 2022

Conversation

davegill
Copy link
Contributor

@davegill davegill commented Dec 19, 2021

TYPE: new feature

KEYWORDS: CMAQ, coupled model, constructing, WRF-CMAQ, executable

SOURCE: David Wong, (US EPA)

DESCRIPTION OF CHANGES:
These modifications are for facilitating constructing a WRF-CMAQ coupled
model without compromising creating a regular WRF executable. The WRF-CMAQ only interacts with the RRTMG SW option. The aerosol information from CMAQ (a chemistry transport model) will affect the aerosol optical depth,aerosol single scattering albedo and aerosol asymmetry parameter calculations within RRTMG model.

Problem:
A WRF-CMAQ coupled model was developed in US EPA in around 2008. The work
has been published in a journal article [1]. Since then, the coupled model has been
applied to various studies and region to validate the effectiveness of the model [2-12].
Throughout the WRF released cycle, the coupled has been updated manually into a subset
of WRF version. The update process was tedious and error prone. With this PR, the
manual update process can be eliminated.

LIST OF MODIFIED FILES:
modified: Makefile
new file: Registry/registry.CMAQ
modified: Registry/registry.em_shared_collection
modified: arch/Config.pl
modified: clean
modified: configure
modified: dyn_em/module_first_rk_step_part1.F
modified: dyn_em/solve_em.F
modified: main/Makefile
modified: main/depend.common
modified: phys/Makefile
new file: phys/complex_number_module.F
modified: phys/module_ra_rrtmg_sw.F
new file: phys/module_ra_rrtmg_aero_optical_util_cmaq.F
modified: phys/module_radiation_driver.F
modified: share/module_check_a_mundo.F
modified: wrftladj/module_first_rk_step_part1_ad.F
modified: wrftladj/module_first_rk_step_part1_tl.F
modified: wrftladj/solve_em_ad.F
modified: wrftladj/solve_em_tl.F

TESTS CONDUCTED:

Three different models were built with given lines of instructions:

  • model 1 (used the original WRF source code without any changes in this PR)

    • went through a regular configure process to build a regular WRF model with the configure command
    • compiled the source code with usual command: compile em_real to generate a WRF executable
      (for discussion purposes, the WRF executable is named p.exe)
  • model 2 (used the new WRF source code with all the changes in this PR)

    • went through a regular configure process to build a regular WRF model with the configure command
    • compiled the source code with usual command: compile em_real to generate a WRF executable.
      (for discussion purposes, the WRF executable is named np.exe)
  • model 3 (used the new WRF source code with all the changes in this PR)

    • issue the following two environment setting commands:

      setenv WRF_CMAQ 1
      setenv IOAPI /home/wdx/lib/x86_64/ifc-18.0/ioapi-3.2-2020220

      (Note: ioapi_3.2-2020220 is another third-party library)

    • went through a regular configure process to build a regular WRF model with the configure command

    • compiled the source code with usual command: compile em_real to generate a WRF executable.
      (for discussion purposes, the WRF executable is named nc.exe)

Case p was based on the executable of model 1, case np was based on the executable of model 2, and
case ncX, where X represents various available options in running the WRF-CMAQ coupled model and detailed
information is provided below, was based on the executable of model 3. All cases were tested
with two days of simulation, 2016/7/1 - 2016/7/2. To verify the correctness of this new PR, a bit-by-bit
comparison of every variable in all time steps on the second day of the wrfout file between two selected
cases.

One of the WRF-CMAQ coupled model running setting, wrf_cmaq_option:

 0 = run WRF only
 1 = run WRF only and           w   producing MCIP like GRID and MET files
 2 = run WRF-CMAQ coupled model w/o producing MCIP like GRID and MET files
 3 = run WRF-CMAQ coupled model w   producing MCIP like GRID and MET files

In these scenarios, X was set to the wrf_cmaq_option and another coupled model
setting, direct_sw_feedback was set to .false., where direct_sw_feedback is a
switch to turn on short wave aerosol direct effect or not.

For an additional case, X = s means

wrf_cmaq_option = 3, and direct_sw_feedback was set to .true.

  • comparing case p and case np

    Identical result was rendered. This means that this new PR does not alter anything in terms of
    constructing a regular WRF executable.

  • comparing case np and nc0

    Identical result was rendered. This means that this new PR can make the WRF-CMAQ coupled behave
    as a regular WRF model

  • comparing case nc0, nc1, nc2 and nc3

    Identical result was rendered. This means that this new PR with or without the chemistry portion
    of the coupled model does not change the WRF result

  • comparing case nc0 and ncs
    Different result was rendered with respect to any variables can be affect by radiation calculation.
    This means the aerosol information from CMAQ was transferred to WRF and radiation calculation through
    the short-wave aerosol direct effect.

The COAWST and CTSM developers, that both use WRF as part of a coupled system, did not find anything
problematic with this set of mods to enable coupling WRF with CMAQ. Thank you John Warner and Negin
Sobhani for your very helpful contributions during the evaluation process.

Here is some of the research work based on the WRF-CMAQ coupled model:

  1. Jia Xing, Jiandong Wang, Rohit Mathur, Shuxiao Wang, Golam Sarwar, Jonathan Pleim, Christian Hogrefe,
    Yuqiang Zhang, Jingkun Jiang, David C. Wong, Jimin Hao,
    "Impacts of aerosol direct effects on tropospheric ozone through changes in atmospheric dynamics and
    photolysis rates",
    Atmos. Chem. Phys., 17, 9869-9883, 2017.

  2. Wang, J., Xing, J., Mathur, R., Pleim, J., Wang, S. Hogrefe, C., Gan, C-M, Wong. D. C., and Hao, J.,
    "Historical trends in PM2.5 related premature mortality during 1990-2010 across the northern hemisphere",
    Environmental Health Perspectives, Vol 125, Number 3, p400-408, March 2017.

  3. Xing, J., Wang, J. Mathur, R., Pleim, J., Wang, S. Hogrefe, C., Gan, C-M, Wong. D. C., and Hao, J.,
    "Unexpected benefits of reducing aerosol cooling effects",
    Environmental Science and Technology, 2016 Jul 19, 50(14):7527-34.

  4. Gan, C-M., Hogrefe, C., Pleim, J., Mathur, R., Xing, J., Wong, D. C., Gilliam, R., Pouliot, G., and Wei, C.,
    "Assessment of the Effects of Horizontal Grid Resolution on Long-Term Air Quality Trends using Coupled
    WRF-CMAQ Simulations",
    Atmospheric Environment, Volume 132, May 2016, Pages 207-216.

  5. Xing, J., Mathur, R., Pleim, J., Hogrefe, C., Gan, C.-M., Wong, D. C., Wei C., and Wang J.,
    "Air pollution and climate response to aerosol direct radiative effects: a modeling study of decadal
    trends across the northern hemisphere",
    Journal of Geophysical Research: Atmospheres, Vol. 120, Issue 23, Pages 12221 - 12236, Dec 2015.

  6. Xing, J., Mathur, R., Pleim, J., Hogrefe, C., Gan, C.-M., Wong, D. C., and Wei C.,
    "Can a coupled meteorology-chemistry model reproduce the historical trend in aerosol direct radiative
    effects over the Northern Hemisphere?",
    Atmos. Chem. Phys., 15, 9997-10018, 2015.

  7. Gan, C-M., Pleim, J., Mathur, R., Hogrefe, C., Long, C., Xing, J., Wong, D. C., Gilliam, R., and Wei, C.,
    "Assessment of long-term WRF-CMAQ simulations for understanding direct aerosol effects on radiation
    "brightening" in the United States",
    Atmos. Chem. Phys., 15, 12193-12209, 2015

  8. Gan, C-M., Binkowski, F., Pleim, J., Xing, J., Wong, D. C., Mathur, R., and Gilliam, R.,
    "Assessment of the aerosol optics component of the coupled WRF-CMAQ model using CARES field campaign data
    and a single column model",
    Atmospheric Environment, Volume 115, Pages 670-682, August 2015.

  9. Hogrefe, C., Pouliot, G., Wong, D. C., Torian, A., Roselle, S., Pleim, J., and Mathur, R.,
    "Annual application and evaluation of the online coupled WRF-CMAQ system over North America under AQMEII phase 2",
    Atmospheric Environment, Volume 115, Pages 1-756, August 2015.

  10. Xing, J., Mathur, R., Pleim, J., Hogrefe, C., Gan, C.-M., Wong, D. C., Wei, C., Gilliam, R., and Pouliot, G.,
    "Observations and modeling of air quality trends over 1990-2010 across the Northern Hemisphere: China, the
    United States and Europe",
    Atmos. Chem. Phys., 15, 2723-2747, 2015.

  11. Wang, J., Wang, S., Jiang, J., Ding, A., Zheng, M., Zhao, B., Wong, D. C., Zhou, W., Zheng, G., Wang, L.,
    Pleim, J. E., and Hao, J.,
    "Impact of aerosol-meteorology interactions on fine particle pollution during China's severe haze episode
    in January 2013",
    Environ. Res. Lett. 9, September 2014.

  12. Wong, D. C., Pleim, J., Mathur, R., Binkowski, F., Otte, T., Gilliam, R., Pouliot, G., Xiu, A., and Kang, D.,
    "WRF-CMAQ two-way coupled system with aerosol feedback: software development and preliminary results",
    Geosci. Model Dev., 5, 299-312, 2012.

RELEASE NOTE: The capability to couple WRF and CMAQ (an air quality model developed at the US EPA) is added. A user can choose to compile WRF as usual or to construct WRF-CMAQ coupled model and then compile. For the latter choice, a user can run the WRF-CMAQ coupled model in an one-way fashion, i.e. meteorological information will be transferred to CMAQ directly to drive its calculation, or in a two-way fashion, i.e. the aerosol information from CMAQ will be fed back to the RRTMG radiation physics in WRF. For those who wants to construct WRF-CMAQ coupled model with WRF version 4.4+ and CMAQ version v5.3.3.3, please refer to https://www.epa.gov/cmaq/access-cmaq-source-code for latest instruction on how to use this capability.

dwongepa and others added 2 commits December 19, 2021 21:07
KEYWORDS: CMAQ, coupled model, constructing, WRF-CMAQ, executable

SOURCE: David Wong, US EPA (wong.david-c@epa.gov)

DESCRIPTION OF CHANGES:
These modifications are for facilitating constructing a WRF-CMAQ coupled
model without compromising creating a regular WRF executable.

Problem:
A WRF-CMAQ coupled model was developed in US EPA in around 2008. The work
has been published in a journal article [1]. Since then, the coupled model has been
applied to various studies and region to validate the effectiveness of the model [2-12].
Throughout the WRF released cycle, the coupled has been updated manually into a subset
of WRF version. The update process was tedious and error prone. With this PR, the
manual update process can be eliminated.

LIST OF MODIFIED FILES: list of changed files (use git diff --name-status master to get formatted list)

newly added file:

  Registry/registry.WRF-CMAQ-twoway
  phys/complex_number_module.F
  phys/module_twoway_ra_rrtmg_sw.F
  phys/module_twoway_rrtmg_aero_optical_util.F

modified file:

  Makefile
  clean
  configure
  Registry/Registry.EM
  Registry/registry.em_shared_collection
  arch/Config.pl
  dyn_em/module_first_rk_step_part1.F
  dyn_em/solve_em.F
  external/io_netcdf/makefile
  main/Makefile
  main/depend.common
  phys/Makefile
  phys/module_ra_rrtmg_sw.F
  phys/module_radiation_driver.F

AGAIN cherry pick to try to fix old WRF with CMAQ
modified:   Makefile
modified:   Registry/Registry.EM
new file:   Registry/registry.WRF-CMAQ-twoway
modified:   Registry/registry.em_shared_collection
modified:   arch/Config.pl
modified:   clean
modified:   configure
modified:   dyn_em/module_first_rk_step_part1.F
modified:   dyn_em/solve_em.F
modified:   external/io_netcdf/makefile
modified:   main/Makefile
modified:   main/depend.common
modified:   phys/Makefile
new file:   phys/complex_number_module.F
modified:   phys/module_ra_rrtmg_sw.F
modified:   phys/module_radiation_driver.F
new file:   phys/module_twoway_ra_rrtmg_sw.F
new file:   phys/module_twoway_rrtmg_aero_optical_util.F

TESTS CONDUCTED:

Three different models were built with given lines of instructions:

 * model 1 (used the original WRF source code without any changes in this PR)

   + went through a regular configure process to build a regular WRF model with the configure command
   + compiled the source code with usual command: compile em_real to generate a WRF executable
     (for discussion purposes, the WRF executable is named p.exe)

 * model 2 (used the new WRF source code with all the changes in this PR)

   + went through a regular configure process to build a regular WRF model with the configure command
   + compiled the source code with usual command: compile em_real to generate a WRF executable.
     (for discussion purposes, the WRF executable is named np.exe)

 * model 3 (used the new WRF source code with all the changes in this PR)

   + issue the following two environment setting commands:

     setenv WRF_CMAQ 1
     setenv IOAPI /home/wdx/lib/x86_64/ifc-18.0/ioapi-3.2-2020220

     (Note: ioapi_3.2-2020220 is another third-party library)

   + went through a regular configure process to build a regular WRF model with the configure command
   + compiled the source code with usual command: compile em_real to generate a WRF executable.
     (for discussion purposes, the WRF executable is named nc.exe)

Case p was based on the executable of model 1, case np was based on the executable of model 2, and
case ncX, where X represents various available options in running the WRF-CMAQ coupled model and detailed
information is provided below, was based on the executable of model 3. All cases were tested
with two days of simulation, 2016/7/1 - 2016/7/2. To verify the correctness of this new PR, a bit-by-bit
comparison of every variable in all time steps on the second day of the wrfout file between two selected
cases.

   One of the WRF-CMAQ coupled model running setting, wrf_cmaq_option:

     0 = run WRF only
     1 = run WRF only and           w   producing MCIP like GRID and MET files
     2 = run WRF-CMAQ coupled model w/o producing MCIP like GRID and MET files
     3 = run WRF-CMAQ coupled model w   producing MCIP like GRID and MET files

   In these scenarios, X was set to the wrf_cmaq_option and another coupled model
   setting, direct_sw_feedback was set to .false., where direct_sw_feedback is a
   switch to turn on short wave aerosol direct effect or not.

   For an additional case, X = s means

   wrf_cmaq_option = 3, and direct_sw_feedback was set to .true.

 * comparing case p and case np

   Identical result was rendered. This means that this new PR does not alter anything in terms of
   constructing a regular WRF executable.

 * comparing case np and nc0

   Identical result was rendered. This means that this new PR can make the WRF-CMAQ coupled behave
   as a regular WRF model

 * comparing case nc0, nc1, nc2 and nc3

   Identical result was rendered. This means that this new PR with or without the chemistry portion
   of the coupled model does not change the WRF result

 * comparing case nc0 and ncs
   Different result was rendered with respect to any variables can be affect by radiation calculation.
   This means the aerosol information from CMAQ was transferred to WRF and radiation calculation through
   the short-wave aerosol direct effect.

Here are some of the research work based on the WRF-CMAQ coupled model:

12. Jia Xing, Jiandong Wang, Rohit Mathur, Shuxiao Wang, Golam Sarwar, Jonathan Pleim, Christian Hogrefe,
Yuqiang Zhang, Jingkun Jiang, David C. Wong, Jimin Hao,
"Impacts of aerosol direct effects on tropospheric ozone through changes in atmospheric dynamics and
photolysis rates",
Atmos. Chem. Phys., 17, 9869-9883, 2017.

11. Wang, J., Xing, J., Mathur, R., Pleim, J., Wang, S. Hogrefe, C., Gan, C-M, Wong. D. C., and Hao, J.,
"Historical trends in PM2.5 related premature mortality during 1990-2010 across the northern hemisphere",
Environmental Health Perspectives, Vol 125, Number 3, p400-408, March 2017.

10. Xing, J., Wang, J. Mathur, R., Pleim, J., Wang, S. Hogrefe, C., Gan, C-M, Wong. D. C., and Hao, J.,
"Unexpected benefits of reducing aerosol cooling effects",
Environmental Science and Technology, 2016 Jul 19, 50(14):7527-34.

 9. Gan, C-M., Hogrefe, C., Pleim, J., Mathur, R., Xing, J., Wong, D. C., Gilliam, R., Pouliot, G., and Wei, C.,
"Assessment of the Effects of Horizontal Grid Resolution on Long-Term Air Quality Trends using Coupled
WRF-CMAQ Simulations",
Atmospheric Environment, Volume 132, May 2016, Pages 207-216.

 8. Xing, J., Mathur, R., Pleim, J., Hogrefe, C., Gan, C.-M., Wong, D. C., Wei C., and Wang J.,
"Air pollution and climate response to aerosol direct radiative effects: a modeling study of decadal
trends across the northern hemisphere",
Journal of Geophysical Research: Atmospheres, Vol. 120, Issue 23, Pages 12221 - 12236, Dec 2015.

 7. Xing, J., Mathur, R., Pleim, J., Hogrefe, C., Gan, C.-M., Wong, D. C., and Wei C.,
"Can a coupled meteorology-chemistry model reproduce the historical trend in aerosol direct radiative
effects over the Northern Hemisphere?",
Atmos. Chem. Phys., 15, 9997-10018, 2015.

 6. Gan, C-M., Pleim, J., Mathur, R., Hogrefe, C., Long, C., Xing, J., Wong, D. C., Gilliam, R., and Wei, C.,
"Assessment of long-term WRF-CMAQ simulations for understanding direct aerosol effects on radiation
"brightening" in the United States",
Atmos. Chem. Phys., 15, 12193-12209, 2015

 5. Gan, C-M., Binkowski, F., Pleim, J., Xing, J., Wong, D. C., Mathur, R., and Gilliam, R.,
"Assessment of the aerosol optics component of the coupled WRF-CMAQ model using CARES field campaign data
and a single column model",
Atmospheric Environment, Volume 115, Pages 670-682, August 2015.

 4. Hogrefe, C., Pouliot, G., Wong, D. C., Torian, A., Roselle, S., Pleim, J., and Mathur, R.,
"Annual application and evaluation of the online coupled WRF-CMAQ system over North America under AQMEII phase 2",
Atmospheric Environment, Volume 115, Pages 1-756, August 2015.

 3. Xing, J., Mathur, R., Pleim, J., Hogrefe, C., Gan, C.-M., Wong, D. C., Wei, C., Gilliam, R., and Pouliot, G.,
"Observations and modeling of air quality trends over 1990-2010 across the Northern Hemisphere: China, the
United States and Europe",
Atmos. Chem. Phys., 15, 2723-2747, 2015.

 2. Wang, J., Wang, S., Jiang, J., Ding, A., Zheng, M., Zhao, B., Wong, D. C., Zhou, W., Zheng, G., Wang, L.,
Pleim, J. E., and Hao, J.,
"Impact of aerosol-meteorology interactions on fine particle pollution during China's severe haze episode
in January 2013",
Environ. Res. Lett. 9, September 2014.

 1. Wong, D. C., Pleim, J., Mathur, R., Binkowski, F., Otte, T., Gilliam, R., Pouliot, G., Xiu, A., and Kang, D.,
"WRF-CMAQ two-way coupled system with aerosol feedback: software development and preliminary results",
Geosci. Model Dev., 5, 299-312, 2012.
@davegill davegill requested review from a team as code owners December 19, 2021 21:31
modified:   phys/module_twoway_rrtmg_aero_optical_util.F
@davegill
Copy link
Contributor Author

@dwongepa @weiwangncar @dudhia
David,
This is the new PR (stolen from the original "CMAQ-WRF coupling" #1432).
Please test THIS branch to see if the CMAQ coupling is still working.

@davegill
Copy link
Contributor Author

@weiwangncar @dudhia
Wait for David Wong to verify these results, before wasting time on a review of a big PR.

@davegill
Copy link
Contributor Author

Status of jenkins is all pass. This includes fixing:

  • the previous real*8 build failures
  • the runtime errors with swint_opt=2 (due entirely to the v4.2.2-ish code base)
Please find result of the WRF regression test cases in the attachment. This build is for Commit ID: 3cd5dcb22b0546982aaa6956ff27b07f7f232344, requested by: davegill for PR: https://github.com/scala-computing/WRF/pull/1609. For any query please send e-mail to David Gill.

    Test Type              | Expected  | Received |  Failed
    = = = = = = = = = = = = = = = = = = = = = = = =  = = = =
    Number of Tests        : 23           24
    Number of Builds       : 60           58
    Number of Simulations  : 159           155        0
    Number of Comparisons  : 96           93        0

    Failed Simulations are: 
    None
    Which comparisons are not bit-for-bit: 
    None

@weiwangncar
Copy link
Collaborator

@dwongepa Do you intend this for other users than EPA to use this capability? If so, where can they find instructions to do so? Do you have any document, web page, etc.. that this PR can point the users to?

@dwongepa
Copy link
Contributor

dwongepa commented Dec 23, 2021 via email

@weiwangncar
Copy link
Collaborator

@dwongepa Thanks for the info. I think a general web page where the info on how to use this capability should be added to the final commit message - which would be what we call 'RELEASE NOTE' in the PR. Can you add some info for what you'd like to announce when the code is released under RELEASE NOTE?

@dwongepa
Copy link
Contributor

dwongepa commented Dec 23, 2021 via email

@weiwangncar
Copy link
Collaborator

@dwongepa As part of the approval depends on the information provided, we would appreciate if that information is added to the PR. Thanks.

@dwongepa
Copy link
Contributor

dwongepa commented Dec 27, 2021 via email

@weiwangncar
Copy link
Collaborator

@dwongepa You can provide some general information. We have an entry for RELEASE NOTE in our PR template, can you fill up something there? That will become a permanent record in github repository and anyone who clones our repository will see that message. The RELEASE NOTE should state what the new capability is and how it is activated in the model. You can look at other PRs to see what is needed if it is not clear to you. We generally use that information provided in the RELEASE NOTE entry to do our release, hence that entry is important. Also, since this is a new PR put together by Dave Gill, can you run some tests by checking out this branch and let us know that your supported capabilities work as expected? Thanks.

@dwongepa
Copy link
Contributor

dwongepa commented Dec 27, 2021 via email

@weiwangncar
Copy link
Collaborator

weiwangncar commented Dec 27, 2021

@dwongepa The last item in your PR message is the entry for RELEASE NOTE. If you click on the three dots to the top right of the PR message window, it will give you an 'edit' option. Stroll down to the bottom of the message, you should see the place you can enter information for the release. Thanks.

@davegill
Copy link
Contributor Author

@dwongepa @weiwangncar
David,
As Wei is saying, what we want is part of the permanent record of the PR. For your chicken and egg conundrum, you can include a reference to a page THAT DOES NOT YET EXIST, for example.

@dwongepa
Copy link
Contributor

dwongepa commented Dec 28, 2021 via email

@weiwangncar
Copy link
Collaborator

@dwongepa I guess you cannot edit this PR since you're not the person who created it. If you send us some text, I can help to fill it up.

@dwongepa
Copy link
Contributor

dwongepa commented Dec 28, 2021 via email

rconfig integer met_file_tstep namelist,wrf_cmaq 1 10000

rconfig logical direct_sw_feedback namelist,wrf_cmaq 1 .false.
rconfig logical feedback_restart namelist,wrf_cmaq 1 .false.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@davegill @dwongepa Should these line be added to Registry.EM_COMMON?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@davegill Do you have a preference where these couple of new lines should be placed? It could be placed in the new registry file added: registry.WRF-CMAQ-twoway, could it?

@davegill
Copy link
Contributor Author

@dwongepa @weiwangncar
David,
I have added you as a collaborator. Please feel free to add the missing lines with a PR.
I was "out"most of yesterday getting a booster, and then today with the after effects of the booster. I am doing quite well now.

@dwongepa
Copy link
Contributor

dwongepa commented Dec 29, 2021 via email

@dwongepa
Copy link
Contributor

dwongepa commented Dec 29, 2021 via email

@dwongepa
Copy link
Contributor

dwongepa commented Jan 25, 2022 via email

@weiwangncar
Copy link
Collaborator

@dwongepa Is the web page listed in the RELEASE NOTE still the right page to point to?

@dwongepa
Copy link
Contributor

dwongepa commented Jan 25, 2022 via email

@davegill
Copy link
Contributor Author

@dudhia

could you commit this one separately

Jimy,
Reasonable request

@davegill
Copy link
Contributor Author

@dudhia @dwongepa

check if this module [twoway_data_module] exists

Jimy,
Other than the use twoway_data_module in solve_em.F (inside a CMAQ ifdef), this module is no where else in the WRF code: fortran files, dependency files, Makefiles. I am wondering if this is something that the procedure for CMAQ coupling handles, such as copies in a file.

@dudhia
Copy link
Collaborator

dudhia commented Jan 25, 2022 via email

@dudhia
Copy link
Collaborator

dudhia commented Jan 25, 2022

Maybe the presence of an interface allows this to compile even without the cmaq_driver code? Does this code compile with just the ifdefs activated?

@davegill
Copy link
Contributor Author

@dudhia

Does this code compile with just the ifdefs activated?

Jimy,
We usually don't care if WRF builds without the other half of the coupled model. I tried, and it fails to build when I turn on the -DWRF_CMAQ=1 cpp flag.

@davegill
Copy link
Contributor Author

@weiwangncar @dudhia
Folks,
I think that this is ready for a review. I still have the ISNAN mods included, but based on our experience today, github will recognize that they are identical to what is in the develop branch.

Wei has asked to be the one to merge the code, so that she can tidy up the official commit message.

@dwongepa
Copy link
Contributor

dwongepa commented Jan 26, 2022 via email

@dudhia
Copy link
Collaborator

dudhia commented Jan 26, 2022 via email

@dwongepa
Copy link
Contributor

dwongepa commented Jan 26, 2022 via email

@weiwangncar
Copy link
Collaborator

@dwongepa Does using wrf_cmaq_option = 1 require the access of CMAQ module?
@davegill Should add this to README.namelist?

@weiwangncar
Copy link
Collaborator

@dwongepa @dudhia Please use github to reply if you can. Thanks.

@dudhia
Copy link
Collaborator

dudhia commented Jan 26, 2022 via email

@davegill
Copy link
Contributor Author

@dwongepa @weiwangncar @dudhia
Folks,
A build issue has been raised:

If you just set WRF_CMAQ to 1 only (as Dave has tried it) and compile the code, it will fail due to missing the CMAQ portion of the code.

Jimy proposed a solution:

A "stub" driver routine could be created to enable a compilation test.

An alternative to consider is a trap in the build system that detects if the cpp flag WRF_CMAQ==1 has been set AND an a priori known file does NOT exist. Either this or Jimy's idea could be added over the next few weeks.

@dudhia
Copy link
Collaborator

dudhia commented Jan 26, 2022 via email

@dudhia
Copy link
Collaborator

dudhia commented Jan 26, 2022

Modified file list needs to be updated to remove one of the new files

@dwongepa
Copy link
Contributor

@weiwangncar a WRF-CMAQ coupled model user will be instructed to include the following block in the namelist:
&wrf_cmaq
wrf_cmaq_option = $wrf_cmaq_option,
wrf_cmaq_freq = $wrf_cmaq_freq,
met_file_tstep = $met_file_tstep,
direct_sw_feedback = $direct_sw_feedback,
feedback_restart = $feedback_restart,
/
For the regular WRF user, it is not required to have it since the default value for each variable is set. The default value for wrf_cmaq_option is 0, i.e. run WRF only (does not require to bring in CMAQ code). If it is set to 1 (part of the coupled model setup), definitely it will require and use the CMAQ code. Hence with this rationale, I don't think we need to mention this namelist block to regular WRF users, however, it will be nice to let them know the existence of WRF-CMAQ coupled model so they can have one more choice for coupled model.

@dwongepa
Copy link
Contributor

@dudhia @davegill I think to setup a trap is a simple and effective way and no compilation is required. I will go ahead to test that idea.

@dwongepa
Copy link
Contributor

@dudhia Since WRF-CMAQ construct has not been a part of WRF package, we have not draft any specific instruction to handle it on the EPA CMAQ web site. It is a chicken-and-egg question. Right now we have a different procedure to guide user to construct WRF-CMAQ coupled model.

@weiwangncar
Copy link
Collaborator

@dwongepa Would you add more or modify the instructions after we approve and merge your PR to the repository?

@dwongepa
Copy link
Contributor

@davegill @dudhia I have modified Config.pl to block a user accidental set environment variable WRF_CMAQ to 1 without CMAQ source code. I have tested and it works. Should I push it up?

@davegill
Copy link
Contributor Author

@weiwangncar

Would you add more or modify the instructions after we approve and merge your PR to the repository?

I thought that the original difficulty was a chicken and egg issue. The EPA cannot provide info on something that does not yet exist. It seems that we get this code into WRF, with a link to an EPA page. Then the EPA can fill out the info on that page. We have about 6 weeks before any official release, which should be plenty of time to fill in web page docs.

@davegill
Copy link
Contributor Author

@dwongepa

I have modified Config.pl to block a user accidental set environment variable WRF_CMAQ to 1 without CMAQ source code. I have tested and it works. Should I push it up?

No. Let us commit the code. You can issue a bug fix to that later. It will be from a different base branch, though.

@dwongepa
Copy link
Contributor

@weiwangncar @davegill Definitely we will modify the WRF-CMAQ coupled model construction procedure ASAP. If you can provide a release date, we can update the web site on the same day.

@weiwangncar weiwangncar merged commit aa9f3b6 into wrf-model:develop Jan 26, 2022
vlakshmanan-scala pushed a commit to scala-computing/WRF that referenced this pull request Apr 4, 2024
* TYPE: new feature

KEYWORDS: CMAQ, WRF-CMAQ coupled model

SOURCE: David Wong, US EPA (wong.david-c@epa.gov)

DESCRIPTION OF CHANGES:
These modifications are for facilitating constructing a WRF-CMAQ coupled model without compromising creating a regular WRF executable. The WRF-CMAQ only interacts with the RRTMG SW option. The aerosol information from CMAQ (a chemistry transport model) will affect the aerosol optical depth,aerosol single scattering albedo and aerosol asymmetry parameter calculations within RRTMG model.

Problem:
A WRF-CMAQ coupled model was developed in US EPA in around 2008. The work has been published in a journal article (Wong et al. 2012). Since then, the coupled model has been applied to various studies and region to validate the effectiveness of the model. Throughout the WRF released cycle, the coupled has been updated manually into a subset of WRF version. The update process was tedious and error prone. With this PR, the manual update process can be eliminated.

Wong, D. C., Pleim, J., Mathur, R., Binkowski, F., Otte, T., Gilliam, R., Pouliot, G., Xiu, A., and Kang, D.,
"WRF-CMAQ two-way coupled system with aerosol feedback: software development and preliminary results", Geosci. Model Dev., 5, 299-312, 2012.

LIST OF MODIFIED FILES:
modified: Makefile
new file: Registry/registry.CMAQ
modified: Registry/registry.em_shared_collection
modified: arch/Config.pl
modified: clean
modified: configure
modified: dyn_em/module_first_rk_step_part1.F
modified: dyn_em/solve_em.F
modified: main/Makefile
modified: main/depend.common
modified: phys/Makefile
new file: phys/complex_number_module.F
modified: phys/module_ra_rrtmg_sw.F
new file: phys/module_ra_rrtmg_aero_optical_util_cmaq.F
modified: phys/module_radiation_driver.F
modified: share/module_check_a_mundo.F
modified: wrftladj/module_first_rk_step_part1_ad.F
modified: wrftladj/module_first_rk_step_part1_tl.F
modified: wrftladj/solve_em_ad.F
modified: wrftladj/solve_em_tl.F

TESTS CONDUCTED:

1. WRF model was built before and after adding the code from this PR, and the results from the WRF model are identical.

2. The coupling model code was compiled by activating environment variable WRF_CMAQ and IOAPI:

     setenv WRF_CMAQ 1
     setenv IOAPI /home/wdx/lib/x86_64/ifc-18.0/ioapi-3.2-2020220
     (Note: ioapi_3.2-2020220 is another third-party library)

3. Normally compiled WRF code and coupled code were tested with two days of simulation, 2016/7/1 - 2016/7/2, verified by a bit-by-bit comparison of every variable whenever is appropriate.

4. All Jenkins tests are passing.

RELEASE NOTE: The capability to couple WRF and CMAQ (an air quality model developed at the US EPA) is added. If one works with CMAQ, one can choose to compile WRF-CMAQ as a coupled model. As a coupled model, a user can run the WRF-CMAQ in an one-way fashion, i.e. meteorological information will be transferred to CMAQ directly to drive its calculation, or in a two-way fashion, i.e. the aerosol information from CMAQ will be fed back to the RRTMG radiation physics in WRF. For users who want to construct WRF-CMAQ coupled model with WRF version 4.4+ and CMAQ version after 4.3.3, please refer to https://www.epa.gov/cmaq/wrf-cmaq-model for latest instruction on how to use this capability. 

Co-authored-by: dwongepa <wong.david-c@epa.gov>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants