Skip to content

Commit

Permalink
CMAQ-WRF coupling: cherry pick (#1609)
Browse files Browse the repository at this point in the history 
* TYPE: new feature

KEYWORDS: CMAQ, WRF-CMAQ coupled model

SOURCE: David Wong, US EPA (wong.david-c@epa.gov)

DESCRIPTION OF CHANGES:
These modifications are for facilitating constructing a WRF-CMAQ coupled model without compromising creating a regular WRF executable. The WRF-CMAQ only interacts with the RRTMG SW option. The aerosol information from CMAQ (a chemistry transport model) will affect the aerosol optical depth,aerosol single scattering albedo and aerosol asymmetry parameter calculations within RRTMG model.

Problem:
A WRF-CMAQ coupled model was developed in US EPA in around 2008. The work has been published in a journal article (Wong et al. 2012). Since then, the coupled model has been applied to various studies and region to validate the effectiveness of the model. Throughout the WRF released cycle, the coupled has been updated manually into a subset of WRF version. The update process was tedious and error prone. With this PR, the manual update process can be eliminated.

Wong, D. C., Pleim, J., Mathur, R., Binkowski, F., Otte, T., Gilliam, R., Pouliot, G., Xiu, A., and Kang, D.,
"WRF-CMAQ two-way coupled system with aerosol feedback: software development and preliminary results", Geosci. Model Dev., 5, 299-312, 2012.

LIST OF MODIFIED FILES:
modified: Makefile
new file: Registry/registry.CMAQ
modified: Registry/registry.em_shared_collection
modified: arch/Config.pl
modified: clean
modified: configure
modified: dyn_em/module_first_rk_step_part1.F
modified: dyn_em/solve_em.F
modified: main/Makefile
modified: main/depend.common
modified: phys/Makefile
new file: phys/complex_number_module.F
modified: phys/module_ra_rrtmg_sw.F
new file: phys/module_ra_rrtmg_aero_optical_util_cmaq.F
modified: phys/module_radiation_driver.F
modified: share/module_check_a_mundo.F
modified: wrftladj/module_first_rk_step_part1_ad.F
modified: wrftladj/module_first_rk_step_part1_tl.F
modified: wrftladj/solve_em_ad.F
modified: wrftladj/solve_em_tl.F

TESTS CONDUCTED:

1. WRF model was built before and after adding the code from this PR, and the results from the WRF model are identical.

2. The coupling model code was compiled by activating environment variable WRF_CMAQ and IOAPI:

     setenv WRF_CMAQ 1
     setenv IOAPI /home/wdx/lib/x86_64/ifc-18.0/ioapi-3.2-2020220
     (Note: ioapi_3.2-2020220 is another third-party library)

3. Normally compiled WRF code and coupled code were tested with two days of simulation, 2016/7/1 - 2016/7/2, verified by a bit-by-bit comparison of every variable whenever is appropriate.

4. All Jenkins tests are passing.

RELEASE NOTE: The capability to couple WRF and CMAQ (an air quality model developed at the US EPA) is added. If one works with CMAQ, one can choose to compile WRF-CMAQ as a coupled model. As a coupled model, a user can run the WRF-CMAQ in an one-way fashion, i.e. meteorological information will be transferred to CMAQ directly to drive its calculation, or in a two-way fashion, i.e. the aerosol information from CMAQ will be fed back to the RRTMG radiation physics in WRF. For users who want to construct WRF-CMAQ coupled model with WRF version 4.4+ and CMAQ version after 4.3.3, please refer to https://www.epa.gov/cmaq/wrf-cmaq-model for latest instruction on how to use this capability. 

Co-authored-by: dwongepa <wong.david-c@epa.gov>
  • Loading branch information
@davegill @dwongepa
davegill and dwongepa authored Jan 26, 2022
1 parent a6d7292 commit aa9f3b6
Show file tree
Hide file tree
Showing 20 changed files with 3,941 additions and 22 deletions.
4 changes: 4 additions & 0 deletions Makefile
View file
Original file line number Diff line number Diff line change
Expand Up @@ -1010,6 +1010,10 @@ physics :
@ echo '--------------------------------------'
if [ $(WRF_CHEM) -eq 0 ] ; then \
( cd phys ; $(MAKE) CF2=" " ) ; \
if [ $(WRF_CMAQ) -eq 1 ] ; then \
@ echo '----------- make cmaq ----------------' ; \
( rm -f main/libcmaqlib.a; cd cmaq ; $(MAKE) -f Makefile.twoway ) ; \
fi \
else \
( cd phys ; $(MAKE) CF2="$(CHEM_FILES2)" ) ; \
fi
Expand Down
59 changes: 59 additions & 0 deletions Registry/registry.CMAQ
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,59 @@
# WRF-CMAQ coupled model
rconfig integer wrf_cmaq_option namelist,wrf_cmaq 1 0
rconfig integer wrf_cmaq_freq namelist,wrf_cmaq 1 1
rconfig integer met_file_tstep namelist,wrf_cmaq 1 10000

rconfig logical direct_sw_feedback namelist,wrf_cmaq 1 .false.
rconfig logical feedback_restart namelist,wrf_cmaq 1 .false.

#state real mass_ws_i ikj twoway_feedback_data 1 - r "mass_ws_i" "Water soluble i mode" "ug/m**3"
#state real mass_ws_j ikj twoway_feedback_data 1 - r "mass_ws_j" "Water soluble j mode" "ug/m**3"
#state real mass_ws_k ikj twoway_feedback_data 1 - r "mass_ws_k" "Water soluble k mode" "ug/m**3"
#state real mass_in_i ikj twoway_feedback_data 1 - r "mass_in_i" "Water insoluble i mode" "ug/m**3"
#state real mass_in_j ikj twoway_feedback_data 1 - r "mass_in_j" "Water insoluble j mode" "ug/m**3"
#state real mass_in_k ikj twoway_feedback_data 1 - r "mass_in_k" "Water insoluble k mode" "ug/m**3"
#state real mass_ec_i ikj twoway_feedback_data 1 - r "mass_ec_i" "Elemental carbon i mode" "ug/m**3"
#state real mass_ec_j ikj twoway_feedback_data 1 - r "mass_ec_j" "Elemental carbon j mode" "ug/m**3"
#state real mass_ec_k ikj twoway_feedback_data 1 - r "mass_ec_k" "Elemental carbon k mode" "ug/m**3"
#state real mass_ss_i ikj twoway_feedback_data 1 - r "mass_ss_i" "Seasalt i mode" "ug/m**3"
#state real mass_ss_j ikj twoway_feedback_data 1 - r "mass_ss_j" "Seasalt j mode" "ug/m**3"
#state real mass_ss_k ikj twoway_feedback_data 1 - r "mass_ss_k" "Seasalt k mode" "ug/m**3"
#state real mass_h2o_i ikj twoway_feedback_data 1 - r "mass_h2o_i" "water i mode" "ug/m**3"
#state real mass_h2o_j ikj twoway_feedback_data 1 - r "mass_h2o_j" "water j mode" "ug/m**3"
#state real mass_h2o_k ikj twoway_feedback_data 1 - r "mass_h2o_k" "water k mode" "ug/m**3"
#state real dgn_i ikj twoway_feedback_data 1 - r "dgn_i" "diameter i mode" "m"
#state real dgn_j ikj twoway_feedback_data 1 - r "dgn_j" "diameter j mode" "m"
#state real dgn_k ikj twoway_feedback_data 1 - r "dgn_k" "diameter k mode" "m"
#state real sig_i ikj twoway_feedback_data 1 - r "sig_i" "standard deviations i mode" ""
#state real sig_j ikj twoway_feedback_data 1 - r "sig_j" "standard deviations j mode" ""
#state real sig_k ikj twoway_feedback_data 1 - r "sig_k" "standard deviations k mode" ""

#state real prev_rainnc ij twoway_feedback_data 1 - r "prev_rainnc" "previous accumlated rainnc" "mm"
#state real prev_rainc ij twoway_feedback_data 1 - r "prev_rainc" "previous accumlated rainc" "mm"

#state real ozone ikj twoway_feedback_data 1 - r "OZONE" "ozone value" ""

#state real sw_gtauxar_01 ikj misc 1 - hr "SW_GTAUXAR_01" "SW Optical depth of lamda = 0.388 um" ""
#state real sw_gtauxar_02 ikj misc 1 - hr "SW_GTAUXAR_02" "SW Optical depth of lamda = 0.533 um" ""
#state real sw_gtauxar_03 ikj misc 1 - hr "SW_GTAUXAR_03" "SW Optical depth of lamda = 0.702 um" ""
#state real sw_gtauxar_04 ikj misc 1 - hr "SW_GTAUXAR_04" "SW Optical depth of lamda = 1.010 um" ""
#state real sw_gtauxar_05 ikj misc 1 - hr "SW_GTAUXAR_05" "SW Optical depth of lamda = 1.271 um" ""
#state real sw_ttauxar_01 ij misc 1 - hr "SW_TTAUXAR_01" "Optical depth sum of SW_GTAUXAR_01" ""
#state real sw_ttauxar_02 ij misc 1 - hr "SW_TTAUXAR_02" "Optical depth sum of SW_GTAUXAR_02" ""
#state real sw_ttauxar_03 ij misc 1 - hr "SW_TTAUXAR_03" "Optical depth sum of SW_GTAUXAR_03" ""
#state real sw_ttauxar_04 ij misc 1 - hr "SW_TTAUXAR_04" "Optical depth sum of SW_GTAUXAR_04" ""
#state real sw_ttauxar_05 ij misc 1 - hr "SW_TTAUXAR_05" "Optical depth sum of SW_GTAUXAR_05" ""
#state real sw_asy_fac_01 ikj misc 1 - hr "SW_ASY_FAC_01" "Corresponding Asymmetry Factor of SW_GTAUXAR_01" ""
#state real sw_asy_fac_02 ikj misc 1 - hr "SW_ASY_FAC_02" "Corresponding Asymmetry Factor of SW_GTAUXAR_02" ""
#state real sw_asy_fac_03 ikj misc 1 - hr "SW_ASY_FAC_03" "Corresponding Asymmetry Factor of SW_GTAUXAR_03" ""
#state real sw_asy_fac_04 ikj misc 1 - hr "SW_ASY_FAC_04" "Corresponding Asymmetry Factor of SW_GTAUXAR_04" ""
#state real sw_asy_fac_05 ikj misc 1 - hr "SW_ASY_FAC_05" "Corresponding Asymmetry Factor of SW_GTAUXAR_05" ""
#state real sw_ssa_01 ikj misc 1 - hr "SW_SSA_01" "Corresponding Single scattering Albedo of SW_GTAUXAR_01" ""
#state real sw_ssa_02 ikj misc 1 - hr "SW_SSA_02" "Corresponding Single scattering Albedo of SW_GTAUXAR_02" ""
#state real sw_ssa_03 ikj misc 1 - hr "SW_SSA_03" "Corresponding Single scattering Albedo of SW_GTAUXAR_03" ""
#state real sw_ssa_04 ikj misc 1 - hr "SW_SSA_04" "Corresponding Single scattering Albedo of SW_GTAUXAR_04" ""
#state real sw_ssa_05 ikj misc 1 - hr "SW_SSA_05" "Corresponding Single scattering Albedo of SW_GTAUXAR_05" ""

#state real sw_zbbcddir ij misc 1 - hr "SW_ZBBCDDIR" "Clear sky downward direct shortwave flux" "W m-2"
#state real sw_dirdflux ij misc 1 - hr "SW_DIRDFLUX" "Direct downward shortwave surface flux" "W m-2"
#state real sw_difdflux ij misc 1 - hr "SW_DIFDFLUX" "Diffuse downward shortwave surface flux" "W m-2"
1 change: 1 addition & 0 deletions Registry/registry.em_shared_collection
View file
Original file line number Diff line number Diff line change
Expand Up @@ -28,3 +28,4 @@ include registry.new3d_wif
include registry.trad_fields
include registry.solar_fields
include registry.diags
include registry.CMAQ
154 changes: 153 additions & 1 deletion arch/Config.pl
View file
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,9 @@
# Be sure to run as ./configure (to avoid getting a system configure command by mistake)
#

use Cwd qw(getcwd);
$wrf_cmaq_option = $ENV{'WRF_CMAQ'}; # determine building WRF-CMAQ coupled model or not

select((select(STDOUT), $|=1)[0]);
$sw_perl_path = perl ;
$sw_netcdf_path = "" ;
Expand Down Expand Up @@ -449,11 +452,155 @@
$latchon = 0 ;
while ( <CONFIGURE_DEFAULTS> )
{
if ( $_ =~ /ifort compiler/ )
{ $lioapi_temp = 'Linux2_x86_64ifort';
}
elsif ( $_ =~ /PGI compiler/ )
{ $lioapi_temp = 'Linux2_x86_64pg';
}
elsif ( $_ =~ /gfortran compiler/ )
{ $lioapi_temp = 'Linux2_x86_64gfort';
}

if ( substr( $_, 0, 5 ) eq "#ARCH" && $latchon == 1 )
{
close CONFIGURE_DEFAULTS ;
if ( $sw_opt_level eq "-f" ) {
open CONFIGURE_DEFAULTS, "cat ./arch/postamble ./arch/noopt_exceptions_f |" or die "horribly" ;

# determine whether variable EM_MODULE_DIR contains -I../cmaq or not
my $file = "Makefile";
open(FH, $file) or die("File $file not found");

$lib_path_wo_cmaq = 1;
while ( my $String = <FH> )
{ if($String =~ /-I..\/dyn_em -I..\/cmaq/)
{ $lib_path_wo_cmaq = 0;
}
}
close (FH);

# determine whether declarations in Registry/registry.CMAQ is commented out or not
my $file = "Registry/registry.CMAQ";
open (FH, $file) or die("File $file not found");

$registry_wo_cmaq = 0;
while ( my $String = <FH> )
{ if($String =~ /#state/)
{ $registry_wo_cmaq = 1;
}
}
close (FH);

# determine whether express #NOWIN LIB_BUNDLED contains $(IOAPI_LIB) or not
my $file = "arch/preamble";
open (FH, $file) or die("File $file not found");

$bundle_wo_ioapi = 1;
while ( my $String = <FH> )
{ if ( $String =~ /IOAPI_LIB/ )
{ $bundle_wo_ioapi = 0;
}
}
close (FH);

if ( $wrf_cmaq_option eq 1 ) # build WRF-CMAQ coupled model
{ if ( $lib_path_wo_cmaq == 1 )
{ open (FILE, "<Makefile") || die "File not found";
my @lines = <FILE>;
close (FILE);

foreach ( @lines )
{ $_ =~ s/ -I..\/dyn_em/ -I..\/dyn_em -I..\/cmaq/g;
}

open (FILE, ">Makefile") || die "File not found";
print FILE @lines;
close (FILE);
}

if ( $registry_wo_cmaq == 1 )
{ open (FILE, "<Registry/registry.CMAQ") || die "File not found";
my @lines = <FILE>;
close (FILE);

foreach (@lines)
{ $_ =~ s/#state/state/g;
}

open (FILE, ">Registry/registry.CMAQ") || die "File not found";
print FILE @lines;
close (FILE);
}

if ( $bundle_wo_ioapi == 1 )
{ open (FILE, "<arch/preamble") || die "File not found";
my @lines = <FILE>;
close (FILE);

foreach (@lines)
{ $_ =~ s/#NOWIN LIB_BUNDLED = \\/#NOWIN LIB_BUNDLED = \$(IOAPI_LIB) \\/g;
}

open (FILE, ">arch/preamble") || die "File not found";
print FILE @lines;
close (FILE);
}

open (FH, '>', wrf_cmaq_path) or die $! ;
$ioapi_path = $ENV{'IOAPI'} ;
print FH "CMAQLIB = libcmaqlib.a \n" ;
print FH "IOAPI = $ioapi_path\n" ;
print FH "LIOAPI = $lioapi\n" ;
print FH "IOAPI_LIB = -L$ioapi_path/$lioapi -lioapi \n" ;
close (FH) ;
open CONFIGURE_DEFAULTS, "cat wrf_cmaq_path ./arch/postamble ./arch/noopt_exceptions_f |" or die "horribly" ;
}
else
{ if ( $lib_path_wo_cmaq == 0 )
{ open (FILE, "<Makefile") || die "File not found";
my @lines = <FILE>;
close (FILE);

foreach (@lines)
{ $_ =~ s/ -I..\/cmaq//g;
}

open (FILE, ">Makefile") || die "File not found";
print FILE @lines;
close (FILE);

}

if ( $registry_wo_cmaq == 0 )
{ open (FILE, "<Registry/registry.CMAQ") || die "File not found";
my @lines = <FILE>;
close (FILE);

foreach(@lines)
{ $_ =~ s/state/#state/g;
}

open (FILE, ">Registry/registry.CMAQ") || die "File not found";
print FILE @lines;
close (FILE);
}

if ( $bundle_wo_ioapi == 0 )
{ open (FILE, "<arch/preamble") || die "File not found";
my @lines = <FILE>;
close (FILE);

foreach (@lines)
{ $_ =~ s/ \$\(IOAPI_LIB\)//g;
}

open (FILE, ">arch/preamble") || die "File not found";
print FILE @lines;
close (FILE);
}

open CONFIGURE_DEFAULTS, "cat ./arch/postamble ./arch/noopt_exceptions_f |" or die "horribly" ;
}
} else {
open CONFIGURE_DEFAULTS, "cat ./arch/postamble ./arch/noopt_exceptions |" or die "horribly" ;
}
Expand Down Expand Up @@ -720,6 +867,7 @@
printf "Compile for nesting? (1=basic, 2=preset moves, 3=vortex following) [default 1]: " ;
}
$response = <STDIN> ;
$lioapi = $lioapi_temp;
}
printf "\n" ;
lc $response ;
Expand Down Expand Up @@ -804,6 +952,10 @@
close POSTAMBLE ;
close ARCH_NOOPT_EXCEPTIONS ;

if ( $wrf_cmaq_option eq 1 )
{ unlink "wrf_cmaq_path";
}

open CONFIGURE_WRF, "> configure.wrf" or die "cannot append configure.wrf" ;
open ARCH_PREAMBLE, "< arch/preamble" or die "cannot open arch/preamble" ;
my @preamble;
Expand Down
8 changes: 6 additions & 2 deletions clean
View file
Original file line number Diff line number Diff line change
Expand Up @@ -3,11 +3,15 @@
set nonomatch


foreach dir ( frame chem share dyn_em phys main tools wrftladj )
foreach dir ( frame chem share dyn_em phys cmaq main tools wrftladj )
if ( -d $dir ) then
( cd $dir ; echo $dir ; /bin/rm -f core wrf *.f90 *.exe *.kmo *.mod *.o *.obj *.inc *.F90 *.a \
if ( $dir == cmaq ) then
( cd $dir ; echo $dir ; /bin/rm -f *.o *.mod ) >& /dev/null
else
( cd $dir ; echo $dir ; /bin/rm -f core wrf *.f90 *.exe *.kmo *.mod *.o *.obj *.inc *.F90 *.a \
db_* Warnings module_state_description.F module_dm.F gmeta \
wrfdata whatiread rsl.* show_domain* ) >& /dev/null
endif
endif
end

Expand Down
7 changes: 7 additions & 0 deletions configure
View file
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,7 @@ while [ $# -ge 1 ]; do
-r8) rword="-r8" ;;
-time) shift ; FORTRAN_COMPILER_TIMER=$1 ;;
chem) WRF_CHEM=1 ;;
cmaq) WRF_CMAQ=1 ;;
kpp) WRF_KPP=1 ;;
radardfi) WRF_DFI_RADAR=1 ;;
wrfda) wrf_core=DA_CORE ;;
Expand Down Expand Up @@ -499,6 +500,12 @@ if [ -n "$WRF_DFI_RADAR" ] ; then
compileflags="${compileflags}!-DWRF_DFI_RADAR=1"
fi
fi

if [ -n "$WRF_CMAQ" ] ; then
echo building WRF with CMAQ option
compileflags="${compileflags}!-DWRF_CMAQ"
fi

if [ -n "$WRF_CHEM" ] ; then
if [ $WRF_CHEM = 1 ] ; then
echo building WRF with chemistry option
Expand Down
Loading

0 comments on commit aa9f3b6

Please sign in to comment.