New module firebrand_spotting for WRF-Fire

[mefrediani]

TYPE: new feature

KEYWORDS: fire firebrand spotting Lagrangian transport passive advection burnout

SOURCE: Maria Frediani, Tim W. Juliano (NCAR-RAL)

DESCRIPTION OF CHANGES:
The new module firebrand_spotting for WRF-Fire is a passive Lagrangian transport parameterization to
advect firebrands. The Firebrand Spotting parameterization was developed for the WRF-Fire component of the
WRF model versions starting at 4.0.1. The parameterization couples to WRF-Fire and uses a Lagrangian
particle transport framework to advect firebrands in the innermost nest of the domain.

The parameterization runs in the atmospheric model inner domain and does not modify model variables (no
feedback to WRF or WRF-Fire). The code comprises two independent modules, one with the physical
processes and another with the necessary MPI wrapping routines that were not yet part of the WRF source
code. The motivation to separate the MPI routines in an independent module was to enable them to be used in
other model parameterizations through a USE statement without importing the firebrand spotting component.
The Firebrand Spotting variables are part of Registry.fire and the subroutine is called from start_em.F and
solve_em.F, after all the physics parameterizations and relevant halos are completed.

When fires are active, the parameterization identifies areas at risk of fire spotting by modeling transport
and physical processes of individual firebrands. Firebrands are released at multiple heights from grid points
along the fire front with high fire rate-of-spread and denser fuel loads. Particles are transported with the
atmospheric flow and consumed by combustion. Firebrands may burnout entirely or land, once they descend
below a given height threshold. Particles that land before complete burnout are accumulated in a 2-D field
during regular intervals.

The likelihood of new fire ignitions due to spotting is computed using the ratio of landed firebrands per grid
point to the total number of landed particles within the corresponding time interval between model outputs.
The ratios are then scaled by a function of fuel load and moisture content at the corresponding grid points.

LIST OF MODIFIED FILES:
M Registry/Registry.EM_COMMON
M Registry/registry.fire
M dyn_em/depend.dyn_em
M dyn_em/solve_em.F
M dyn_em/start_em.F
M main/depend.common
M phys/Makefile
A phys/module_firebrand_spotting.F
A phys/module_firebrand_spotting_mpi.F

TESTS CONDUCTED:

1. The module was designed for high-resolution simulations and tested using large-eddy simulation (LES) in
   the inner nest. Simulations for various case studies in Colorado have been done.
2. All tests have passed (latest commit: afc9142, after switching pbl from MYNN to YSU). There were no
   differences among the serial and MPI builds with 1 and 12 processors. The tests were all done from a restart
   file. The time step for these runs was 5s and the outputs were compared after 20s, 40s, and 1 min.
3. Jenkins tests are all passing.

RELEASE NOTE: A new module to parameterize firebrand spotting for WRF-Fire is added. This is a passive Lagrangian transport scheme to transport and burnout firebrands generated at the fire front. The scheme is activated when ifire == 2 by setting the namelist option fs_firebrand_gen_lim to an integer greater than zero (default is 0, i.e. scheme is off). It runs with dmpar and serially compiled code and in the inner nest (grid_id == max_dom). It was designed and tested using a mesoscale to LES domain configuration.

[dudhia]

@pedro-jm can you give your opinion on this PR too?
Why is there no fire module code in this PR? Do the firebrands not feed back to the fire?

[pedro-jm]

@dudhia

The code releases and moves the firebrands with the atmospheric dynamics simulating their combustion too. If the firebrands end up in the ground the code tracks where do they land. So the user can identify locations that could be affected by fire spotting. In the future this could be extended to ignite fires at certain locations. But that is not currently done so that is why there is not feedback to the fire code.

The code is well organized in modules and the bulk of the development is in 2 new modules which is desirable for maintenance.

[dudhia]

Thanks for the clarification, Pedro.

…

On Wed, Nov 17, 2021 at 10:58 AM pedro-jm ***@***.***> wrote: @dudhia <https://github.com/dudhia> The code releases and moves the firebrands with the atmospheric dynamics simulating their combustion too. If the firebrands end up in the ground the code tracks where do they land. So the user can identify locations that could be affected by fire spotting. In the future this could be extended to ignite fires at certain locations. But that is not currently done so that is why there is not feedback to the fire code. The code is well organized in modules and the bulk of the development is in 2 new modules which is desirable for maintenance. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1540 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AEIZ77G2LQUKDZFCHTWXPFTUMPUKTANCNFSM5BUXUTUA> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

[davegill]

@mefrediani @twjuliano @pedro @dudhia
Folks,
Every serial and OpenMP build is getting a compile failure due to some MPI issues. The WRF model does not usually permit communications in the physics directory, but there are some precedents for it. There are bare MPI calls, and we have usually tried to hide those completely from the user (such as you have done with the fs_mpi_ functions). First, let's get the code to pass the regression tests, then look at what needs to be restructured.

1. We need at least most of module_firebrand_spotting_mpi.F to be stubbed out if we are NOT building with MPI. You could use a couple ifdefs in the module_firebrand_spotting_mpi.F file to wipe out most of the code. Leave the "module" statements, the "contains" statement, and perhaps nothing else. The test in WRF for not doing an MPI build is this ifdef combination:

#if ( defined(DM_PARALLEL)  &&   ! defined(STUBMPI) )
   some mpi command, such as any of your fs_pi_ functions.
#endif

2. Anywhere you have the function calls fs_mpi_ in the module_firebrand_spotting.F need to be enclosed in the MPI ifdef that is shown above. Not always, but usually, those parts of the code are not required when there is no communication.

To see if your modified code is working, try to build the code serially (for example, with GNU it would be option 32).

[mefrediani]

Hi Dave,
Thanks for your input. I'm gonna start working on the serial compilation and let you know when it's done.

[davegill]

@mefrediani

Thanks for your input. I'm gonna start working on the serial compilation and let you know when it's done.

Keep us in the loop as this is a bit of an infrastructure issue, we do not want you to fix this in the wrong way. Let me know if a google meet would help out (gill@ucar.edu), and we can set something up.

[mefrediani]

Hi Dave,
The modified code is compiling with GNU and Intel (both serial and dmpar). I included a summary of the modifications I made below. Just let me know if you think they will not work for whatever reason. I'll wait for instructions on what to do next.
Thanks!

I enclosed the following statements within ifdef blocks:

1. USE MPI and other USE statements associated with MPI routines
2. IF or DO blocks exclusively containing calls to MPI functions. (I thought it didn't make sense to enclose only the line with the function call and leave the loop doing nothing)
3. The code section that receives particles from an adjacent tile (this is irrelevant in a serial compilation so I used one ifdef on the entire section)
4. I added one #else statement to initialize two variables:

#if ( defined(DM_PARALLEL)  &&   ! defined(STUBMPI) )                                                                                                                                                                                    
       ...
        IF ( wrf_dm_on_monitor() ) THEN
            IamMaster = .TRUE.
            MasterId = my_id
        ENDIF
#else
        IamMaster = .TRUE.  ! initialized with FALSE. It should be TRUE in serial compilation    
        ntiles = 1                                                                                                                                        
#endif

5. And on module_firebrand_spotting_mpi.F I also had to enclose all MPI routines under CONTAINS (and their corresponding PUBLIC declarations) because of variables such as MPI_FLOAT. As you suggested, pretty much everything has been hidden.

[davegill]

@mefrediani
Maria,
This looks like a large amount of work so far!

Would you do a SHORT test to verify that you are getting the same answers with the serial build and with MPI.

1. Build with a flag that removes optimization. Typically we do this with "configure -d"
2. Only do about 10-12 time steps of simulation. Do you need a restart file to get everything spun-up? You can use the same restart file, if that is the case, for both MPI and serial.
3. Compare model output that has files that would appear to be different. Usually, that is the standard wrfout* files.
4. To compare two different wrfout* files (that we are assuming should be identical), use the utility WRF/external/io_netcdf/diffwrf. The syntax is diffwrf <wrf file serial> <wrf file mpi>
5. If the files are the same, then you do not get any printout other than some headings of table. If you do have differences, then you will see a list of the fields that are different.

[davegill]

@mefrediani @weiwangncar
Maria,

1. We are mostly OK on this PR. Since this is brand new, we just need to hear about the ability to get bit-wise identical results. We would really like to have this completed by early January. Once we cut over the v4.4 branch from develop, any mods that are not yet on the develop branch do not make the next official release.
2. If there is a publication associated with this PR, that can be listed in the the RELEASE NOTES section. We are pretty informal, so "in press", "accepted", "under review" - we take ANYTHING!

[weiwangncar]

@mefrediani It may also be helpful to add some information about how the capability is activated in the model in the release note section. Thanks.

[mefrediani]

@weiwangncar I have a draft (attached) with a detailed description of the module, namelist parameters, and output variables. I will check with my colleagues if they have any reviewing to suggest and follow up with a final version in the coming days.

In the meantime, it would be helpful if you could send me instructions on how to prepare this document for you, for example, specific section names, tables, etc.

Thank you!

Firebrand Spotting Parameterization - Description.pdf

[weiwangncar]

@mefrediani There are several levels of documentation you can provide. For this PR, you can simply add something like 'this option is activated by namelist X'. If this option can be used with other fire options (or not), it may be useful to state. As for the full documentation, we have a WRF Fire chapter in our User's Guide: https://www2.mmm.ucar.edu/wrf/users/docs/user_guide_v4/v4.3/users_guide_chap-fire.html, and you can see if your new option can fit somewhere in that chapter. We also have a link to WRF Wildland Fire page: https://www2.mmm.ucar.edu/wrf/users/fire/wrf-fire.html, and at the bottom of the page, there is a link to RAL's Fire page: https://ral.ucar.edu/solutions/products/wrf-fire-wildland-fire-modeling. If your work is in someway related to the work with the RAL projects, feel free to add your document there. Let us know where you think the document fits the best.

[davegill]

@weiwangncar @dudhia @mefrediani
Folks,

1. This code is not set up to give bit-wise identical results with a restart.
2. This code is not set up to give bit-wise identical results with differing core counts from MPI.

[weiwangncar]

@davegill Do you mean this code breaks bit-for-bit and restart checks even when the option is not used?

[davegill]

@weiwangncar

Do you mean this code breaks bit-for-bit and restart checks even when the option is not used?

No. This code works fine with the existing regression tests. When using firebrands, THAT code does not give the same MPI values with different core counts, and it does not give the same restart values.

[mefrediani]

@davegill, @weiwangncar, @dudhia I've been investigating the issue with the restart and it looks like the release-v4.3 commit 2adcfda, i.e., without the firebrand module, doesn't run when compiled with -d. I set 12 cpus for this test case, with the namelist set to run with ifire = 0, 2 (i.e. fire model is on in the inner nest), and writing a restart file. A gnu-openmpi build on casper crashed with the following error:

d02 2020-10-20_21:00:00 module_io.F: in wrf_open_for_write_begin, FileName = wrfout_d02_2020-10-20_21:00:00 d02 2020-10-20_21:00:00 calling outsub in open_w_dataset [crhtc38:118794:0:118794] Caught signal 11 (Segmentation fault: Sent by the kernel at address (nil)) /glade/scratch/frediani/WRF-Fire/bittest/builds/casper-WRF-release-gnuopenmpi-d/share/output_wrf.f90: [ output_wrf_() ] ... 1036 newswitch = switch 1037 p => grid%head_statevars%next 1038 DO WHILE ( ASSOCIATED( p ) ) ==> 1039 IF ( p%ProcOrient .NE. 'X' .AND. p%ProcOrient .NE. 'Y' ) THEN 1040 IF ( p%Ndim .EQ. 0 ) THEN 1041 IF ((p%Restart.AND.switch.EQ.restart_only).OR.on_stream(p%streams,newswitch)) THEN 1042 IF ( in_use_for_config(grid%id,TRIM(p%VarName)) ) THEN

Then, after turning Fire off on the namelist (but with some Fire related fields still in the wrfinput files) it crashes about 400 lines later:

d02 2020-10-20_21:00:00 module_io.F: in wrf_open_for_write_begin, FileName = wrfout_d02_2020-10-20_21:00:00 d02 2020-10-20_21:00:00 calling outsub in open_w_dataset [crhtc62:271846:0:271846] Caught signal 11 (Segmentation fault: Sent by the kernel at address (nil)) /glade/scratch/frediani/WRF-Fire/bittest/builds/casper-WRF-release-gnuopenmpi-d/share/output_wrf.f90: [ output_wrf_() ] ... 1454 1455 1456 ==> 1457 DO itrace = PARAM_FIRST_SCALAR , p%num_table(grid%id) 1458 IF ((p%Restart.AND.switch.EQ.restart_only).OR.on_stream(p%streams_table(grid%id,itrace)%stream,newswitch)) THEN 1459 dname = p%dname_table( grid%id, itrace ) 1460 IF (p%Ntl.GT.0.AND.switch.NE.restart_only)dname=dname(1:len(TRIM(dname))-2)

I'm not sure how to debug these possible issues in the firebrand module given that an mpi run with n>1 configured with -d doesn't run in the unmodified code. I wonder if this is an issue with this specific case, with WRF-Fire, or with the Fire-related fields in the wrfinput files which are on the refined fire grid. Please let me know how to proceed. The executables were built here: /glade/scratch/frediani/WRF-Fire/bittest/builds/casper-WRF-release-gnuopenmpi-d/ The test case is here: /glade/scratch/frediani/WRF-Fire/bittest/runs/casper-test-release-gnuopenmpi12-d The various test cases are on the same path, in these folders: casper-test2-release-gnuopenmpi12-d-namelistNoFire casper-test3-release-gnuopenmpi12-d-namelistNoRestart casper-test4-release-gnuopenmpi12-d-namelistNoRestartNoFire

[davegill]

@mefrediani @weiwangncar
Maria,
Well if you are not just a bunch of great news! :)
Thanks for getting us a local dataset to try out, it really helps.
Usually these segfaults with the output are related to packaging of variables.
Dave

[mefrediani]

@weiwangncar Here's the updated documentation for the PR, user's guide, and web pages. I can also share a google doc with you, if that's easier. Just let me know.

WRF documentation - Firebrand Spotting Parameterization - Google Docs.pdf

@mefrediani There are several levels of documentation you can provide. For this PR, you can simply add something like 'this option is activated by namelist X'. If this option can be used with other fire options (or not), it may be useful to state. As for the full documentation, we have a WRF Fire chapter in our User's Guide: https://www2.mmm.ucar.edu/wrf/users/docs/user_guide_v4/v4.3/users_guide_chap-fire.html, and you can see if your new option can fit somewhere in that chapter. We also have a link to WRF Wildland Fire page: https://www2.mmm.ucar.edu/wrf/users/fire/wrf-fire.html, and at the bottom of the page, there is a link to RAL's Fire page: https://ral.ucar.edu/solutions/products/wrf-fire-wildland-fire-modeling. If your work is in someway related to the work with the RAL projects, feel free to add your document there. Let us know where you think the document fits the best.

[weiwangncar]

@mefrediani Thanks for sharing the pdf file. We will add the content to the appropriate places on our end. It may be convenient to add some of this documentation to the RAL Wildfire page, so that you can maintain and update easily. I've added your PR description to your PR message (last section). Please check and see if it is ok with you. Remember this will be a permanent record in git-cloned code for everyone to see.

[davegill]

@weiwangncar @mefrediani
Folks,
Once cheyenne is up next week, we can look a the segfault issue - again likely due to packaging.

[mefrediani]

@davegill @weiwangncar
The issue with the release version is related to packaging of PBL variables while running MYNN in the outer domain and LES in the inner. Thanks for the tip ;)

It seems that the bug fix to this issue (#1596) is not on the develop branch yet. I was able to test it out by adding this line to Registry.EM_COMMON:
package nopblscheme bl_pbl_physics==0 - scalar:qke_adv;state:qke,tke_pbl,sh3d,tsq,qsq,cov,el_pbl
When I built the develop branch with the nopblscheme package, it worked.

I'll try to wrap up firebrand spotting tests using a different PBL scheme. I see that YSU (pbl=1) has no package variables, so it may work without the nopblscheme work around. I'll let you know how it goes.

[mefrediani]

@davegill @weiwangncar @dudhia

All tests have passed (latest commit: afc9142, after switching pbl from MYNN to YSU).

There were no differences among the serial and MPI builds with 1 and 12 processors. The tests were all done from a restart file.

The timestep for these runs was 5s and the outputs were compared after 20s, 40s, and 1 min.
The runs are here: /glade/scratch/frediani/WRF-Fire/bittest/runs/casper-test-firebrands-afc9142f-*
The log from diffwrf is here: /glade/scratch/frediani/WRF-Fire/bittest/runs/diffwrf.log

I hope you and your families are well. Stay safe.

[davegill]

@mefrediani @weiwangncar @dudhia

1. The jenkins tests are all passing (as before).

Please find result of the WRF regression test cases in the attachment. This build is for Commit ID: 6a4678fab1ceff64ed0f4bcee699f84036ed044d, requested by: mefrediani for PR: https://github.com/scala-computing/WRF/pull/1540. For any query please send e-mail to David Gill.

    Test Type              | Expected  | Received |  Failed
    = = = = = = = = = = = = = = = = = = = = = = = =  = = = =
    Number of Tests        : 23           24
    Number of Builds       : 60           58
    Number of Simulations  : 159           155        0
    Number of Comparisons  : 96           93        0

    Failed Simulations are: 
    None
    Which comparisons are not bit-for-bit: 
    None

2. Maria is able to get bit-for-bit results with serial vs MPI (1 vs 12 cores) with her test case.

[davegill]

@weiwangncar @dudhia
This PR is passing the regtests, and Maria has conducted the restart and parallel tests. This is ready for review.