Add a small 3km predefined grid over Indianapolis for testing

[gsketefian]

DESCRIPTION OF CHANGES:

This PR adds a new predefined 3km grid over Indianapolis (grid name is SUBCONUS_Ind_3km). This is a 200 x 200 grid (so much smaller than the 3km CONUS grid, which has 1748 x 1038 points). It is intended for use in the WE2E tests. The PR also adds a single WE2E test for this grid.

TESTS CONDUCTED:

Successfully ran the new WE2E test (SUBCONUS_Ind_3km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v15p2) on Hera.

CONTRIBUTORS (optional):

@mark-a-potts , @JeffBeck-NOAA

[JeffBeck-NOAA]

I'm wondering if we shouldn't initialize this test with HRRR data and use the RRFS_v1beta or FV3_HRRR SDF instead.

[gsketefian]

I could do that, but I figured it's going to happen anyway when many other tests are switched from the RRFS_CONUS_3km grid to this one. Let me know how you want to proceed.

[JeffBeck-NOAA]

It's fine for now! Approving.

[ywangwof]

I am wondering whether 600 km x 600 km domain is too small? The WoFS uses 900 km x 900 km (i.e. 300 x 300 grid points) and makes 6 hour forecasts. Convective systems may not be full developed before it is impacted by the lateral boundary conditions. For WE2E testing purpose, a 200 x 200 grid will not make much difference comparing with a 300 x 300 grid. Then, why not choose a grid that is more physical meaningful? Just my two cents. BTW, I see no significant weather events within the domain with both cases on 20190701 and on 20190615? Thanks.

[gsketefian]

I am wondering whether 600 km x 600 km domain is too small? The WoFS uses 900 km x 900 km (i.e. 300 x 300 grid points) and makes 6 hour forecasts. Convective systems may not be full developed before it is impacted by the lateral boundary conditions. For WE2E testing purpose, a 200 x 200 grid will not make much difference comparing with a 300 x 300 grid. Then, why not choose a grid that is more physical meaningful? Just my two cents. BTW, I see no significant weather events within the domain with both cases on 20190701 and on 20190615? Thanks.

Well the intent of this grid was just to run a test, not ensure scientific integrity. I wanted it to be about the same as the 25 km CONUS grid (which is 202 x 116). A 300x300 grid (which has 2.25 times as many cells) might be too big but I'm ok with it if EPIC testers (@mark-a-potts, @EdwardSnyder-NOAA) are ok with it. I actually already have a 400x400 grid also centered on Indianapolis that I decided not to use to keep the tests smaller.

The 200x200 was also a grid we decided to use for the upcoming second SRW Training, and it had to be small so that students finish their tests in the allotted time (20-30 minutes) on the cloud. But the grid for the training does't have to be the same as the one for testing. Although you should see interesting weather if you start on 2019061518 (that's what I ran) since then there were tornadoes in Indiana (I looked at my vorticity and/.or updraft helicity results (can't remember which) and could see "interesting" weather there). And I did see boundary artifacts.

What we should definitely do is include a WoFS grid (I know it moves around). It can be one that can be shifted around (as we discussed by email) and used in other tests.

[ywangwof]

Thanks for the clarification. I will try to add a WoFS grid and the python tool to move around the grid.

[EdwardSnyder-NOAA]

@gsketefian - It is true that this grid will be used to ensure the model runs, not scientific integrity. What would be the difference in run time between the two grids? I cannot imagine running the test at the 300 x 300 grid would increase the run time too much. Probably on the order of minutes? If that is the case, I'd be fine switching to the 300 x 300 grid.

[christopherwharrop-noaa]

@gsketefian / @EdwardSnyder-NOAA / @ywangwof - For testing purposes, the size isn't the only consideration. Code coverage is also to be considered, as well as situations that present challenges. Code paths are determined by configuration choices, but also by weather conditions being simulated. Terrain is also a factor that can affect how the code behaves. We don't need to have one single test that captures all of these things, but when designing cases for testing purposes, it is important to consider more than the horizontal dimensions and resolution of the grid. The more scenarios we can capture with a fast run over a small domain, the more robust the testing will be. I'm not raising any objections to this PR, just commenting on the design strategy of tests in general.

[christopherwharrop-noaa]

Just a quick clarification about my comment above -- We absolutely DO need the tests to run quickly, so small domains are absolutely necessary for that purpose. I just meant say that there are other things to think about, too.

[gsketefian]

@EdwardSnyder-NOAA On Hera with the default resources (i.e. no computational resources specified in the test's configuration file), the timing for the 200x200 grid is as follows:

       CYCLE                    TASK             JOBID          STATE    EXIT STATUS     TRIES      DURATION
============================================================================================================
201906151800               make_grid          30039667      SUCCEEDED              0         1          11.0
201906151800               make_orog          30039680      SUCCEEDED              0         1          24.0
201906151800          make_sfc_climo          30039756      SUCCEEDED              0         1          46.0
201906151800           get_extrn_ics          30039668      SUCCEEDED              0         1           7.0
201906151800          get_extrn_lbcs          30039669      SUCCEEDED              0         1           8.0
201906151800                make_ics          30039784      SUCCEEDED              0         1          27.0
201906151800               make_lbcs          30039785      SUCCEEDED              0         1          24.0
201906151800                run_fcst          30039845      SUCCEEDED              0         1         770.0
201906151800           run_post_f000          30040053      SUCCEEDED              0         1           9.0
201906151800           run_post_f001          30040076      SUCCEEDED              0         1          12.0
201906151800           run_post_f002          30040140      SUCCEEDED              0         1          11.0
201906151800           run_post_f003          30040159      SUCCEEDED              0         1          10.0
201906151800           run_post_f004          30040169      SUCCEEDED              0         1          10.0
201906151800           run_post_f005          30040170      SUCCEEDED              0         1          10.0
201906151800           run_post_f006          30040171      SUCCEEDED              0         1          10.0

To check what happens for a 300x300 grid, I created such a grid over Indianapolis and reran the experiment. With the same computational resources as the 200x200 experiment (including the same MPI layout of the forecast model, which is 5x5), the timing is as follows:

       CYCLE                    TASK            JOBID          STATE    EXIT STATUS     TRIES      DURATION
===========================================================================================================
201906151800               make_grid         30225690      SUCCEEDED              0         1          11.0
201906151800               make_orog         30225701      SUCCEEDED              0         1          28.0
201906151800          make_sfc_climo         30225711      SUCCEEDED              0         1          51.0
201906151800           get_extrn_ics         30225691      SUCCEEDED              0         1           8.0
201906151800          get_extrn_lbcs         30225692      SUCCEEDED              0         1           8.0
201906151800                make_ics         30225777      SUCCEEDED              0         1          29.0
201906151800               make_lbcs         30225776      SUCCEEDED              0         1          26.0
201906151800                run_fcst         30225796      SUCCEEDED              0         1        1558.0
201906151800           run_post_f000         30226603      SUCCEEDED              0         1          14.0
201906151800           run_post_f001         30226694      SUCCEEDED              0         1          13.0
201906151800           run_post_f002         30227293      SUCCEEDED              0         1          11.0
201906151800           run_post_f003         30227351      SUCCEEDED              0         1          12.0
201906151800           run_post_f004         30227679      SUCCEEDED              0         1          23.0
201906151800           run_post_f005         30227824      SUCCEEDED              0         1          11.0
201906151800           run_post_f006         30227825      SUCCEEDED              0         1          11.0

So you can see the forecast model takes about twice as long to complete (with the other task run times being much smaller). In this case, it's an extra 13 minutes for the forecast task (I would think that's a lot if you're running on this grid all the time).

[gsketefian]

@gsketefian / @EdwardSnyder-NOAA / @ywangwof - For testing purposes, the size isn't the only consideration. Code coverage is also to be considered, as well as situations that present challenges. Code paths are determined by configuration choices, but also by weather conditions being simulated. Terrain is also a factor that can affect how the code behaves. We don't need to have one single test that captures all of these things, but when designing cases for testing purposes, it is important to consider more than the horizontal dimensions and resolution of the grid. The more scenarios we can capture with a fast run over a small domain, the more robust the testing will be. I'm not raising any objections to this PR, just commenting on the design strategy of tests in general.

@christopherwharrop-noaa The weather conditions are definitely very important. That's why currently there are multiple cases (dates) in the set of WE2E tests. They arose because at one point or another an experiment failed on some date (usually one representing some sort of severe weather), which was then debugged, and that date saved in one of the tests. But now we're paring down the number of dates to reduce the size of the test data. We certainly can't keep all dates that failed in the tests; I'm not sure what a good balance is.

It's similar for terrain. We've so far supported only CONUS grids in releases, so the tests have concentrated on those, but to make the App really robust, we should add the Alaska domains to the tests.

For this PR, I'm thinking of only the release -- i.e. to have a grid that will allow more rapid testing.

[EdwardSnyder-NOAA]

@gsketefian - Thanks for running the comparison between the two grids so we can see the difference in runtimes. Since we want to run this grid often, resorting to 200 x 200 would be sufficient for the release.

[gsketefian]

@mark-a-potts If you're ok with this grid for your testing, can you give an approval for the PR? Thanks.