Issue with Tiling and Multiple Processors in Noah-MP 4.0.1 with Sprinkler Irrigation Scheme

[smodanesi]

Hello LISF Community,
I am conducting experiments using Noah-MP version 4.0.1, coupled with the sprinkler irrigation scheme to obtain irrigation quantities over CONUS. Since I am running simulations at a spatial resolution of 0.25°, I wanted to activate the tiling option to obtain more accurate irrigation values. However, I encountered an issue when running irrigation with tiling across multiple processors.
I have attached two images that show maps of cumulative annual irrigation, mean annual soil moisture, cumulative annual evapotranspiration, and mean annual leaf area index, using both 8 and 36 processors. As you’ll notice, the 36-processor run presents an error where areas that should be irrigated are left with no irrigation (see the large squares with zero values in the file output_36_processors.jpg as compared with the output_8_processors.jpg). Please, note that this run was obtained using the lis.config option "decompose by processors: 'true'"; however similar issues can be found setting this option equal to "False" (just the big squares with zero values are located in another position).
I attempted to debug the code, checking potential issues such as the irrigation scale at the tiling level, but I haven’t been able to find the error. I also attach my ldt.config and lis.config files for your reference, in case there might be an issue with the tiling setup in the LDT input NetCDF (or in the lis.config file).
I would greatly appreciate any insights or suggestions you may have to help resolve this issue. If you need any additional information please just let me know.
Thank you very much for your assistance.
Best regards,
Sara
ldt.config_CONUS_tiling.txt
lis.config_CONUS_tiling.txt

[KathyNie]

Hi Sara,

Your ldt.config and lis.config seem good to me and I did not see anything that might be causing this issue of no irrigation for those grid cells. I have a few suggestions to test out:

1. I noticed that you were using 2 processors for LDT, not sure if this may cause the abnormal behavior, but I wonder if you could take a look and the irrigation faction variable in the lis_input file and see if that irrigation fraction looks ok? And could you generate the lis_input using only a single processor and see if lis_input may be different from the one generated by 2 processors? Besides, are you using the same lis_input file for both 8-processor and 36-processor simulations?

2. Could you try to comment out the whole line of "Decompose by processes: .true." and rerun with 66 processor layout? In addition, can you test for a few more processor set up such as 136, 361, 44 and so on to see if the no-irrigation pattern changes along with the processor layout?

Best,
Wanshu

[smodanesi]

Dear Wanshu,
Thank you for your prompt response and helpful suggestions. I’ve implemented the steps you recommended and would like to share the results and provide further details below:

1. I re-ran LDT using only one processor, and the NetCDF output is identical to the one generated using two processors. Maybe something to say: for the irrigation fraction, we replaced the GRIPC map (the default in LIS) with a new Landsat-derived irrigation fraction map based on this dataset https://lpdaac.usgs.gov/products/lgrip30v001/. We converted it into the same format as the GRIPC map, and LIS successfully reads it as if it was GRIPC . I’ve attached an image comparing the two irrigation fraction maps (Fig1_IRRIGFRAC). Regarding your second question, yes, I am using the same LDT file for both simulations.
2. As suggested, I commented out the line #Decompose by processes: .true. and ran four small test simulations (cold start: 1990/1991) over CONUS using 8, 12, 24, and 36 processors. Unfortunately, I couldn't test with more processors, as I currently have access to only one node with 36 processors. However, a colleague working with me on irrigation in LIS has access to additional resources, and we could switch to that system if needed for further testing. Despite these changes, I still observe the MPI stripe issue. Please see the attached figures (IRR_1991_MPI_issue.jpeg and SSM_1991_MPI_issue.jpeg) for reference.
3. I had already examined the code in debug mode, initially suspecting an issue related to the Irrigation Scale, but I couldn’t identify any errors. One question: Did you use irrigation with tiling in combination with the MODIS IGBP land cover?
   Thank you again for your assistance. If you have any further suggestions or insights, we would greatly appreciate them.
   Best,
   Sara
   
   
   

[KathyNie]

Hi Sara,

Thanks for sharing additional info and conducting these tests. I ran a few tests on our server with almost identical settings as your ldt.config and lis.config, except for using different met forcings (MERRA2+IMERG). I also specified irrigation with tiling in combination with the MODIS IGBP land cover as indicated in your configuration. I did not run into the strip issues using 36 processors as well as a few other combinations.

I wonder if that is due to the env set up, which I'm not quite familiar with. If you are able to test with the system that your colleague has been working on and get back, that would be perfect. I'm also cc'ing my colleague @jvgeiger here to see if he has any suggestions.

Best,
Wanshu

[lbusschaert]

Dear Wanshu,

Thank you for looking further into this problem. Sara and I looked further into the code and were able to fix the issue. Our KUL-RSDA LISF branch is now already updated with the following PR (KUL-RSDA#48). Should I open an issue to also include this PR upstream (in the NASA-LIS)?

Best,
Louise

[KathyNie]

Hi Louise,

Great catch! I'm glad that you've found a fix for this issue. It's interesting that it is caused by "dt". Yes, I encourage you to submit a PR to the master branch. And meanwhile, I'm ccing my colleague Kristi @karsenau for review.

Hey Kristi - have you experienced any similar issue as Sara and Louise mentioned here with sprinkler irrigation in Noah-MP.4.0.1?

Best,
Wanshu

[dmocko]

Hi @lbusschaert

Thank you so much for identifying and resolving this bug!

Yes - please submit a PR to the LISF GitHub. Because this is a bug, if you can, please submit the PR to this branch:
support/lisf-public-7.5

But before you submit a PR, please open an Issue on the LISF GitHub. You might review CONTRIBUTING.md in our main directory.

You can see our preferred procedure in this Working with GitHub documentation (particularly Step 5b):
https://nasa-lis.github.io/LISF/working_with_github/working_with_github.html#sec_step5b

You should follow the "Second time" steps in this Step (not the "First time", as the support/lisf-public-7.5 branch already exists).

After the PR is merged, we will also migrate it to our master (and other) branches.

Thanks again!

[karsenau]

Hi @lbusschaert and @smodanesi:

Thank you both for this catch of the uninitialized NOAHMP401_struc(n)%dt term in that irrigation routine and replacing it with the actual model timestep that gets used by NoahMP401 (NOAHMP401_struc(n)%ts).

I've reviewed and agree with this update.

So when ready, as @dmocko suggested, you can submit the PR to our support/lisf-public-7.5 branch as a bug fix.

Thanks again!

[dmocko]

Thanks to @lbusschaert and @smodanesi for identifying and resolving this issue.

The bug fix has been merged via PR #1624