release/public-v1: fix grib2 read errors with large task counts (draft PR)

[climbfuji]

This PR is an attempt to fox the grib2 read errors encountered with large task counts randomly on several machines. The issue is described in length in ufs-community/ufs-mrweather-app#190, together with a suggestion from @GeorgeGayno-NOAA in ufs-community/ufs-mrweather-app#190 (comment) (implemented here).

Changes in this PR:

• sorc/chgres_cube.fd/model_grid.F90: read grib2 inventory only with task 0, let all other processes wait for it to complete
• correct syntax for MPI_ABORT calls in various files (from @DusanJovic-NOAA)

Testing:

• Installed a new/test version of NCEPLIBS ufs-v1.1.0 that contains this bugfix on orion and ran the full MRW App regression tests, all of them passed.
• Due to the random nature of the error, repeated runs of the regression test suite will be conducted on orion over night, and if successful also on Cheyenne tomorrow.

@ligiabernardet @panll FYI

[climbfuji]

@GeorgeGayno-NOAA Should we decide to proceed with this PR, I want to suggest to pull in your bugfix PR for develop that corrected the calls (arguments) to mpi_abort etc. What do you think?

[GeorgeGayno-NOAA]

@GeorgeGayno-NOAA Should we decide to proceed with this PR, I want to suggest to pull in your bugfix PR for develop that corrected the calls (arguments) to mpi_abort etc. What do you think?

I have no problem with that. I am sure @DusanJovic-NOAA would agree.

[climbfuji]

@GeorgeGayno-NOAA Should we decide to proceed with this PR, I want to suggest to pull in your bugfix PR for develop that corrected the calls (arguments) to mpi_abort etc. What do you think?

I have no problem with that. I am sure @DusanJovic-NOAA would agree.

Thanks, @GeorgeGayno-NOAA. I cherry-picked the commits relevant for release/public-v1 from #148, resolved the conflicts and updated this PR. Would you please take a look? I'll test it with the MRW App on a few platforms so that we can make sure it is working as intended.

[GeorgeGayno-NOAA]

@GeorgeGayno-NOAA Should we decide to proceed with this PR, I want to suggest to pull in your bugfix PR for develop that corrected the calls (arguments) to mpi_abort etc. What do you think?

I have no problem with that. I am sure @DusanJovic-NOAA would agree.

Thanks, @GeorgeGayno-NOAA. I cherry-picked the commits relevant for release/public-v1 from #148, resolved the conflicts and updated this PR. Would you please take a look? I'll test it with the MRW App on a few platforms so that we can make sure it is working as intended.

@climbfuji Your updates look correct.

[climbfuji]

@GeorgeGayno-NOAA Should we decide to proceed with this PR, I want to suggest to pull in your bugfix PR for develop that corrected the calls (arguments) to mpi_abort etc. What do you think?

I have no problem with that. I am sure @DusanJovic-NOAA would agree.

Thanks, @GeorgeGayno-NOAA. I cherry-picked the commits relevant for release/public-v1 from #148, resolved the conflicts and updated this PR. Would you please take a look? I'll test it with the MRW App on a few platforms so that we can make sure it is working as intended.

@climbfuji Your updates look correct.

@GeorgeGayno-NOAA this works as expected, tested on Cheyenne with both GNU and Intel and on Orion with Intel. Can you please approve and merge?

[climbfuji]

Thanks, @GeorgeGayno-NOAA - merged and retagged, pushed the tag to the repo.