Add option to use shared scotch libs

[ulmononian]

Pull Request Summary

Enable the (optional) use of shared scotch libraries.

Description

Some changes to cmake/FindSCOTCH.cmake and model/src/CMakeLists.txt are added here so that shared scotch libraries can be used when appropriate/wanted. Future spack-stack installations will most likely install only the shared libraries, so the capability will become necessary. Static library options need to be retain for operational machines where shared libraries are not supported.

Please also include the following information:
Potential reviewers: @MatthewMasarik-NOAA @JessicaMeixner-NOAA

• enhancement
• Not sure if it will change results

Issue(s) addressed

Fixes #1021

Commit Message

Enable the (optional) use of shared scotch libraries.

Check list

• Branch is up to date with the authoritative repository (NOAA-EMC) develop branch.
• Checked the checklist for a developer submitting to develop.
• If a version number update is required, checked the updating version number checklist.
• If a new feature was added, a regression test for testing the new feature is added.

Testing

• How were these changes tested?
  using the PDLIB RT cmake settings from the UFS-WM (i.e. cmake -DAPP=S2SW -DCCPP_SUITES=FV3_GFS_v17_coupled_p8 -DPDLIB=ON ..)
• Are the changes covered by regression tests? (If not, why? Do new tests need to be added?)
  I am only sure of how they would be covered in the UFS-WM S2SW/S2SWA configurations where PDLIB is turned on.
• Have the matrix regression tests been run (if yes, please note HPC and compiler)?
  No
• Please indicate the expected changes in the regression test output, (Note the list of known non-identical tests.)
• Please provide the summary output of matrix.comp (matrix.Diff.txt, matrixCompFull.txt and matrixCompSummary.txt):

[ulmononian]

this was tested within the ufs-wm compilation stage using the spack-stack/1.4.0 scotch/7.0.3 installation (note this is a static set of scotch libraries). the option to set USE_SHARED_SCOTCH as a cmake arg. was added, so that static or shared can be specified in the cmake command, i.e.:

cmake -DAPP=S2SW -DCCPP_SUITES=FV3_GFS_v17_coupled_p8 -DPDLIB=ON -DUSE_SHARED_SCOTCH=[ON/OFF] ..

i tested it with USE_SHARED_SCOTCH both ON and OFF and the behavior was as expected, i.e., when set to ON, it finds the static scotch libs and proceeds to finish the cmake step; when OFF, it throws an error letting the user know that the requested (shared) libs are not present.

[MatthewMasarik-NOAA]

@ulmononian has this been tested by anyone beyond the compilation stage? I've found it's possible for scotch and WW3 to build without error, but crash at runtime when it enters the scotch call. Please confirm the functionality works at runtime as well.

[ulmononian]

@MatthewMasarik-NOAA it has been tested in compilation and in ufs-wm rts. it does not change the outcome of the ufs-wm rt's where PDLIB=ON; that is, they still fail in the scotch call. basically, testing this modification in the wm only checks if the USE_SHARED_FLAG is working, and as we only have static scotch libs. installed, it just finds those.

if someone on the ww3 team could test these changes in the internal ww3 tests, i would appreciate it. i am happy to try and run them myself, but would need instruction.

[MatthewMasarik-NOAA]

@MatthewMasarik-NOAA it has been tested in compilation and in ufs-wm rts. it does not change the outcome of the ufs-wm rt's where PDLIB=ON; that is, they still fail in the scotch call. basically, testing this modification in the wm only checks if the USE_SHARED_FLAG is working, and as we only have static scotch libs. installed, it just finds those.

if someone on the ww3 team could test these changes in the internal ww3 tests, i would appreciate it. i am happy to try and run them myself, but would need instruction.

@ulmononian is this in reference to intel or gnu spack-stack? Previously I verified the intel spack-stack SCOTCH install built and ran successfully (see comment, JCSDA/spack-stack#550 (comment)). If the current static intel spack-stack version is failing to run, then that's news to me. I would suggest resolving that issue before moving on to building with shared.

[ulmononian]

@MatthewMasarik-NOAA sorry -- you are correct that only the gnu spack-stack scotch installation is failing.

this pr is just to add the option for using shared libraries in the future. we won't switch from static to shared on the spack side until discussing with your team & as you said, resolving the static (gnu) issues first.

[MatthewMasarik-NOAA]

@MatthewMasarik-NOAA sorry -- you are correct that only the gnu spack-stack scotch installation is failing.

this pr is just to add the option for using shared libraries in the future. we won't switch from static to shared on the spack side until discussing with your team & as you said, resolving the static (gnu) issues first.

@ulmononian, okay glad to hear the intel spack-stack is working.

I understand this is just an option. Though if we add the ability to select an option, the functionality it selects should be working, right? If not, doesn't this just kick the same problem down the road?

[MatthewMasarik-NOAA]

Hi @ulmononian, I'll mention that when we were trying to resolve scotch issues on orion sometime ago, I went down a similar path trying to build SCOTCH/WW3 with spack (not spack-stack, though). It built SCOTCH default as .so files, so I made similar changes to FindSCOTCH.cmake to use those instead of .a. It seemed it was able to build without reporting an error, but when run it crashes in initialization where execution hits the SCOTCH call. I wasn't successful and that was my only attempt at using shared instead of static files. I might suggest seeing if you can build/run WW3 from SCOTCH .so files using hpc-stack first, if you can do that, try with spack-stack.

As far as testing, I think you are going about it the easiest way, using the a coupled UFS RT. I'm happy to give you a demo on how you could debug in WW3 standalone though, if that's something you'd like to try. Whichever of those you use you will always have to rebuild SCOTCH + WW3, and then do a run as well, at least up to model init, because that is the only way you can confirm a successful build. Since the builds and run details like job card resources are all hidden when you use rt.sh in UFS, I think it's going to be easiest continuing with testing like that, vs. standalone WW3, but I will teach you if you like.

[MatthewMasarik-NOAA]

@ulmononian we've merged in a fix for the CI. Please sync when you have a chance

[MatthewMasarik-NOAA]

Hi @ulmononian, there hasn't been any activity on this in awhile. Do you plan to pursue in the near future, or should this be a draft for the time being? Thanks for any status update

[ulmononian]

@MatthewMasarik-NOAA sorry about the radio silence. this was removed from my radar. i don't think i'll be able to pursue this for a while, so it can be kept as a draft (or closed).

[MatthewMasarik-NOAA]

@MatthewMasarik-NOAA sorry about the radio silence. this was removed from my radar. i don't think i'll be able to pursue this for a while, so it can be kept as a draft (or closed).

Hi @ulmononian, no worries. thanks for letting me know where things stand. If spack-stack does go to just shared builds then this will be nice to have, and I suspect it is already pretty close. So I think we should keep it in the queue for the time being. It looks like as a maintainer I'm not able to mark the PR as a draft, could you mark it a draft? thanks again for the update.