Modulefiles and Issues with Discover SCU17/18

[dmocko]

The below information only applies directly to team members who are compiling and trying to run
LISF on Discover SCU17/18, aka the Milan nodes:

Thanks to @jvgeiger for the below text (and for all his great work developing these module files!).

GNU modulefiles are not yet available for the Milan nodes. The GNU compilers are not yet fully
integrated with the Intel MPI library. NCCS is working on this.

---

We have two modulefiles for compiling/running LISF on the Milan nodes.
Both are found in the master branch of the NASA-LIS/LISF repository.

1. env/discover/sles15/lisf_7.5_intel_2023.2.1
2. env/discover/sles15/lisf_7.5_intel_2023.2.1_openmpi_5.0.0

lisf_7.5_intel_2023.2.1 is for running LIS using the Intel MPI library.

lisf_7.5_intel_2023.2.1_openmpi_5.0.0 is for running LIS using the OpenMPI library.

I do not have any guidance on which modulefile to use. All I can suggest is to try one.
If your parallel LIS job has issues, then try the other.

To compile and run on Milan, first copy the above two modulefiles into your ~/privatemodules directory. Then:

• Make sure that you are on Milan - ssh discover-mil.
• Load one of the new modulefiles; for example, module load lisf_7.5_intel_2023.2.1.
• Create a lis/make/user.cfg file containing:
  VIC.4.1.1: Off
  VIC.4.1.2: Off
• Configure (using "-1" or "2", see below) and compile as usual.
• Submit your job to the Milan nodes - '#SBATCH --contraint=mil'.

Here are a list of issues.
When I mention compiling with "-3", etc., I mean the optimization level selected at this configure prompt:

% ./configure
Optimization level (-3=strict checks with warnings, -2=strict checks, -1=debug, 0,1,2,3, default=2):

Issue 1:

The Intel compiler crashes when trying to compile VIC. I opened a ticket about this, so for now use the above user.cfg workaround. I plan to disable VIC by default in an upcoming commit (note: disable, not remove). If you really need VIC, then contact me. I do have a kluge (workaround).

Issue 2:

The Intel compiler aborts when compiling LIS with "-2".

The problem is the new Intel icx compiler (and underlying clang compiler) no longer supports -check= and -ftrapuv compiler switches. For now, do not compile with "-2". If you need to compile with "-2", then remove the -check= and -ftrapuv switches from the CFLAGS in the configure.lis file. Before I can fix our build scripts, I must review the other HPC systems that we use because we may still need the old Intel compiler.

LDT has this same issue when compiling with "-3".

LVT has this same issue when compiling with "-3".

Issue 3:

The Intel compilers abort when compiling LISF (LDT,LIS,LVT) with "-3" using multiple jobs. By multiple jobs, I mean ./compile -j 16. LISF will successfully compile using "-3" with ./compile. I am still investigating.

Issue 4:

LDT fails to link when using both "-3" and "-2". I am still investigating.

Issue 5:

OpenMPI crashes when an executable is compiled with the -fpe0 switch. So, if you execute LDT or LIS using mpirun (for example mpirun -n 1024 ./LIS ... or mpirun -n 1 ./LDT ...), then you must remove the -fpe0 switches from your configure.{ldt,lis} file before compiling. If you run

./LDT ldt.config

then you do not have to edit your configure.ldt file before compiling.

I suggest that you simply remove the -fpe0 switches from the configure.ldt and configure.lis files.

Note: This does not apply to LVT. It does not use MPI.

Note: This applies to using OpenMPI, not to using Intel MPI.

Finally, to compile and run on the older Skylake and Cascade Lake nodes:

• Make sure that you are on the older login nodes - ssh discover.
• Load either lisf_7.5_gnu_11.2.0_impi_2021.4.0 or lisf_7.5_intel_2021.4.0; for example, module load lisf_7.5_intel_2021.4.0.
• Configure and compile as usual.
• Submit your job to the older nodes - '#SBATCH --constraint="[sky|cas]"'.