Another issue building libmesh on an HPC #29134

TheGreatCid · 2024-11-25T17:09:53Z

TheGreatCid
Nov 25, 2024

Check these boxes if you have followed the posting rules.

I have consulted the Posting Guidelines.
I have searched the Discussions Forum and MOOSE Framework Troubleshooting and have not found what I was looking for
Q&A Getting Started is the most appropriate category for my question (trouble installing, beginner user, ...)

Issue or question about MOOSE

I am running into the following error(posted in the errors box) when getting to the compilation step of update_and_rebuild_libmesh.sh

Any insight would be appreciated.

Thanks!

(Optional) code in question / simulation log / errors

../src/base/libmesh.C:501:28: error: incompatible pointer to integer conversion assigning to 'communicator' (aka 'int') from 'MPI_Comm' (aka 'ompi_communicator_t *')
      this->_comm->get() = PETSC_COMM_SELF;
                           ^~~~~~~~~~~~~~~
/ascldap/users/detorre/raccoon/moose/petsc/include/petscsys.h:276:25: note: expanded from macro 'PETSC_COMM_SELF'
#define PETSC_COMM_SELF MPI_COMM_SELF
                        ^~~~~~~~~~~~~
/projects/aue/hpc/deploy/b306e83c/linux-rhel8-x86_64/gcc-12.1.0/openmpi-4.1.6-fa5w34i/include/mpi.h:1165:23: note: expanded from macro 'MPI_COMM_SELF'
#define MPI_COMM_SELF OMPI_PREDEFINED_GLOBAL(MPI_Comm, ompi_mpi_comm_self)
                      ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/projects/aue/hpc/deploy/b306e83c/linux-rhel8-x86_64/gcc-12.1.0/openmpi-4.1.6-fa5w34i/include/mpi.h:404:46: note: expanded from macro 'OMPI_PREDEFINED_GLOBAL'
#define OMPI_PREDEFINED_GLOBAL(type, global) (static_cast<type> (static_cast<void *> (&(global))))
                                             ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
../src/base/libmesh.C:508:7: error: no matching function for call to 'MPI_Abort'
      CHKERRABORT(libMesh::GLOBAL_COMM_WORLD,ierr);
      ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/ascldap/users/detorre/raccoon/moose/petsc/include/petscerror.h:760:32: note: expanded from macro 'CHKERRABORT'
#define CHKERRABORT(comm, ...) PetscCallAbort(comm, __VA_ARGS__)
                               ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/ascldap/users/detorre/raccoon/moose/petsc/include/petscerror.h:725:15: note: expanded from macro 'PetscCallAbort'
        (void)MPI_Abort(comm, (PetscMPIInt)ierr_petsc_call_abort_); \
              ^~~~~~~~~
/projects/aue/hpc/deploy/b306e83c/linux-rhel8-x86_64/gcc-12.1.0/openmpi-4.1.6-fa5w34i/include/mpi.h:1324:20: note: candidate function not viable: no known conversion from 'int' to 'MPI_Comm' (aka 'ompi_communicator_t *') for 1st argument
OMPI_DECLSPEC  int MPI_Abort(MPI_Comm comm, int errorcode);
                   ^
../src/base/libmesh.C:521:11: error: no matching function for call to 'MPI_Abort'
          CHKERRABORT(libMesh::GLOBAL_COMM_WORLD,ierr);
          ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/ascldap/users/detorre/raccoon/moose/petsc/include/petscerror.h:760:32: note: expanded from macro 'CHKERRABORT'
#define CHKERRABORT(comm, ...) PetscCallAbort(comm, __VA_ARGS__)
                               ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/ascldap/users/detorre/raccoon/moose/petsc/include/petscerror.h:725:15: note: expanded from macro 'PetscCallAbort'
        (void)MPI_Abort(comm, (PetscMPIInt)ierr_petsc_call_abort_); \
              ^~~~~~~~~
/projects/aue/hpc/deploy/b306e83c/linux-rhel8-x86_64/gcc-12.1.0/openmpi-4.1.6-fa5w34i/include/mpi.h:1324:20: note: candidate function not viable: no known conversion from 'int' to 'MPI_Comm' (aka 'ompi_communicator_t *') for 1st argument
OMPI_DECLSPEC  int MPI_Abort(MPI_Comm comm, int errorcode);
                   ^
../src/base/libmesh.C:532:55: error: no matching function for call to 'MPI_Abort'
      ierr = DMRegister(DMLIBMESH, DMCreate_libMesh); CHKERRABORT(libMesh::GLOBAL_COMM_WORLD,ierr);
                                                      ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/ascldap/users/detorre/raccoon/moose/petsc/include/petscerror.h:760:32: note: expanded from macro 'CHKERRABORT'
#define CHKERRABORT(comm, ...) PetscCallAbort(comm, __VA_ARGS__)
                               ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/ascldap/users/detorre/raccoon/moose/petsc/include/petscerror.h:725:15: note: expanded from macro 'PetscCallAbort'
        (void)MPI_Abort(comm, (PetscMPIInt)ierr_petsc_call_abort_); \
              ^~~~~~~~~
/projects/aue/hpc/deploy/b306e83c/linux-rhel8-x86_64/gcc-12.1.0/openmpi-4.1.6-fa5w34i/include/mpi.h:1324:20: note: candidate function not viable: no known conversion from 'int' to 'MPI_Comm' (aka 'ompi_communicator_t *') for 1st argument
OMPI_DECLSPEC  int MPI_Abort(MPI_Comm comm, int errorcode);
                   ^
../src/base/libmesh.C:795:9: error: no matching function for call to 'MPI_Abort'
        CHKERRABORT(libMesh::GLOBAL_COMM_WORLD, ierr);
        ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/ascldap/users/detorre/raccoon/moose/petsc/include/petscerror.h:760:32: note: expanded from macro 'CHKERRABORT'
#define CHKERRABORT(comm, ...) PetscCallAbort(comm, __VA_ARGS__)
                               ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/ascldap/users/detorre/raccoon/moose/petsc/include/petscerror.h:725:15: note: expanded from macro 'PetscCallAbort'
        (void)MPI_Abort(comm, (PetscMPIInt)ierr_petsc_call_abort_); \
              ^~~~~~~~~
/projects/aue/hpc/deploy/b306e83c/linux-rhel8-x86_64/gcc-12.1.0/openmpi-4.1.6-fa5w34i/include/mpi.h:1324:20: note: candidate function not viable: no known conversion from 'int' to 'MPI_Comm' (aka 'ompi_communicator_t *') for 1st argument
OMPI_DECLSPEC  int MPI_Abort(MPI_Comm comm, int errorcode);
                   ^
../src/base/libmesh.C:799:13: error: no matching function for call to 'MPI_Abort'
            CHKERRABORT(libMesh::GLOBAL_COMM_WORLD, ierr);
            ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/ascldap/users/detorre/raccoon/moose/petsc/include/petscerror.h:760:32: note: expanded from macro 'CHKERRABORT'
#define CHKERRABORT(comm, ...) PetscCallAbort(comm, __VA_ARGS__)
                               ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/ascldap/users/detorre/raccoon/moose/petsc/include/petscerror.h:725:15: note: expanded from macro 'PetscCallAbort'
        (void)MPI_Abort(comm, (PetscMPIInt)ierr_petsc_call_abort_); \
              ^~~~~~~~~
/projects/aue/hpc/deploy/b306e83c/linux-rhel8-x86_64/gcc-12.1.0/openmpi-4.1.6-fa5w34i/include/mpi.h:1324:20: note: candidate function not viable: no known conversion from 'int' to 'MPI_Comm' (aka 'ompi_communicator_t *') for 1st argument
OMPI_DECLSPEC  int MPI_Abort(MPI_Comm comm, int errorcode);
                   ^
../src/base/libmesh.C:818:7: error: no matching function for call to 'MPI_Abort'
      CHKERRABORT(libMesh::GLOBAL_COMM_WORLD, ierr);
      ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/ascldap/users/detorre/raccoon/moose/petsc/include/petscerror.h:760:32: note: expanded from macro 'CHKERRABORT'
#define CHKERRABORT(comm, ...) PetscCallAbort(comm, __VA_ARGS__)
                               ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/ascldap/users/detorre/raccoon/moose/petsc/include/petscerror.h:725:15: note: expanded from macro 'PetscCallAbort'
        (void)MPI_Abort(comm, (PetscMPIInt)ierr_petsc_call_abort_); \
              ^~~~~~~~~
/projects/aue/hpc/deploy/b306e83c/linux-rhel8-x86_64/gcc-12.1.0/openmpi-4.1.6-fa5w34i/include/mpi.h:1324:20: note: candidate function not viable: no known conversion from 'int' to 'MPI_Comm' (aka 'ompi_communicator_t *') for 1st argument
OMPI_DECLSPEC  int MPI_Abort(MPI_Comm comm, int errorcode);
                   ^
  CXX      src/error_estimation/libmesh_opt_la-hp_coarsentest.lo
  CXX      src/error_estimation/libmesh_opt_la-hp_selector.lo
  CXX      src/error_estimation/libmesh_opt_la-hp_singular.lo
  CXX      src/error_estimation/libmesh_opt_la-jump_error_estimator.lo
  CXX      src/error_estimation/libmesh_opt_la-kelly_error_estimator.lo
7 errors generated.
  CXX      src/error_estimation/libmesh_opt_la-patch_recovery_error_estimator.lo
make[1]: *** [Makefile:26734: src/base/libmesh_opt_la-libmesh.lo] Error 1
make[1]: *** Waiting for unfinished jobs....
make[1]: Leaving directory '/home/detorre/raccoon/moose/libmesh/build'
make: *** [Makefile:34013: all-recursive] Error 1

Encountering Errors? Please include diagnostic output

[detorre@swa910 scripts]$ ./diagnostics.sh 

##################################################################################################
Influential Environment Variables

CC=/projects/aue/hpc/builds/x86_64/rhel8/ba17d7f2/clang-16.0.6/install/linux-rhel8-x86_64/clang-16.0.6/llvm-16.0.6-dk6jfj6/bin/clang
CONDA_CHANNEL=https://conda.software.inl.gov/public
CONDA_EXE=/home/detorre/miniforge/bin/conda
CONDA_PYTHON_EXE=/home/detorre/miniforge/bin/python
CONDA_SHLVL=0
CURL_CA_BUNDLE=
CXX=/projects/aue/hpc/builds/x86_64/rhel8/ba17d7f2/clang-16.0.6/install/linux-rhel8-x86_64/clang-16.0.6/llvm-16.0.6-dk6jfj6/bin/clang++
F77=/projects/aue/hpc/builds/x86_64/rhel8/ba17d7f2/clang-16.0.6/install/linux-rhel8-x86_64/clang-16.0.6/llvm-16.0.6-dk6jfj6/bin/flang-new
F90=mpif90
FC=/projects/aue/hpc/builds/x86_64/rhel8/ba17d7f2/clang-16.0.6/install/linux-rhel8-x86_64/clang-16.0.6/llvm-16.0.6-dk6jfj6/bin/flang-new
http_proxy=http://proxy.sandia.gov:80
https_proxy=http://proxy.sandia.gov:80
HTTPS_PROXY=http://proxy.sandia.gov:80
LD_LIBRARY_PATH=/projects/aue/hpc/deploy/b306e83c/linux-rhel8-x86_64/gcc-12.1.0/libfabric-1.19.0-3yvk7qz/lib64:/projects/aue/hpc/deploy/b306e83c/linux-rhel8-x86_64/gcc-12.1.0/libfabric-1.19.0-3yvk7qz/lib
MODULESHOME=/usr/share/lmod/lmod
MOOSE_JOBS=25
PATH=/projects/aue/hpc/builds/x86_64/rhel8/ba17d7f2/clang-16.0.6/install/linux-rhel8-x86_64/clang-16.0.6/llvm-16.0.6-dk6jfj6/bin:/projects/aue/hpc/deploy/b306e83c/linux-rhel8-x86_64/gcc-12.1.0/openmpi-4.1.6-fa5w34i/bin:/projects/aue/hpc/builds/x86_64/rhel8/ba17d7f2/gcc-12.1.0/install/linux-rhel8-x86_64/gcc-10.3.0/gcc-12.1.0-xpyjv5s/bin:/projects/aue/hpc/deploy/f9a38d2c/linux-rhel8-x86_64/gcc-12.3.0/binutils-2.42-xd7fipb/bin:/ascldap/users/detorre/bin:/home/detorre/miniforge/condabin:/usr/lpp/mmfs/bin:/opt/ibm/MCStore/bin:/opt/ibm/MCStore/scripts:/usr/lib64/ccache:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin
REQUESTS_CA_BUNDLE=/etc/ssl/certs/snl-plus-zscaler.crt
SSL_CERT_FILE=/etc/ssl/certs/snl-plus-zscaler.crt

##################################################################################################
Compiler(s) (CC CXX FC F77 F90):

CC=/projects/aue/hpc/builds/x86_64/rhel8/ba17d7f2/clang-16.0.6/install/linux-rhel8-x86_64/clang-16.0.6/llvm-16.0.6-dk6jfj6/bin/clang
CC version:     clang version 16.0.6 (https://github.com/spack/spack.git da0813b049b016b2bcd021abcf2e520ac8bb41ea)

CXX=/projects/aue/hpc/builds/x86_64/rhel8/ba17d7f2/clang-16.0.6/install/linux-rhel8-x86_64/clang-16.0.6/llvm-16.0.6-dk6jfj6/bin/clang++
CXX version:    clang version 16.0.6 (https://github.com/spack/spack.git da0813b049b016b2bcd021abcf2e520ac8bb41ea)

FC=/projects/aue/hpc/builds/x86_64/rhel8/ba17d7f2/clang-16.0.6/install/linux-rhel8-x86_64/clang-16.0.6/llvm-16.0.6-dk6jfj6/bin/flang-new
FC version:     flang-new version 16.0.6 (https://github.com/spack/spack.git da0813b049b016b2bcd021abcf2e520ac8bb41ea)

F77=/projects/aue/hpc/builds/x86_64/rhel8/ba17d7f2/clang-16.0.6/install/linux-rhel8-x86_64/clang-16.0.6/llvm-16.0.6-dk6jfj6/bin/flang-new
F77 version:    flang-new version 16.0.6 (https://github.com/spack/spack.git da0813b049b016b2bcd021abcf2e520ac8bb41ea)

F90=/projects/aue/hpc/deploy/b306e83c/linux-rhel8-x86_64/gcc-12.1.0/openmpi-4.1.6-fa5w34i/bin/mpif90
F90 -show:
/projects/aue/hpc/builds/x86_64/rhel8/ba17d7f2/gcc-12.1.0/install/linux-rhel8-x86_64/gcc-10.3.0/gcc-12.1.0-xpyjv5s/bin/gfortran -I////////projects/aue/hpc/deploy/b306e83c/linux-rhel8-x86_64/gcc-12.1.0/openmpi-4.1.6-fa5w34i/include -pthread -I////////projects/aue/hpc/deploy/b306e83c/linux-rhel8-x86_64/gcc-12.1.0/openmpi-4.1.6-fa5w34i/lib -L////////projects/aue/hpc/deploy/b306e83c/linux-rhel8-x86_64/gcc-12.1.0/openmpi-4.1.6-fa5w34i/lib -L/projects/aue/hpc/deploy/b306e83c/linux-rhel8-x86_64/gcc-12.1.0/hwloc-2.9.3-ylujs7h/lib -L/projects/aue/hpc/deploy/b306e83c/linux-rhel8-x86_64/gcc-12.1.0/libevent-2.1.12-oiai2u4/lib -Wl,-rpath,/projects/aue/hpc/builds/x86_64/rhel8/ba17d7f2/gcc-12.1.0/install/linux-rhel8-x86_64/gcc-10.3.0/gcc-12.1.0-xpyjv5s/lib/gcc/x86_64-pc-linux-gnu/12.1.0 -Wl,-rpath,/projects/aue/hpc/builds/x86_64/rhel8/ba17d7f2/gcc-12.1.0/install/linux-rhel8-x86_64/gcc-10.3.0/gcc-12.1.0-xpyjv5s/lib64 -Wl,-rpath -Wl,////////projects/aue/hpc/deploy/b306e83c/linux-rhel8-x86_64/gcc-12.1.0/openmpi-4.1.6-fa5w34i/lib -Wl,-rpath -Wl,/projects/aue/hpc/deploy/b306e83c/linux-rhel8-x86_64/gcc-12.1.0/hwloc-2.9.3-ylujs7h/lib -Wl,-rpath -Wl,/projects/aue/hpc/deploy/b306e83c/linux-rhel8-x86_64/gcc-12.1.0/libevent-2.1.12-oiai2u4/lib -lmpi_usempif08 -lmpi_usempi_ignore_tkr -lmpi_mpifh -lmpi
F90 version:    GNU Fortran (Spack GCC) 12.1.0

OK

##################################################################################################
Python Sanity Checks

Verify `/usr/bin/env python3 --version` (reporting as: Python 3.11.8),
matches versions for: `which python3 && which python`

OK

##################################################################################################
Python Modules (TestHarness, run-ability)

OK

Answered by TheGreatCid

Nov 25, 2024

I may have figured it out. I set those variable before loading the modules with mpicxx and mpicc. Swapping the order now gives me this diag

##################################################################################################
Influential Environment Variables

CC=/projects/aue/hpc/deploy/b306e83c/linux-rhel8-x86_64/gcc-12.1.0/openmpi-4.1.6-fa5w34i/bin/mpicc
CONDA_CHANNEL=https://conda.software.inl.gov/public
CONDA_EXE=/home/detorre/miniforge/bin/conda
CONDA_PYTHON_EXE=/home/detorre/miniforge/bin/python
CONDA_SHLVL=0
CURL_CA_BUNDLE=
CXX=/projects/aue/hpc/deploy/b306e83c/linux-rhel8-x86_64/gcc-12.1.0/openmpi-4.1.6-fa5w34i/bin/mpicxx
F77=/projects/aue/hpc/deploy/b306e83c/linux-r…

View full answer

GiudGiud · 2024-11-25T17:28:53Z

GiudGiud
Nov 25, 2024
Collaborator

@milljm
@dschwen there's flang

20 replies

TheGreatCid Nov 25, 2024
Author

This is what I exported

export CC=/projects/aue/hpc/deploy/b306e83c/linux-rhel8-x86_64/gcc-12.1.0/openmpi-4.1.6-fa5w34i/bin/mpicc
export CXX=/projects/aue/hpc/deploy/b306e83c/linux-rhel8-x86_64/gcc-12.1.0/openmpi-4.1.6-fa5w34i/bin/mpicxx
export FC=/projects/aue/hpc/deploy/b306e83c/linux-rhel8-x86_64/gcc-12.1.0/openmpi-4.1.6-fa5w34i/bin/mpif90
export F90=/projects/aue/hpc/deploy/b306e83c/linux-rhel8-x86_64/gcc-12.1.0/openmpi-4.1.6-fa5w34i/bin/mpif90
export F77=/projects/aue/hpc/deploy/b306e83c/linux-rhel8-x86_64/gcc-12.1.0/openmpi-4.1.6-fa5w34i/bin/mpif77

Here is the new diag

##################################################################################################
Influential Environment Variables

CC=/projects/aue/hpc/builds/x86_64/rhel8/ba17d7f2/gcc-12.1.0/install/linux-rhel8-x86_64/gcc-10.3.0/gcc-12.1.0-xpyjv5s/bin/gcc
CONDA_CHANNEL=https://conda.software.inl.gov/public
CONDA_EXE=/home/detorre/miniforge/bin/conda
CONDA_PYTHON_EXE=/home/detorre/miniforge/bin/python
CONDA_SHLVL=0
CURL_CA_BUNDLE=
CXX=/projects/aue/hpc/builds/x86_64/rhel8/ba17d7f2/gcc-12.1.0/install/linux-rhel8-x86_64/gcc-10.3.0/gcc-12.1.0-xpyjv5s/bin/g++
F77=/projects/aue/hpc/builds/x86_64/rhel8/ba17d7f2/gcc-12.1.0/install/linux-rhel8-x86_64/gcc-10.3.0/gcc-12.1.0-xpyjv5s/bin/gfortran
F90=/projects/aue/hpc/deploy/b306e83c/linux-rhel8-x86_64/gcc-12.1.0/openmpi-4.1.6-fa5w34i/bin/mpif90
FC=/projects/aue/hpc/builds/x86_64/rhel8/ba17d7f2/gcc-12.1.0/install/linux-rhel8-x86_64/gcc-10.3.0/gcc-12.1.0-xpyjv5s/bin/gfortran
http_proxy=http://proxy.sandia.gov:80
https_proxy=http://proxy.sandia.gov:80
HTTPS_PROXY=http://proxy.sandia.gov:80
LD_LIBRARY_PATH=/projects/aue/hpc/deploy/b306e83c/linux-rhel8-x86_64/gcc-12.1.0/libfabric-1.19.0-3yvk7qz/lib64:/projects/aue/hpc/deploy/b306e83c/linux-rhel8-x86_64/gcc-12.1.0/libfabric-1.19.0-3yvk7qz/lib
MODULESHOME=/usr/share/lmod/lmod
MOOSE_JOBS=25
PATH=/projects/aue/hpc/deploy/b306e83c/linux-rhel8-x86_64/gcc-12.1.0/openmpi-4.1.6-fa5w34i/bin:/projects/aue/hpc/builds/x86_64/rhel8/ba17d7f2/gcc-12.1.0/install/linux-rhel8-x86_64/gcc-10.3.0/gcc-12.1.0-xpyjv5s/bin:/projects/aue/hpc/deploy/f9a38d2c/linux-rhel8-x86_64/gcc-12.3.0/binutils-2.42-xd7fipb/bin:/ascldap/users/detorre/bin:/home/detorre/miniforge/condabin:/usr/lpp/mmfs/bin:/opt/ibm/MCStore/bin:/opt/ibm/MCStore/scripts:/usr/lib64/ccache:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin
REQUESTS_CA_BUNDLE=/etc/ssl/certs/snl-plus-zscaler.crt
SSL_CERT_FILE=/etc/ssl/certs/snl-plus-zscaler.crt

##################################################################################################
Compiler(s) (CC CXX FC F77 F90):

CC=/projects/aue/hpc/builds/x86_64/rhel8/ba17d7f2/gcc-12.1.0/install/linux-rhel8-x86_64/gcc-10.3.0/gcc-12.1.0-xpyjv5s/bin/gcc
CC version:     gcc (Spack GCC) 12.1.0

CXX=/projects/aue/hpc/builds/x86_64/rhel8/ba17d7f2/gcc-12.1.0/install/linux-rhel8-x86_64/gcc-10.3.0/gcc-12.1.0-xpyjv5s/bin/g++
CXX version:    g++ (Spack GCC) 12.1.0

FC=/projects/aue/hpc/builds/x86_64/rhel8/ba17d7f2/gcc-12.1.0/install/linux-rhel8-x86_64/gcc-10.3.0/gcc-12.1.0-xpyjv5s/bin/gfortran
FC version:     GNU Fortran (Spack GCC) 12.1.0

F77=/projects/aue/hpc/builds/x86_64/rhel8/ba17d7f2/gcc-12.1.0/install/linux-rhel8-x86_64/gcc-10.3.0/gcc-12.1.0-xpyjv5s/bin/gfortran
F77 version:    GNU Fortran (Spack GCC) 12.1.0

F90=/projects/aue/hpc/deploy/b306e83c/linux-rhel8-x86_64/gcc-12.1.0/openmpi-4.1.6-fa5w34i/bin/mpif90
F90 -show:
/projects/aue/hpc/builds/x86_64/rhel8/ba17d7f2/gcc-12.1.0/install/linux-rhel8-x86_64/gcc-10.3.0/gcc-12.1.0-xpyjv5s/bin/gfortran -I////////projects/aue/hpc/deploy/b306e83c/linux-rhel8-x86_64/gcc-12.1.0/openmpi-4.1.6-fa5w34i/include -pthread -I////////projects/aue/hpc/deploy/b306e83c/linux-rhel8-x86_64/gcc-12.1.0/openmpi-4.1.6-fa5w34i/lib -L////////projects/aue/hpc/deploy/b306e83c/linux-rhel8-x86_64/gcc-12.1.0/openmpi-4.1.6-fa5w34i/lib -L/projects/aue/hpc/deploy/b306e83c/linux-rhel8-x86_64/gcc-12.1.0/hwloc-2.9.3-ylujs7h/lib -L/projects/aue/hpc/deploy/b306e83c/linux-rhel8-x86_64/gcc-12.1.0/libevent-2.1.12-oiai2u4/lib -Wl,-rpath,/projects/aue/hpc/builds/x86_64/rhel8/ba17d7f2/gcc-12.1.0/install/linux-rhel8-x86_64/gcc-10.3.0/gcc-12.1.0-xpyjv5s/lib/gcc/x86_64-pc-linux-gnu/12.1.0 -Wl,-rpath,/projects/aue/hpc/builds/x86_64/rhel8/ba17d7f2/gcc-12.1.0/install/linux-rhel8-x86_64/gcc-10.3.0/gcc-12.1.0-xpyjv5s/lib64 -Wl,-rpath -Wl,////////projects/aue/hpc/deploy/b306e83c/linux-rhel8-x86_64/gcc-12.1.0/openmpi-4.1.6-fa5w34i/lib -Wl,-rpath -Wl,/projects/aue/hpc/deploy/b306e83c/linux-rhel8-x86_64/gcc-12.1.0/hwloc-2.9.3-ylujs7h/lib -Wl,-rpath -Wl,/projects/aue/hpc/deploy/b306e83c/linux-rhel8-x86_64/gcc-12.1.0/libevent-2.1.12-oiai2u4/lib -lmpi_usempif08 -lmpi_usempi_ignore_tkr -lmpi_mpifh -lmpi
F90 version:    GNU Fortran (Spack GCC) 12.1.0

OK

##################################################################################################
Python Sanity Checks

Verify `/usr/bin/env python3 --version` (reporting as: Python 3.11.8),
matches versions for: `which python3 && which python`

OK

##################################################################################################
Python Modules (TestHarness, run-ability)

OK

GiudGiud Nov 25, 2024
Collaborator

so weird, the diag still point to the other one

CXX=/projects/aue/hpc/builds/x86_64/rhel8/ba17d7f2/gcc-12.1.0/install/linux-rhel8-x86_64/gcc-10.3.0/gcc-12.1.0-xpyjv5s/bin/g++

@milljm is the diagnostics following symlinks basically?

milljm Nov 25, 2024
Maintainer

anything under the banner:

##################################################################################################
Influential Environment Variables

Is doing a simple env | grep CXX and the like.

As for everything else, we start to delve into using other tools. To answer your question, there is a possible "yes" to using symlinks. We're doing things like:

which $CXX
$(which $CXX) --version

etc

Edit:
In the case of MPI, we dig deeper by doing:

$(which $CXX) -show | cut -d\  -f1 | xargs which

To try and discover what the current MPI wrapper is set to use as its compiler.

TheGreatCid Nov 25, 2024
Author

I may have figured it out. I set those variable before loading the modules with mpicxx and mpicc. Swapping the order now gives me this diag

##################################################################################################
Influential Environment Variables

CC=/projects/aue/hpc/deploy/b306e83c/linux-rhel8-x86_64/gcc-12.1.0/openmpi-4.1.6-fa5w34i/bin/mpicc
CONDA_CHANNEL=https://conda.software.inl.gov/public
CONDA_EXE=/home/detorre/miniforge/bin/conda
CONDA_PYTHON_EXE=/home/detorre/miniforge/bin/python
CONDA_SHLVL=0
CURL_CA_BUNDLE=
CXX=/projects/aue/hpc/deploy/b306e83c/linux-rhel8-x86_64/gcc-12.1.0/openmpi-4.1.6-fa5w34i/bin/mpicxx
F77=/projects/aue/hpc/deploy/b306e83c/linux-rhel8-x86_64/gcc-12.1.0/openmpi-4.1.6-fa5w34i/bin/mpif77
F90=/projects/aue/hpc/deploy/b306e83c/linux-rhel8-x86_64/gcc-12.1.0/openmpi-4.1.6-fa5w34i/bin/mpif90
FC=/projects/aue/hpc/deploy/b306e83c/linux-rhel8-x86_64/gcc-12.1.0/openmpi-4.1.6-fa5w34i/bin/mpif90
http_proxy=http://proxy.sandia.gov:80
https_proxy=http://proxy.sandia.gov:80
HTTPS_PROXY=http://proxy.sandia.gov:80
LD_LIBRARY_PATH=/projects/aue/hpc/deploy/b306e83c/linux-rhel8-x86_64/gcc-12.1.0/libfabric-1.19.0-3yvk7qz/lib64:/projects/aue/hpc/deploy/b306e83c/linux-rhel8-x86_64/gcc-12.1.0/libfabric-1.19.0-3yvk7qz/lib
MODULESHOME=/usr/share/lmod/lmod
MOOSE_JOBS=25
PATH=/projects/aue/hpc/deploy/b306e83c/linux-rhel8-x86_64/gcc-12.1.0/openmpi-4.1.6-fa5w34i/bin:/projects/aue/hpc/builds/x86_64/rhel8/ba17d7f2/gcc-12.1.0/install/linux-rhel8-x86_64/gcc-10.3.0/gcc-12.1.0-xpyjv5s/bin:/projects/aue/hpc/deploy/f9a38d2c/linux-rhel8-x86_64/gcc-12.3.0/binutils-2.42-xd7fipb/bin:/ascldap/users/detorre/bin:/home/detorre/miniforge/condabin:/usr/lpp/mmfs/bin:/opt/ibm/MCStore/bin:/opt/ibm/MCStore/scripts:/usr/lib64/ccache:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin
REQUESTS_CA_BUNDLE=/etc/ssl/certs/snl-plus-zscaler.crt
SSL_CERT_FILE=/etc/ssl/certs/snl-plus-zscaler.crt

##################################################################################################
Compiler(s) (CC CXX FC F77 F90):

CC=/projects/aue/hpc/deploy/b306e83c/linux-rhel8-x86_64/gcc-12.1.0/openmpi-4.1.6-fa5w34i/bin/mpicc
CC -show:
/projects/aue/hpc/builds/x86_64/rhel8/ba17d7f2/gcc-12.1.0/install/linux-rhel8-x86_64/gcc-10.3.0/gcc-12.1.0-xpyjv5s/bin/gcc -I////////projects/aue/hpc/deploy/b306e83c/linux-rhel8-x86_64/gcc-12.1.0/openmpi-4.1.6-fa5w34i/include -pthread -L////////projects/aue/hpc/deploy/b306e83c/linux-rhel8-x86_64/gcc-12.1.0/openmpi-4.1.6-fa5w34i/lib -L/projects/aue/hpc/deploy/b306e83c/linux-rhel8-x86_64/gcc-12.1.0/hwloc-2.9.3-ylujs7h/lib -L/projects/aue/hpc/deploy/b306e83c/linux-rhel8-x86_64/gcc-12.1.0/libevent-2.1.12-oiai2u4/lib -Wl,-rpath,/projects/aue/hpc/builds/x86_64/rhel8/ba17d7f2/gcc-12.1.0/install/linux-rhel8-x86_64/gcc-10.3.0/gcc-12.1.0-xpyjv5s/lib/gcc/x86_64-pc-linux-gnu/12.1.0 -Wl,-rpath,/projects/aue/hpc/builds/x86_64/rhel8/ba17d7f2/gcc-12.1.0/install/linux-rhel8-x86_64/gcc-10.3.0/gcc-12.1.0-xpyjv5s/lib64 -Wl,-rpath -Wl,////////projects/aue/hpc/deploy/b306e83c/linux-rhel8-x86_64/gcc-12.1.0/openmpi-4.1.6-fa5w34i/lib -Wl,-rpath -Wl,/projects/aue/hpc/deploy/b306e83c/linux-rhel8-x86_64/gcc-12.1.0/hwloc-2.9.3-ylujs7h/lib -Wl,-rpath -Wl,/projects/aue/hpc/deploy/b306e83c/linux-rhel8-x86_64/gcc-12.1.0/libevent-2.1.12-oiai2u4/lib -lmpi
CC version:     gcc (Spack GCC) 12.1.0

CXX=/projects/aue/hpc/deploy/b306e83c/linux-rhel8-x86_64/gcc-12.1.0/openmpi-4.1.6-fa5w34i/bin/mpicxx
CXX -show:
/projects/aue/hpc/builds/x86_64/rhel8/ba17d7f2/gcc-12.1.0/install/linux-rhel8-x86_64/gcc-10.3.0/gcc-12.1.0-xpyjv5s/bin/g++ -I////////projects/aue/hpc/deploy/b306e83c/linux-rhel8-x86_64/gcc-12.1.0/openmpi-4.1.6-fa5w34i/include -pthread -L////////projects/aue/hpc/deploy/b306e83c/linux-rhel8-x86_64/gcc-12.1.0/openmpi-4.1.6-fa5w34i/lib -L/projects/aue/hpc/deploy/b306e83c/linux-rhel8-x86_64/gcc-12.1.0/hwloc-2.9.3-ylujs7h/lib -L/projects/aue/hpc/deploy/b306e83c/linux-rhel8-x86_64/gcc-12.1.0/libevent-2.1.12-oiai2u4/lib -Wl,-rpath,/projects/aue/hpc/builds/x86_64/rhel8/ba17d7f2/gcc-12.1.0/install/linux-rhel8-x86_64/gcc-10.3.0/gcc-12.1.0-xpyjv5s/lib/gcc/x86_64-pc-linux-gnu/12.1.0 -Wl,-rpath,/projects/aue/hpc/builds/x86_64/rhel8/ba17d7f2/gcc-12.1.0/install/linux-rhel8-x86_64/gcc-10.3.0/gcc-12.1.0-xpyjv5s/lib64 -Wl,-rpath -Wl,////////projects/aue/hpc/deploy/b306e83c/linux-rhel8-x86_64/gcc-12.1.0/openmpi-4.1.6-fa5w34i/lib -Wl,-rpath -Wl,/projects/aue/hpc/deploy/b306e83c/linux-rhel8-x86_64/gcc-12.1.0/hwloc-2.9.3-ylujs7h/lib -Wl,-rpath -Wl,/projects/aue/hpc/deploy/b306e83c/linux-rhel8-x86_64/gcc-12.1.0/libevent-2.1.12-oiai2u4/lib -lmpi
CXX version:    g++ (Spack GCC) 12.1.0

FC=/projects/aue/hpc/deploy/b306e83c/linux-rhel8-x86_64/gcc-12.1.0/openmpi-4.1.6-fa5w34i/bin/mpif90
FC -show:
/projects/aue/hpc/builds/x86_64/rhel8/ba17d7f2/gcc-12.1.0/install/linux-rhel8-x86_64/gcc-10.3.0/gcc-12.1.0-xpyjv5s/bin/gfortran -I////////projects/aue/hpc/deploy/b306e83c/linux-rhel8-x86_64/gcc-12.1.0/openmpi-4.1.6-fa5w34i/include -pthread -I////////projects/aue/hpc/deploy/b306e83c/linux-rhel8-x86_64/gcc-12.1.0/openmpi-4.1.6-fa5w34i/lib -L////////projects/aue/hpc/deploy/b306e83c/linux-rhel8-x86_64/gcc-12.1.0/openmpi-4.1.6-fa5w34i/lib -L/projects/aue/hpc/deploy/b306e83c/linux-rhel8-x86_64/gcc-12.1.0/hwloc-2.9.3-ylujs7h/lib -L/projects/aue/hpc/deploy/b306e83c/linux-rhel8-x86_64/gcc-12.1.0/libevent-2.1.12-oiai2u4/lib -Wl,-rpath,/projects/aue/hpc/builds/x86_64/rhel8/ba17d7f2/gcc-12.1.0/install/linux-rhel8-x86_64/gcc-10.3.0/gcc-12.1.0-xpyjv5s/lib/gcc/x86_64-pc-linux-gnu/12.1.0 -Wl,-rpath,/projects/aue/hpc/builds/x86_64/rhel8/ba17d7f2/gcc-12.1.0/install/linux-rhel8-x86_64/gcc-10.3.0/gcc-12.1.0-xpyjv5s/lib64 -Wl,-rpath -Wl,////////projects/aue/hpc/deploy/b306e83c/linux-rhel8-x86_64/gcc-12.1.0/openmpi-4.1.6-fa5w34i/lib -Wl,-rpath -Wl,/projects/aue/hpc/deploy/b306e83c/linux-rhel8-x86_64/gcc-12.1.0/hwloc-2.9.3-ylujs7h/lib -Wl,-rpath -Wl,/projects/aue/hpc/deploy/b306e83c/linux-rhel8-x86_64/gcc-12.1.0/libevent-2.1.12-oiai2u4/lib -lmpi_usempif08 -lmpi_usempi_ignore_tkr -lmpi_mpifh -lmpi
FC version:     GNU Fortran (Spack GCC) 12.1.0

F77=/projects/aue/hpc/deploy/b306e83c/linux-rhel8-x86_64/gcc-12.1.0/openmpi-4.1.6-fa5w34i/bin/mpif77
F77 -show:
/projects/aue/hpc/builds/x86_64/rhel8/ba17d7f2/gcc-12.1.0/install/linux-rhel8-x86_64/gcc-10.3.0/gcc-12.1.0-xpyjv5s/bin/gfortran -I////////projects/aue/hpc/deploy/b306e83c/linux-rhel8-x86_64/gcc-12.1.0/openmpi-4.1.6-fa5w34i/include -pthread -I////////projects/aue/hpc/deploy/b306e83c/linux-rhel8-x86_64/gcc-12.1.0/openmpi-4.1.6-fa5w34i/lib -L////////projects/aue/hpc/deploy/b306e83c/linux-rhel8-x86_64/gcc-12.1.0/openmpi-4.1.6-fa5w34i/lib -L/projects/aue/hpc/deploy/b306e83c/linux-rhel8-x86_64/gcc-12.1.0/hwloc-2.9.3-ylujs7h/lib -L/projects/aue/hpc/deploy/b306e83c/linux-rhel8-x86_64/gcc-12.1.0/libevent-2.1.12-oiai2u4/lib -Wl,-rpath,/projects/aue/hpc/builds/x86_64/rhel8/ba17d7f2/gcc-12.1.0/install/linux-rhel8-x86_64/gcc-10.3.0/gcc-12.1.0-xpyjv5s/lib/gcc/x86_64-pc-linux-gnu/12.1.0 -Wl,-rpath,/projects/aue/hpc/builds/x86_64/rhel8/ba17d7f2/gcc-12.1.0/install/linux-rhel8-x86_64/gcc-10.3.0/gcc-12.1.0-xpyjv5s/lib64 -Wl,-rpath -Wl,////////projects/aue/hpc/deploy/b306e83c/linux-rhel8-x86_64/gcc-12.1.0/openmpi-4.1.6-fa5w34i/lib -Wl,-rpath -Wl,/projects/aue/hpc/deploy/b306e83c/linux-rhel8-x86_64/gcc-12.1.0/hwloc-2.9.3-ylujs7h/lib -Wl,-rpath -Wl,/projects/aue/hpc/deploy/b306e83c/linux-rhel8-x86_64/gcc-12.1.0/libevent-2.1.12-oiai2u4/lib -lmpi_usempif08 -lmpi_usempi_ignore_tkr -lmpi_mpifh -lmpi
F77 version:    GNU Fortran (Spack GCC) 12.1.0

F90=/projects/aue/hpc/deploy/b306e83c/linux-rhel8-x86_64/gcc-12.1.0/openmpi-4.1.6-fa5w34i/bin/mpif90
F90 -show:
/projects/aue/hpc/builds/x86_64/rhel8/ba17d7f2/gcc-12.1.0/install/linux-rhel8-x86_64/gcc-10.3.0/gcc-12.1.0-xpyjv5s/bin/gfortran -I////////projects/aue/hpc/deploy/b306e83c/linux-rhel8-x86_64/gcc-12.1.0/openmpi-4.1.6-fa5w34i/include -pthread -I////////projects/aue/hpc/deploy/b306e83c/linux-rhel8-x86_64/gcc-12.1.0/openmpi-4.1.6-fa5w34i/lib -L////////projects/aue/hpc/deploy/b306e83c/linux-rhel8-x86_64/gcc-12.1.0/openmpi-4.1.6-fa5w34i/lib -L/projects/aue/hpc/deploy/b306e83c/linux-rhel8-x86_64/gcc-12.1.0/hwloc-2.9.3-ylujs7h/lib -L/projects/aue/hpc/deploy/b306e83c/linux-rhel8-x86_64/gcc-12.1.0/libevent-2.1.12-oiai2u4/lib -Wl,-rpath,/projects/aue/hpc/builds/x86_64/rhel8/ba17d7f2/gcc-12.1.0/install/linux-rhel8-x86_64/gcc-10.3.0/gcc-12.1.0-xpyjv5s/lib/gcc/x86_64-pc-linux-gnu/12.1.0 -Wl,-rpath,/projects/aue/hpc/builds/x86_64/rhel8/ba17d7f2/gcc-12.1.0/install/linux-rhel8-x86_64/gcc-10.3.0/gcc-12.1.0-xpyjv5s/lib64 -Wl,-rpath -Wl,////////projects/aue/hpc/deploy/b306e83c/linux-rhel8-x86_64/gcc-12.1.0/openmpi-4.1.6-fa5w34i/lib -Wl,-rpath -Wl,/projects/aue/hpc/deploy/b306e83c/linux-rhel8-x86_64/gcc-12.1.0/hwloc-2.9.3-ylujs7h/lib -Wl,-rpath -Wl,/projects/aue/hpc/deploy/b306e83c/linux-rhel8-x86_64/gcc-12.1.0/libevent-2.1.12-oiai2u4/lib -lmpi_usempif08 -lmpi_usempi_ignore_tkr -lmpi_mpifh -lmpi
F90 version:    GNU Fortran (Spack GCC) 12.1.0

OK

##################################################################################################
Python Sanity Checks

Verify `/usr/bin/env python3 --version` (reporting as: Python 3.11.8),
matches versions for: `which python3 && which python`

OK

##################################################################################################
Python Modules (TestHarness, run-ability)

OK

Answer selected by TheGreatCid

GiudGiud Nov 25, 2024
Collaborator

this looks nice and consistent to me.
they were indeed symlinks.

now does petsc build? and libmesh?

TheGreatCid Nov 25, 2024
Author

They do! Thank you for helping me sort this out

milljm Nov 25, 2024
Maintainer

This should have alerted as to an error:

##################################################################################################
Compiler(s) (CC CXX FC F77 F90):

CC=/projects/aue/hpc/builds/x86_64/rhel8/ba17d7f2/gcc-12.1.0/install/linux-rhel8-x86_64/gcc-10.3.0/gcc-12.1.0-xpyjv5s/bin/gcc
# <<<<<<  NO `-show` information. Supposed to WARN here as to not using an MPI wrapper.
CC version:     gcc (Spack GCC) 12.1.0

CXX=/projects/aue/hpc/builds/x86_64/rhel8/ba17d7f2/gcc-12.1.0/install/linux-rhel8-x86_64/gcc-10.3.0/gcc-12.1.0-xpyjv5s/bin/g++
# <<<<<<  Same as above
CXX version:    g++ (Spack GCC) 12.1.0

FC=/projects/aue/hpc/builds/x86_64/rhel8/ba17d7f2/gcc-12.1.0/install/linux-rhel8-x86_64/gcc-10.3.0/gcc-12.1.0-xpyjv5s/bin/gfortran
# <<<<<< etc...
FC version:     GNU Fortran (Spack GCC) 12.1.0

F77=/projects/aue/hpc/builds/x86_64/rhel8/ba17d7f2/gcc-12.1.0/install/linux-rhel8-x86_64/gcc-10.3.0/gcc-12.1.0-xpyjv5s/bin/gfortran
# <<<<<<
F77 version:    GNU Fortran (Spack GCC) 12.1.0

F90=/projects/aue/hpc/deploy/b306e83c/linux-rhel8-x86_64/gcc-12.1.0/openmpi-4.1.6-fa5w34i/bin/mpif90
F90 -show:
# <trimmed>
F90 version:    GNU Fortran (Spack GCC) 12.1.0

OK

I suppose by finding an MPI wrapper at the end, the script triggered an OK for MPI for the entire stack. Something we should fix...

Edit... indeed. I can reproduce that behavior:

export F90=gfortran
./diagnostrics
<trimmed>


which $F90; /home/milljm/miniforge3/envs/moose/bin/gfortran
$F90 --version; GNU Fortran (conda-forge gcc 12.3.0-7) 12.3.0
FAIL: gfortran appears not to be an MPI wrapper


FAIL: One or more compiler environment variables not set, or set incorrectly
################################################################################

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Another issue building libmesh on an HPC #29134

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 1 comment 20 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Select a reply

Another issue building libmesh on an HPC #29134

TheGreatCid Nov 25, 2024

Check these boxes if you have followed the posting rules.

Issue or question about MOOSE

(Optional) code in question / simulation log / errors

Encountering Errors? Please include diagnostic output

Replies: 1 comment · 20 replies

GiudGiud Nov 25, 2024 Collaborator

TheGreatCid Nov 25, 2024 Author

GiudGiud Nov 25, 2024 Collaborator

milljm Nov 25, 2024 Maintainer

TheGreatCid Nov 25, 2024 Author

GiudGiud Nov 25, 2024 Collaborator

TheGreatCid Nov 25, 2024 Author

milljm Nov 25, 2024 Maintainer

TheGreatCid
Nov 25, 2024

Replies: 1 comment 20 replies

GiudGiud
Nov 25, 2024
Collaborator

TheGreatCid Nov 25, 2024
Author

GiudGiud Nov 25, 2024
Collaborator

milljm Nov 25, 2024
Maintainer

TheGreatCid Nov 25, 2024
Author

GiudGiud Nov 25, 2024
Collaborator

TheGreatCid Nov 25, 2024
Author

milljm Nov 25, 2024
Maintainer