configure error for submodule libmesh with external petsc on cluster

[BoZeng1997]

Hi,

I am trying to configure the submodule libmesh inside moose with an external PETSc on cluster. The external PETSc was successfully installed and passed all test provided by PETSc. I have specified the correct PETSC_DIR when running ./update_and_rebuild_libmesh.sh in moose/scripts and PETSc was shown found. However, this is the error I got:

<<< Found PETSc 3.17.2 installation in /hpc/group/dolbowlab/bz75/petsc ... >>>
checking whether we can compile a trivial PETSc program... no
checking for TAO support via PETSc... no
configure: error: *** PETSc was not found, but --enable-petsc-required was specified.

The details related to PETSc not found in moose/libmesh/build/config.log are pasted below. I do have the missing files in my $PETSC_DIR/include.

(There are other error in the log file but I am not sure if they are the reason the configure was aborted. I can provide them if needed.)

configure:49429: result: <<< Found PETSc 3.17.2 installation in /hpc/group/dolbowlab/bz75/petsc ... >>>
configure:49439: checking whether we can compile a trivial PETSc program
configure:49470: mpicxx -std=gnu++17 -c     conftest.cpp >&5
conftest.cpp:155:20: fatal error: petsc.h: No such file or directory
  155 |           #include <petsc.h>
      |                    ^~~~~~~~~
compilation terminated.
configure:49470: $? = 1
configure: failed program was:
| /* confdefs.h */
| #define PACKAGE_NAME "libmesh"
| #define PACKAGE_TARNAME "libmesh"
| #define PACKAGE_VERSION "1.7.0-pre"
| #define PACKAGE_STRING "libmesh 1.7.0-pre"
| #define PACKAGE_BUGREPORT "https://github.com/libMesh/libmesh/discussions"
| #define PACKAGE_URL "http://libmesh.github.io"
| #define CONFIGURE_INFO "../configure  '--enable-silent-rules' '--enable-unique-id' '--disable-warnings' '--with-thread-model=openmp' '--disable-maintainer-mode' '--enable-hdf5' '--enable-petsc-hypre-required' '--enable-metaphysicl-required' '--with-cxx-std-min=2014' '--without-gdb-command' '--with-methods=opt oprof devel dbg' '--prefix=/hpc/group/dolbowlab/bz75/projects/raccoon/moose/scripts/../libmesh/installed' '--with-future-timpi-dir=/hpc/group/dolbowlab/bz75/projects/raccoon/moose/scripts/../libmesh/installed' 'INSTALL=/hpc/group/dolbowlab/bz75/projects/raccoon/moose/scripts/../libmesh/build-aux/install-sh -C' 'PETSC_DIR=/hpc/group/dolbowlab/bz75/petsc' 'PETSC_ARCH=/hpc/group/dolbowlab/bz75/petsc/arch-linux-c-opt' 'METHODS=opt oprof devel dbg' 'SLEPC_DIR=/hpc/group/dolbowlab/bz75/petsc/arch-linux-c-opt'"
| #define PACKAGE "libmesh"
| #define VERSION "1.7.0-pre"
| #define HAVE_CXX17 1
| #define HAVE_CXX11_AUTO 1
| #define HAVE_CXX11_RANGEFOR 1
| #define HAVE_CXX11_INITIALIZER_LIST 1
| #define HAVE_CXX11_UNIQUE_PTR 1
| #define HAVE_CXX11_MAKE_UNIQUE_WORKAROUND 1
| #define HAVE_CXX11_TUPLE 1
| #define HAVE_CXX11_LAMBDA 1
| #define HAVE_CXX11_FIXED_TYPE_ENUM 1
| #define HAVE_CXX11_FIXED_TYPE_ENUM_FWD 1
| #define HAVE_CXX11_OVERRIDE 1
| #define HAVE_CXX11_MOVE_CONSTRUCTORS 1
| #define HAVE_CXX11_DELETED_FUNCTIONS 1
| #define HAVE_CXX11_DEFAULTED_FUNCTIONS 1
| #define HAVE_CXX11_NULLPTR 1
| #define HAVE_CXX11_FINAL 1
| #define HAVE_CXX11_DECLTYPE 1
| #define HAVE_CXX11_SHARED_PTR 1
| #define HAVE_CXX11_RVALUE_REFERENCES 1
| #define HAVE_CXX11_TO_STRING 1
| #define HAVE_CXX11_CONSTEXPR 1
| #define HAVE_CXX11_VARIADIC_TEMPLATES 1
| #define HAVE_CXX11_ALIAS_DECLARATIONS 1
| #define HAVE_CXX17_SPLICING 1
| #define HAVE_CXX14 1
| #define HAVE_CXX11 1
| #define HAVE_CXX14_MAKE_UNIQUE 1
| #define HAVE_CXX11_REGEX 1
| #define HAVE_CXX11_THREAD 1
| #define HAVE_CXX11_CONDITION_VARIABLE 1
| #define HAVE_CXX11_TYPE_TRAITS 1
| #define HAVE_CXX11_INVERSE_HYPERBOLIC_SINE 1
| #define HAVE_CXX11_INVERSE_HYPERBOLIC_COSINE 1
| #define HAVE_CXX11_INVERSE_HYPERBOLIC_TANGENT 1
| #define HAVE_CXX11_INVERSE_HYPERBOLIC_SINE_COMPLEX 1
| #define HAVE_CXX11_INVERSE_HYPERBOLIC_COSINE_COMPLEX 1
| #define HAVE_CXX11_INVERSE_HYPERBOLIC_TANGENT_COMPLEX 1
| #define HAVE_CXX11_ERF 1
| #define HAVE_CXX17_FALLTHROUGH_ATTRIBUTE_BUT_DISABLED 1
| #define HAVE_DOUBLE_UNDERSCORE_ATTRIBUTE_FALLTHROUGH 1
| #define STDC_HEADERS 1
| #define HAVE_SYS_TYPES_H 1
| #define HAVE_SYS_STAT_H 1
| #define HAVE_STDLIB_H 1
| #define HAVE_STRING_H 1
| #define HAVE_MEMORY_H 1
| #define HAVE_STRINGS_H 1
| #define HAVE_INTTYPES_H 1
| #define HAVE_STDINT_H 1
| #define HAVE_UNISTD_H 1
| #define HAVE_DLFCN_H 1
| #define LT_OBJDIR ".libs/"
| #define MAJOR_VERSION 1
| #define MINOR_VERSION 7
| #define MICRO_VERSION 0
| #define LIB_VERSION "1.7.0-pre"
| #define LIB_RELEASE ""
| #define CXX "mpicxx -std=gnu++17"
| #define IO_COMPATIBILITY_VERSION "1.7.0"
| #define HAVE_NAMESPACES /**/
| #define DEFAULT_DOUBLE_PRECISION 1
| #define DEFAULT_SCALAR_TYPE double
| #define restrict __restrict
| #define HAVE_PWD_H 1
| #define HAVE_GETPWUID 1
| #define ENABLE_EXCEPTIONS 1
| #define ENABLE_TIMESTAMPS 1
| #define SIZEOF_SHORT_INT 2
| #define SIZEOF_INT 4
| #define SIZEOF_UNSIGNED_INT 4
| #define SIZEOF_SIZE_T 8
| #define SIZEOF_LONG_INT 8
| #define SIZEOF_FLOAT 4
| #define SIZEOF_DOUBLE 8
| #define SIZEOF_VOID_P 8
| #define SIZEOF_FUNCTION_POINTER 8
| #define HAVE_RTTI /**/
| #define HAVE_GETOPT_H 1
| #define HAVE_SYS_TIME_H 1
| #define HAVE_CSIGNAL 1
| #define HAVE_SYS_RESOURCE_H 1
| #define HAVE_LOCALE /**/
| #define HAVE_SSTREAM /**/
| #define HAVE_STRINGSTREAM /**/
| #define HAVE_FENV_H 1
| #define HAVE_XMMINTRIN_H 1
| #define HAVE_FEENABLEEXCEPT 1
| #define HAVE_FEDISABLEEXCEPT 1
| #define HAVE_DECL_SIGACTION 1
| #define HAVE_MKDIR /**/
| #define HAVE_DECL__MKDIR 0
| #define HAVE_MKSTEMP 1
| #define HAVE_GETTIMEOFDAY 1
| #define HAVE_SYS_UTSNAME_H 1
| #define HAVE_STD_UNORDERED_MULTIMAP 1
| #define BEST_UNORDERED_MULTIMAP std::unordered_multimap
| #define INCLUDE_UNORDERED_MULTIMAP <unordered_map>
| #define HAVE_STD_UNORDERED_MAP 1
| #define BEST_UNORDERED_MAP std::unordered_map
| #define INCLUDE_UNORDERED_MAP <unordered_map>
| #define HAVE_STD_UNORDERED_MULTISET 1
| #define BEST_UNORDERED_MULTISET std::unordered_multiset
| #define INCLUDE_UNORDERED_MULTISET <unordered_set>
| #define HAVE_STD_UNORDERED_SET 1
| #define BEST_UNORDERED_SET std::unordered_set
| #define INCLUDE_UNORDERED_SET <unordered_set>
| #define HAVE_STD_HASH 1
| #define BEST_HASH std::hash
| #define INCLUDE_HASH <functional>
| #define DEFINE_HASH_STRING /**/
| #define DEFINE_HASH_POINTERS /**/
| #define HAVE_DLOPEN 1
| #define HAVE_GCC_ABI_DEMANGLE 1
| #define HAVE_GLIBC_BACKTRACE 1
| #define GDB_COMMAND "no"
| #define ENABLE_UNIQUE_PTR 1
| #define ENABLE_DEPRECATED 1
| #define FORWARD_DECLARE_ENUMS 1
| #define REQUIRE_SEPARATE_NAMESPACE 1
| #define BOUNDARY_ID_BYTES 2
| #define DOF_ID_BYTES 4
| #define PROCESSOR_ID_BYTES 4
| #define SUBDOMAIN_ID_BYTES 2
| #define ENABLE_UNIQUE_ID 1
| #define UNIQUE_ID_BYTES 8
| #define ENABLE_AMR 1
| #define ENABLE_VSMOOTHER 1
| #define ENABLE_PERIODIC 1
| #define ENABLE_DIRICHLET 1
| #define MAPVECTOR_CHUNK_SIZE 1
| #define ENABLE_GHOSTED 1
| #define ENABLE_NODE_VALENCE 1
| #define DIM 3
| #define ENABLE_HIGHER_ORDER_SHAPES 1
| #define ENABLE_SECOND_DERIVATIVES 1
| #define USE_REAL_NUMBERS 1
| #define ENABLE_REFERENCE_COUNTING 1
| #define HAVE_BOOST 1
| #define HAVE_BOOST_MOVELIB_UNIQUE_PTR 1
| #define HAVE_MPI 1
| /* end confdefs.h.  */
| 
|           #undef MAJOR_VERSION
|           #undef MINOR_VERSION
|           #include <petsc.h>
|           static char help[]="";
| 
|           int main(int argc, char **argv)
|           {
|             PetscInitialize(&argc, &argv, (char*)0,help);
|             PetscFinalize();
|             return 0;
|           }
| 
configure:49477: result: no
configure:49497: checking for TAO support via PETSc
configure:49531: mpicc -c     conftest.c >&5
conftest.c:155:20: fatal error: petsctao.h: No such file or directory
  155 |           #include <petsctao.h>
      |                    ^~~~~~~~~~~~
compilation terminated.
configure:49531: $? = 1
configure: failed program was:
| /* confdefs.h */
| #define PACKAGE_NAME "libmesh"
| #define PACKAGE_TARNAME "libmesh"
| #define PACKAGE_VERSION "1.7.0-pre"
| #define PACKAGE_STRING "libmesh 1.7.0-pre"
| #define PACKAGE_BUGREPORT "https://github.com/libMesh/libmesh/discussions"
| #define PACKAGE_URL "http://libmesh.github.io"
| #define CONFIGURE_INFO "../configure  '--enable-silent-rules' '--enable-unique-id' '--disable-warnings' '--with-thread-model=openmp' '--disable-maintainer-mode' '--enable-hdf5' '--enable-petsc-hypre-required' '--enable-metaphysicl-required' '--with-cxx-std-min=2014' '--without-gdb-command' '--with-methods=opt oprof devel dbg' '--prefix=/hpc/group/dolbowlab/bz75/projects/raccoon/moose/scripts/../libmesh/installed' '--with-future-timpi-dir=/hpc/group/dolbowlab/bz75/projects/raccoon/moose/scripts/../libmesh/installed' 'INSTALL=/hpc/group/dolbowlab/bz75/projects/raccoon/moose/scripts/../libmesh/build-aux/install-sh -C' 'PETSC_DIR=/hpc/group/dolbowlab/bz75/petsc' 'PETSC_ARCH=/hpc/group/dolbowlab/bz75/petsc/arch-linux-c-opt' 'METHODS=opt oprof devel dbg' 'SLEPC_DIR=/hpc/group/dolbowlab/bz75/petsc/arch-linux-c-opt'"
| #define PACKAGE "libmesh"
| #define VERSION "1.7.0-pre"
| #define HAVE_CXX17 1
| #define HAVE_CXX11_AUTO 1
| #define HAVE_CXX11_RANGEFOR 1
| #define HAVE_CXX11_INITIALIZER_LIST 1
| #define HAVE_CXX11_UNIQUE_PTR 1
| #define HAVE_CXX11_MAKE_UNIQUE_WORKAROUND 1
| #define HAVE_CXX11_TUPLE 1
| #define HAVE_CXX11_LAMBDA 1
| #define HAVE_CXX11_FIXED_TYPE_ENUM 1
| #define HAVE_CXX11_FIXED_TYPE_ENUM_FWD 1
| #define HAVE_CXX11_OVERRIDE 1
| #define HAVE_CXX11_MOVE_CONSTRUCTORS 1
| #define HAVE_CXX11_DELETED_FUNCTIONS 1
| #define HAVE_CXX11_DEFAULTED_FUNCTIONS 1
| #define HAVE_CXX11_NULLPTR 1
| #define HAVE_CXX11_FINAL 1
| #define HAVE_CXX11_DECLTYPE 1
| #define HAVE_CXX11_SHARED_PTR 1
| #define HAVE_CXX11_RVALUE_REFERENCES 1
| #define HAVE_CXX11_TO_STRING 1
| #define HAVE_CXX11_CONSTEXPR 1
| #define HAVE_CXX11_VARIADIC_TEMPLATES 1
| #define HAVE_CXX11_ALIAS_DECLARATIONS 1
| #define HAVE_CXX17_SPLICING 1
| #define HAVE_CXX14 1
| #define HAVE_CXX11 1
| #define HAVE_CXX14_MAKE_UNIQUE 1
| #define HAVE_CXX11_REGEX 1
| #define HAVE_CXX11_THREAD 1
| #define HAVE_CXX11_CONDITION_VARIABLE 1
| #define HAVE_CXX11_TYPE_TRAITS 1
| #define HAVE_CXX11_INVERSE_HYPERBOLIC_SINE 1
| #define HAVE_CXX11_INVERSE_HYPERBOLIC_COSINE 1
| #define HAVE_CXX11_INVERSE_HYPERBOLIC_TANGENT 1
| #define HAVE_CXX11_INVERSE_HYPERBOLIC_SINE_COMPLEX 1
| #define HAVE_CXX11_INVERSE_HYPERBOLIC_COSINE_COMPLEX 1
| #define HAVE_CXX11_INVERSE_HYPERBOLIC_TANGENT_COMPLEX 1
| #define HAVE_CXX11_ERF 1
| #define HAVE_CXX17_FALLTHROUGH_ATTRIBUTE_BUT_DISABLED 1
| #define HAVE_DOUBLE_UNDERSCORE_ATTRIBUTE_FALLTHROUGH 1
| #define STDC_HEADERS 1
| #define HAVE_SYS_TYPES_H 1
| #define HAVE_SYS_STAT_H 1
| #define HAVE_STDLIB_H 1
| #define HAVE_STRING_H 1
| #define HAVE_MEMORY_H 1
| #define HAVE_STRINGS_H 1
| #define HAVE_INTTYPES_H 1
| #define HAVE_STDINT_H 1
| #define HAVE_UNISTD_H 1
| #define HAVE_DLFCN_H 1
| #define LT_OBJDIR ".libs/"
| #define MAJOR_VERSION 1
| #define MINOR_VERSION 7
| #define MICRO_VERSION 0
| #define LIB_VERSION "1.7.0-pre"
| #define LIB_RELEASE ""
| #define CXX "mpicxx -std=gnu++17"
| #define IO_COMPATIBILITY_VERSION "1.7.0"
| #define HAVE_NAMESPACES /**/
| #define DEFAULT_DOUBLE_PRECISION 1
| #define DEFAULT_SCALAR_TYPE double
| #define restrict __restrict
| #define HAVE_PWD_H 1
| #define HAVE_GETPWUID 1
| #define ENABLE_EXCEPTIONS 1
| #define ENABLE_TIMESTAMPS 1
| #define SIZEOF_SHORT_INT 2
| #define SIZEOF_INT 4
| #define SIZEOF_UNSIGNED_INT 4
| #define SIZEOF_SIZE_T 8
| #define SIZEOF_LONG_INT 8
| #define SIZEOF_FLOAT 4
| #define SIZEOF_DOUBLE 8
| #define SIZEOF_VOID_P 8
| #define SIZEOF_FUNCTION_POINTER 8
| #define HAVE_RTTI /**/
| #define HAVE_GETOPT_H 1
| #define HAVE_SYS_TIME_H 1
| #define HAVE_CSIGNAL 1
| #define HAVE_SYS_RESOURCE_H 1
| #define HAVE_LOCALE /**/
| #define HAVE_SSTREAM /**/
| #define HAVE_STRINGSTREAM /**/
| #define HAVE_FENV_H 1
| #define HAVE_XMMINTRIN_H 1
| #define HAVE_FEENABLEEXCEPT 1
| #define HAVE_FEDISABLEEXCEPT 1
| #define HAVE_DECL_SIGACTION 1
| #define HAVE_MKDIR /**/
| #define HAVE_DECL__MKDIR 0
| #define HAVE_MKSTEMP 1
| #define HAVE_GETTIMEOFDAY 1
| #define HAVE_SYS_UTSNAME_H 1
| #define HAVE_STD_UNORDERED_MULTIMAP 1
| #define BEST_UNORDERED_MULTIMAP std::unordered_multimap
| #define INCLUDE_UNORDERED_MULTIMAP <unordered_map>
| #define HAVE_STD_UNORDERED_MAP 1
| #define BEST_UNORDERED_MAP std::unordered_map
| #define INCLUDE_UNORDERED_MAP <unordered_map>
| #define HAVE_STD_UNORDERED_MULTISET 1
| #define BEST_UNORDERED_MULTISET std::unordered_multiset
| #define INCLUDE_UNORDERED_MULTISET <unordered_set>
| #define HAVE_STD_UNORDERED_SET 1
| #define BEST_UNORDERED_SET std::unordered_set
| #define INCLUDE_UNORDERED_SET <unordered_set>
| #define HAVE_STD_HASH 1
| #define BEST_HASH std::hash
| #define INCLUDE_HASH <functional>
| #define DEFINE_HASH_STRING /**/
| #define DEFINE_HASH_POINTERS /**/
| #define HAVE_DLOPEN 1
| #define HAVE_GCC_ABI_DEMANGLE 1
| #define HAVE_GLIBC_BACKTRACE 1
| #define GDB_COMMAND "no"
| #define ENABLE_UNIQUE_PTR 1
| #define ENABLE_DEPRECATED 1
| #define FORWARD_DECLARE_ENUMS 1
| #define REQUIRE_SEPARATE_NAMESPACE 1
| #define BOUNDARY_ID_BYTES 2
| #define DOF_ID_BYTES 4
| #define PROCESSOR_ID_BYTES 4
| #define SUBDOMAIN_ID_BYTES 2
| #define ENABLE_UNIQUE_ID 1
| #define UNIQUE_ID_BYTES 8
| #define ENABLE_AMR 1
| #define ENABLE_VSMOOTHER 1
| #define ENABLE_PERIODIC 1
| #define ENABLE_DIRICHLET 1
| #define MAPVECTOR_CHUNK_SIZE 1
| #define ENABLE_GHOSTED 1
| #define ENABLE_NODE_VALENCE 1
| #define DIM 3
| #define ENABLE_HIGHER_ORDER_SHAPES 1
| #define ENABLE_SECOND_DERIVATIVES 1
| #define USE_REAL_NUMBERS 1
| #define ENABLE_REFERENCE_COUNTING 1
| #define HAVE_BOOST 1
| #define HAVE_BOOST_MOVELIB_UNIQUE_PTR 1
| #define HAVE_MPI 1
| /* end confdefs.h.  */
| 
|           #undef MAJOR_VERSION
|           #undef MINOR_VERSION
|           #include <petsctao.h>
|           static char help[]="";
| 
|           int main(int argc, char **argv)
|           {
|             Tao tao;
|             PetscInitialize(&argc, &argv, (char*)0,help);
|             TaoCreate(PETSC_COMM_WORLD,&tao);
|             TaoDestroy(&tao);
|             PetscFinalize();
|             return 0;
|           }
| 
configure:49539: result: no
configure:49698: error: *** PETSc was not found, but --enable-petsc-required was specified.

[milljm]

Can you attach the libmesh_diagnostic.log which should reside in moose/scripts? It contains additional information that might help shed some light on the issue. If that file does not exist, please re-run moose/scripts/update_and_rebuild_libmesh.sh using the same environment you plan on having it succeed in.

Also, can you provide the results of an ldd on a PETSc library you are using (this will help us understand what compiler was being used to build):

ldd $PETSC_DIR/lib/libpetsc.so

[BoZeng1997]

After setting the compilers and $PETSC_ARCH, update_and_rebuild_libmesh.sh ran to complete. However, I got many warnings during the compilation. Here are the logs.
libmesh_diagnostic.log
config.log

When I continued to compile a custom branch of moose, I got the following errors

Compiling C++ (in opt mode) /hpc/group/dolbowlab/bz75/projects/raccoon/moose/framework/src/utils/PetscDMMoose.C...
/hpc/group/dolbowlab/bz75/projects/raccoon/moose/framework/src/utils/PetscDMMoose.C:104:13: error: Use SETERRQ() (since version 3.17) [-Werror]
  104 |              DMMOOSE);
      |             ^~~~~~~~~~                          
/hpc/group/dolbowlab/bz75/projects/raccoon/moose/framework/src/utils/PetscDMMoose.C:133:13: error: Use SETERRQ() (since version 3.17) [-Werror]
  133 |              DMMOOSE);
      |             ^~~~~~~~~~                          
/hpc/group/dolbowlab/bz75/projects/raccoon/moose/framework/src/utils/PetscDMMoose.C:160:13: error: Use SETERRQ() (since version 3.17) [-Werror]
  160 |              DMMOOSE);
      |             ^~~~~~~~~~                          
/hpc/group/dolbowlab/bz75/projects/raccoon/moose/framework/src/utils/PetscDMMoose.C:184:13: error: Use SETERRQ() (since version 3.17) [-Werror]
  184 |              DMMOOSE);
      |             ^~~~~~~~~~                          
/hpc/group/dolbowlab/bz75/projects/raccoon/moose/framework/src/utils/PetscDMMoose.C:208:13: error: Use SETERRQ() (since version 3.17) [-Werror]
  208 |              DMMOOSE);
      |             ^~~~~~~~~~                          
/hpc/group/dolbowlab/bz75/projects/raccoon/moose/framework/src/utils/PetscDMMoose.C:232:13: error: Use SETERRQ() (since version 3.17) [-Werror]
  232 |              DMMOOSE);
      |             ^~~~~~~~~~                          
/hpc/group/dolbowlab/bz75/projects/raccoon/moose/framework/src/utils/PetscDMMoose.C:256:13: error: Use SETERRQ() (since version 3.17) [-Werror]
  256 |              DMMOOSE);
      |             ^~~~~~~~~~                          
/hpc/group/dolbowlab/bz75/projects/raccoon/moose/framework/src/utils/PetscDMMoose.C:284:13: error: Use SETERRQ() (since version 3.17) [-Werror]
  284 |              DMMOOSE);
      |             ^~~~~~~~~~                          
/hpc/group/dolbowlab/bz75/projects/raccoon/moose/framework/src/utils/PetscDMMoose.C:311:13: error: Use SETERRQ() (since version 3.17) [-Werror]
  311 |              DMMOOSE);
      |             ^~~~~~~~~~                          
/hpc/group/dolbowlab/bz75/projects/raccoon/moose/framework/src/utils/PetscDMMoose.C:338:13: error: Use SETERRQ() (since version 3.17) [-Werror]
  338 |              DMMOOSE);
      |             ^~~~~~~~~~                          
/hpc/group/dolbowlab/bz75/projects/raccoon/moose/framework/src/utils/PetscDMMoose.C:365:13: error: Use SETERRQ() (since version 3.17) [-Werror]
  365 |              DMMOOSE);
      |             ^~~~~~~~~~                          
/hpc/group/dolbowlab/bz75/projects/raccoon/moose/framework/src/utils/PetscDMMoose.C:394:13: error: Use SETERRQ() (since version 3.17) [-Werror]
  394 |              DMMOOSE);
      |             ^~~~~~~~~~                          
/hpc/group/dolbowlab/bz75/projects/raccoon/moose/framework/src/utils/PetscDMMoose.C:402:13: error: Use SETERRQ() (since version 3.17) [-Werror]
  402 |              displaced.size());
      |             ^~~~~~~~~~~~~~~~~~~                 
/hpc/group/dolbowlab/bz75/projects/raccoon/moose/framework/src/utils/PetscDMMoose.C:435:13: error: Use SETERRQ() (since version 3.17) [-Werror]
  435 |              DMMOOSE);
      |             ^~~~~~~~~~                          
/hpc/group/dolbowlab/bz75/projects/raccoon/moose/framework/src/utils/PetscDMMoose.C:443:13: error: Use SETERRQ() (since version 3.17) [-Werror]
  443 |              displaced.size());
      |             ^~~~~~~~~~~~~~~~~~~                 
/hpc/group/dolbowlab/bz75/projects/raccoon/moose/framework/src/utils/PetscDMMoose.C:473:13: error: Use SETERRQ() (since version 3.17) [-Werror]
  473 |              DMMOOSE);
      |             ^~~~~~~~~~                          
/hpc/group/dolbowlab/bz75/projects/raccoon/moose/framework/src/utils/PetscDMMoose.C:496:13: error: Use SETERRQ() (since version 3.17) [-Werror]
  496 |              DMMOOSE);
      |             ^~~~~~~~~~                          
/hpc/group/dolbowlab/bz75/projects/raccoon/moose/framework/src/utils/PetscDMMoose.C:547:13: error: Use SETERRQ() (since version 3.17) [-Werror]
  547 |              DMMOOSE);
      |             ^~~~~~~~~~                          
/hpc/group/dolbowlab/bz75/projects/raccoon/moose/framework/src/utils/PetscDMMoose.C:1052:13: error: Use SETERRQ() (since version 3.17) [-Werror]
 1052 |             SETERRQ1(((PetscObject)dm)->comm, PETSC_ERR_PLIB, "Failed to embed split %D", d);
      |             ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/hpc/group/dolbowlab/bz75/projects/raccoon/moose/framework/src/utils/PetscDMMoose.C:1357:13: error: Use SETERRQ() (since version 3.17) [-Werror]
 1357 |              DMMOOSE);
      |             ^~~~~~~~~~                          
/hpc/group/dolbowlab/bz75/projects/raccoon/moose/framework/src/utils/PetscDMMoose.C:1417:13: error: Use SETERRQ() (since version 3.17) [-Werror]
 1417 |              DMMOOSE);
      |             ^~~~~~~~~~                          
/hpc/group/dolbowlab/bz75/projects/raccoon/moose/framework/src/utils/PetscDMMoose.C:1621:13: error: Use SETERRQ() (since version 3.17) [-Werror]
 1621 |              DMMOOSE);
      |             ^~~~~~~~~~                          
/hpc/group/dolbowlab/bz75/projects/raccoon/moose/framework/src/utils/PetscDMMoose.C:1653:13: error: Use SETERRQ() (since version 3.17) [-Werror]
 1653 |              DMMOOSE);
      |             ^~~~~~~~~~                          
/hpc/group/dolbowlab/bz75/projects/raccoon/moose/framework/src/utils/PetscDMMoose.C:1923:13: error: Use SETERRQ() (since version 3.17) [-Werror]
 1923 |              DMMOOSE);
      |             ^~~~~~~~~~                          
/hpc/group/dolbowlab/bz75/projects/raccoon/moose/framework/src/utils/PetscDMMoose.C:2014:13: error: Use SETERRQ() (since version 3.17) [-Werror]
 2014 |              DMMOOSE);
      |             ^~~~~~~~~~                          
/hpc/group/dolbowlab/bz75/projects/raccoon/moose/framework/src/utils/PetscDMMoose.C:2139:13: error: Use SETERRQ() (since version 3.17) [-Werror]
 2139 |              maxcontacts);
      |             ^~~~~~~~~~~~~~                      
/hpc/group/dolbowlab/bz75/projects/raccoon/moose/framework/src/utils/PetscDMMoose.C:2160:13: error: Use SETERRQ() (since version 3.17) [-Werror]
 2160 |                  sz);
      |             ^    ~~~~                           
/hpc/group/dolbowlab/bz75/projects/raccoon/moose/framework/src/utils/PetscDMMoose.C:2220:13: error: Use SETERRQ() (since version 3.17) [-Werror]
 2220 |              maxcontacts);
      |             ^~~~~~~~~~~~~~                      
/hpc/group/dolbowlab/bz75/projects/raccoon/moose/framework/src/utils/PetscDMMoose.C:2241:13: error: Use SETERRQ() (since version 3.17) [-Werror]
 2241 |                  sz);
      |             ^    ~~~~                           
/hpc/group/dolbowlab/bz75/projects/raccoon/moose/framework/src/utils/PetscDMMoose.C:2307:13: error: Use SETERRQ() (since version 3.17) [-Werror]
 2307 |                nnsplits);
      |             ^  ~~~~~~~~~~                       
cc1plus: all warnings being treated as errors
make: *** [/hpc/group/dolbowlab/bz75/projects/raccoon/moose/framework/build.mk:145: /hpc/group/dolbowlab/bz75/projects/raccoon/moose/framework/src/utils/PetscDMMoose.x86_64-pc-linux-gnu.opt.lo] Error 1

I see that moose is now using submodule PETSc-3.16.5. Do you think my latest error is from a too advance PETSc-3.17.2?

[GiudGiud]

why are all warnings treated as errors? What are the compile flags?

3.17.2 is not officially supported yet.

[BoZeng1997]

The compile flags did treat all warning as error.
The warnings are gone after I switched to PETSc-3.16.5.

[milljm]

MOOSE will use what-ever-petsc libMesh was built against. And libMesh will either use what-ever PETSC_DIR is set to, or submodule petsc... But it would appear you have things running. So, your call on where we head next :)

While latest PETSc isn't supported, we applaud attempts and reporting back any fixes you may find/solve along the way!

[BoZeng1997]

I think this issue is solved.
However, I compiled this stand alone PETSc in hope to fix another issue, which remains an issue now. I am following the debug guide to produce backtrace. I will open a new discussion soon.
Thank you!