Petsc Floating Point Exception with large deltaT

[jcc242]

Hello,

Thank you for making this amazing library. I compiled the library on my system for OpenFOAM 8. I started from the of90 version of RheoTool and had to make a few changes to the source of RheoTool. I've tested several of the segregated solvers and was able to recover the validation results.

Then I tested the coupled PNP rheoEFoam solver: tutorials/rheoEFoam/ICEO/NernstPlanckCoupled. When I use the default deltaT, I get a floating point exception with Petsc:

[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: Caught signal number 8 FPE: Floating Point Exception,probably divide by zero
[0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
[0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
[0]PETSC ERROR: likely location of problem given in stack below
[0]PETSC ERROR: --------------------- Stack Frames ------------------------------------
[0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available,
[0]PETSC ERROR: INSTEAD the line number of the start of the function
[0]PETSC ERROR: is given.
[0]PETSC ERROR: [0] MatMult_SeqAIJ line 1298 /blues/gpfs/home/ssergien/soft/petsc-3.8.0/src/mat/impls/aij/seq/aij.c
[0]PETSC ERROR: [0] MatMult line 2299 /blues/gpfs/home/ssergien/soft/petsc-3.8.0/src/mat/interface/matrix.c
[0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[0]PETSC ERROR: Signal received
[0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[0]PETSC ERROR: Petsc Release Version 3.8.0, Sep, 26, 2017
[0]PETSC ERROR: Unknown Name on a linux-gnu-intel named knl-0266 by jchristopher Mon Feb 6 15:26:55 2023
[0]PETSC ERROR: Configure options --prefix=/soft/bebop/petsc/3.8.0 PETSC_ARCH=linux-gnu-intel --with-cc=mpiicc --with-cxx=mpiicpc --with-fc=mpiifort CFLAGS="-axCOMMON-AVX512,MIC-AVX512,CORE-AVX512,CORE-AVX2,AVX -fPIC" CXXFLAGS="-axCOMMON-AVX512,MIC-AVX512,CORE-AVX512,CORE-AVX2,AVX -fPIC" FFLAGS="-axCOMMON-AVX512,MIC-AVX512,CORE-AVX512,CORE-AVX2,AVX -fPIC" --with-metis-dir=/soft/spack-0.10.1/opt/spack/linux-centos7-x86_64/intel-17.0.4/metis-5.1.0-rw7ulwytxwujvf5pbv77m3iugqx67lz5 --with-parmetis-dir=/soft/spack-0.10.1/opt/spack/linux-centos7-x86_64/intel-17.0.4/parmetis-4.0.3-okja4ke2nxurblhoqdwtkyikg43gpce7 --with-hdf5-dir=/soft/spack-0.10.1/opt/spack/linux-centos7-x86_64/intel-17.0.4/hdf5-1.10.1-vzncl5hvkrdioypigebpug7uw3hbto5z --with-blaslapack-dir=/soft/spack-0.10.1/opt/spack/linux-centos7-x86_64/intel-17.0.4/intel-mkl-2017.3.196-jyjmyutcol3veyt4bazftyqkxxzmfd7o/pkg/mkl/lib/intel64_lin --with-netcdf-dir=/soft/spack-0.10.1/opt/spack/linux-centos7-x86_64/intel-17.0.4/netcdf-4.4.1.1-prsuusl3t5bfhbamqgobax2fxzvez6d2 --with-boost-dir=/soft/spack-0.10.1/opt/spack/linux-centos7-x86_64/intel-17.0.4/boost-1.63.0-yyrgc3rfpmaw7i32yrmu64lwzmb2crsk --with-superlu_dist-dir=/soft/spack-0.10.1/opt/spack/linux-centos7-x86_64/intel-17.0.4/superlu-dist-5.1.3-dljju447aeatnvhvmdizfjq7vxtwt7r2 --with-fftw-dir=/soft/spack-0.10.1/opt/spack/linux-centos7-x86_64/intel-17.0.4/fftw-3.3.7-bsjsf45ay6jwxautz7n3kcmztbt7bndw --with-scalapack-lib="-L/soft/spack-0.10.1/opt/spack/linux-centos7-x86_64/intel-17.0.4/intel-mkl-2017.3.196-jyjmyutcol3veyt4bazftyqkxxzmfd7o/pkg/mkl/lib/intel64_lin -lmkl_scalapack_lp64 -lmkl_blacs_intelmpi_lp64" --with-mumps-dir=/soft/spack-0.10.1/opt/spack/linux-centos7-x86_64/intel-17.0.4/mumps-5.1.1-6pxszht7dou6e4l3xzqjwrrbaltryjp3
[0]PETSC ERROR: #1 User provided function() line 0 in unknown file
application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0
[unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=59
:
system msg for write_line failure : Bad file descriptor

This is using CentOS 7.0 using Petsc 3.8.0 installed by the system admin. I had to compile OpenFOAM and RheoTool with the Intel 17.04 compiler. When I decrease the deltaT to less than 4.e-5, then the solver progresses.

Have you seen this error before and know how to solve with a larger deltaT? Perhaps a different preconditioner or linear solve approach is required?

Thank you!

[fppimenta]

OpenFOAM v 8.0 is not supported, so that I cannot really provide any guidance. Some changes that you have done might be behind what you see. However, as a first go I would just use a more recent version of petsc. The tutorial must run with the settings it has (the fact that it runs with a lower time step does not mean that the issue is fixed).

[jcc242]

Hi, thank you for the response! I upgraded petsc to 3.16.5 and unfortunately that didn't fix things. So I upgraded to OpenFOAM v9.0 and rebuilt everything with the GNU compilers and OpenMPI 4 and petsc 3.16.5. The tutorial case I mentioned originally is now working with the original setting (i.e. time step size of 0.01). I'll upgrade my solver to OFv9.0 and try to implement the coupled equation solver from RheoTool. Thank you again for making this library and your help!