hypre-multiprecision build #850

oseikuffuor1 · 2023-03-10T02:45:02Z

This PR updates various directories in hypre to support multiprecision build.

The build system is modified to include the --enable-mixed-precision option to prescribe the multiprecision build
New preprocessor macros have been added to facilitate automatic code transformation
Updates to hypre directories to effect multiprecision build.

** This PR is currently defined for building hypre with configure. Cmake support will be added later. Also, GPU support is incomplete for now.

src/configure

victorapm · 2023-03-10T02:59:11Z

Thanks for all this work, Daniel!

I'm trying to build hypre via ./configure --enable-mixed-precision --enable-debug, however, I'm getting the error below. Maybe I'm missing something?

processing C object MuP_obj.dir/utilities_flt/F90_HYPRE_error.o ...
mpicc -g -Wall  -DHAVE_CONFIG_H -I.. -I./.. -I./../struct_mv -I.           -DMP_BUILD_SINGLE=1 -c -o MuP_obj.dir/utilities_flt/F90_HYPRE_error.o F90_HYPRE_error.c
F90_HYPRE_error.c:19:30: error: implicit declaration of function 'HYPRE_GetError_flt' is invalid in C99 [-Werror,-Wimplicit-function-declaration]
   *result = (hypre_F90_Int) HYPRE_GetError();
                             ^
./utilities_mup_func.h:65:24: note: expanded from macro 'HYPRE_GetError'
#define HYPRE_GetError HYPRE_MULTIPRECISION_FUNC ( HYPRE_GetError )
                       ^
./multiprecision.h:72:38: note: expanded from macro 'HYPRE_MULTIPRECISION_FUNC'
#define HYPRE_MULTIPRECISION_FUNC(a) CONCAT_(a, FLT_SUFFIX)
                                     ^
./multiprecision.h:18:23: note: expanded from macro 'CONCAT_'
#define CONCAT_(a, b) CONCAT2_(a, b)
                      ^
./multiprecision.h:17:24: note: expanded from macro 'CONCAT2_'
#define CONCAT2_(a, b) a ## _ ## b
                       ^
<scratch space>:64:1: note: expanded from here
HYPRE_GetError_flt
^
F90_HYPRE_error.c:28:30: error: implicit declaration of function 'HYPRE_CheckError_flt' is invalid in C99 [-Werror,-Wimplicit-function-declaration]
   *result = (hypre_F90_Int) HYPRE_CheckError(
                             ^
./utilities_mup_func.h:61:26: note: expanded from macro 'HYPRE_CheckError'
#define HYPRE_CheckError HYPRE_MULTIPRECISION_FUNC ( HYPRE_CheckError )
                         ^
./multiprecision.h:72:38: note: expanded from macro 'HYPRE_MULTIPRECISION_FUNC'
#define HYPRE_MULTIPRECISION_FUNC(a) CONCAT_(a, FLT_SUFFIX)
                                     ^
./multiprecision.h:18:23: note: expanded from macro 'CONCAT_'
#define CONCAT_(a, b) CONCAT2_(a, b)
                      ^
./multiprecision.h:17:24: note: expanded from macro 'CONCAT2_'
#define CONCAT2_(a, b) a ## _ ## b
                       ^
<scratch space>:69:1: note: expanded from here
HYPRE_CheckError_flt
^
F90_HYPRE_error.c:28:30: note: did you mean 'HYPRE_CheckError'?
./utilities_mup_func.h:61:26: note: expanded from macro 'HYPRE_CheckError'
#define HYPRE_CheckError HYPRE_MULTIPRECISION_FUNC ( HYPRE_CheckError )
                         ^
./multiprecision.h:72:38: note: expanded from macro 'HYPRE_MULTIPRECISION_FUNC'
#define HYPRE_MULTIPRECISION_FUNC(a) CONCAT_(a, FLT_SUFFIX)
                                     ^
./multiprecision.h:18:23: note: expanded from macro 'CONCAT_'
#define CONCAT_(a, b) CONCAT2_(a, b)
                      ^
./multiprecision.h:17:24: note: expanded from macro 'CONCAT2_'
#define CONCAT2_(a, b) a ## _ ## b
                       ^
<scratch space>:69:1: note: expanded from here
HYPRE_CheckError_flt
^
./HYPRE_utilities.h:144:11: note: 'HYPRE_CheckError' declared here
HYPRE_Int HYPRE_CheckError(HYPRE_Int hypre_ierr, HYPRE_Int hypre_error_code);
          ^
F90_HYPRE_error.c:37:30: error: implicit declaration of function 'HYPRE_GetErrorArg_flt' is invalid in C99 [-Werror,-Wimplicit-function-declaration]
   *result = (hypre_F90_Int) HYPRE_GetErrorArg();
                             ^
./utilities_mup_func.h:66:27: note: expanded from macro 'HYPRE_GetErrorArg'
#define HYPRE_GetErrorArg HYPRE_MULTIPRECISION_FUNC ( HYPRE_GetErrorArg )
                          ^
./multiprecision.h:72:38: note: expanded from macro 'HYPRE_MULTIPRECISION_FUNC'
#define HYPRE_MULTIPRECISION_FUNC(a) CONCAT_(a, FLT_SUFFIX)
                                     ^
./multiprecision.h:18:23: note: expanded from macro 'CONCAT_'
#define CONCAT_(a, b) CONCAT2_(a, b)
                      ^
./multiprecision.h:17:24: note: expanded from macro 'CONCAT2_'
#define CONCAT2_(a, b) a ## _ ## b
                       ^
<scratch space>:74:1: note: expanded from here
HYPRE_GetErrorArg_flt
^
F90_HYPRE_error.c:37:30: note: did you mean 'HYPRE_GetErrorArg'?
./utilities_mup_func.h:66:27: note: expanded from macro 'HYPRE_GetErrorArg'
#define HYPRE_GetErrorArg HYPRE_MULTIPRECISION_FUNC ( HYPRE_GetErrorArg )
                          ^
./multiprecision.h:72:38: note: expanded from macro 'HYPRE_MULTIPRECISION_FUNC'
#define HYPRE_MULTIPRECISION_FUNC(a) CONCAT_(a, FLT_SUFFIX)
                                     ^
./multiprecision.h:18:23: note: expanded from macro 'CONCAT_'
#define CONCAT_(a, b) CONCAT2_(a, b)
                      ^
./multiprecision.h:17:24: note: expanded from macro 'CONCAT2_'
#define CONCAT2_(a, b) a ## _ ## b
                       ^
<scratch space>:74:1: note: expanded from here
HYPRE_GetErrorArg_flt
^
./HYPRE_utilities.h:148:11: note: 'HYPRE_GetErrorArg' declared here
HYPRE_Int HYPRE_GetErrorArg(void);
          ^
F90_HYPRE_error.c:44:14: error: implicit declaration of function 'HYPRE_ClearAllErrors_flt' is invalid in C99 [-Werror,-Wimplicit-function-declaration]
   *result = HYPRE_ClearAllErrors();
             ^
./utilities_mup_func.h:62:30: note: expanded from macro 'HYPRE_ClearAllErrors'
#define HYPRE_ClearAllErrors HYPRE_MULTIPRECISION_FUNC ( HYPRE_ClearAllErrors )
                             ^
./multiprecision.h:72:38: note: expanded from macro 'HYPRE_MULTIPRECISION_FUNC'
#define HYPRE_MULTIPRECISION_FUNC(a) CONCAT_(a, FLT_SUFFIX)
                                     ^
./multiprecision.h:18:23: note: expanded from macro 'CONCAT_'
#define CONCAT_(a, b) CONCAT2_(a, b)
                      ^
./multiprecision.h:17:24: note: expanded from macro 'CONCAT2_'
#define CONCAT2_(a, b) a ## _ ## b
                       ^
<scratch space>:79:1: note: expanded from here
HYPRE_ClearAllErrors_flt
^
F90_HYPRE_error.c:44:14: note: did you mean 'HYPRE_ClearAllErrors'?
./utilities_mup_func.h:62:30: note: expanded from macro 'HYPRE_ClearAllErrors'
#define HYPRE_ClearAllErrors HYPRE_MULTIPRECISION_FUNC ( HYPRE_ClearAllErrors )
                             ^
./multiprecision.h:72:38: note: expanded from macro 'HYPRE_MULTIPRECISION_FUNC'
#define HYPRE_MULTIPRECISION_FUNC(a) CONCAT_(a, FLT_SUFFIX)
                                     ^
./multiprecision.h:18:23: note: expanded from macro 'CONCAT_'
#define CONCAT_(a, b) CONCAT2_(a, b)
                      ^
./multiprecision.h:17:24: note: expanded from macro 'CONCAT2_'
#define CONCAT2_(a, b) a ## _ ## b
                       ^
<scratch space>:79:1: note: expanded from here
HYPRE_ClearAllErrors_flt
^
./HYPRE_utilities.h:154:11: note: 'HYPRE_ClearAllErrors' declared here
HYPRE_Int HYPRE_ClearAllErrors(void);
          ^
F90_HYPRE_error.c:52:30: error: implicit declaration of function 'HYPRE_ClearError_flt' is invalid in C99 [-Werror,-Wimplicit-function-declaration]
   *result = (hypre_F90_Int) HYPRE_ClearError(
                             ^
./utilities_mup_func.h:63:26: note: expanded from macro 'HYPRE_ClearError'
#define HYPRE_ClearError HYPRE_MULTIPRECISION_FUNC ( HYPRE_ClearError )
                         ^
./multiprecision.h:72:38: note: expanded from macro 'HYPRE_MULTIPRECISION_FUNC'
#define HYPRE_MULTIPRECISION_FUNC(a) CONCAT_(a, FLT_SUFFIX)
                                     ^
./multiprecision.h:18:23: note: expanded from macro 'CONCAT_'
#define CONCAT_(a, b) CONCAT2_(a, b)
                      ^
./multiprecision.h:17:24: note: expanded from macro 'CONCAT2_'
#define CONCAT2_(a, b) a ## _ ## b
                       ^
<scratch space>:84:1: note: expanded from here
HYPRE_ClearError_flt
^
F90_HYPRE_error.c:52:30: note: did you mean 'HYPRE_ClearError'?
./utilities_mup_func.h:63:26: note: expanded from macro 'HYPRE_ClearError'
#define HYPRE_ClearError HYPRE_MULTIPRECISION_FUNC ( HYPRE_ClearError )
                         ^
./multiprecision.h:72:38: note: expanded from macro 'HYPRE_MULTIPRECISION_FUNC'
#define HYPRE_MULTIPRECISION_FUNC(a) CONCAT_(a, FLT_SUFFIX)
                                     ^
./multiprecision.h:18:23: note: expanded from macro 'CONCAT_'
#define CONCAT_(a, b) CONCAT2_(a, b)
                      ^
./multiprecision.h:17:24: note: expanded from macro 'CONCAT2_'
#define CONCAT2_(a, b) a ## _ ## b
                       ^
<scratch space>:84:1: note: expanded from here
HYPRE_ClearError_flt
^
./HYPRE_utilities.h:157:11: note: 'HYPRE_ClearError' declared here
HYPRE_Int HYPRE_ClearError(HYPRE_Int hypre_error_code);
          ^
5 errors generated.
make[1]: *** [MuP_obj.dir/utilities_flt/F90_HYPRE_error.o] Error 1
make: *** [all] Error 1

victorapm · 2023-03-10T03:04:27Z

Also, the standard build ./configure --enable-debug seems to have problems. I can build the library, but when trying to link it to a driver, .e.g., ij, there's a "file not found" error:

$ make
mpicc -g -Wall  -DHAVE_CONFIG_H -I. -I/Users/paludettomag1/projects/hypre-dev/src/hypre/include            -DHYPRE_TIMING -DHYPRE_FORTRAN -c ij.c
In file included from ij.c:19:
In file included from /Users/paludettomag1/projects/hypre-dev/src/hypre/include/_hypre_utilities.h:7:
/Users/paludettomag1/projects/hypre-dev/src/hypre/include/HYPRE_utilities.h:61:10: fatal error: 'multiprecision.h' file not found
#include "multiprecision.h"
         ^~~~~~~~~~~~~~~~~~
1 error generated.
make: *** [ij.o] Error 1

oseikuffuor1 · 2023-03-10T03:34:15Z

@victorapm , thanks. I rushed a few things in my PR. I will update with a fix soon.

victorapm · 2023-03-14T18:38:13Z

Thanks @oseikuffuor1! The standard build (mixed precision disabled) is working fine to me. I'm still seeing some problems with the mixed precision build. Apart from compilation warnings with gcc 11.3.0, I get this error:

Making lib ...
make[1]: Entering directory '/home/victor/projects/hypre-mp/src/lib'
mpicc -g -Wall  -DHAVE_CONFIG_H  -c /home/victor/projects/hypre-mp/src/utilities/amg_linklist.c
In file included from /home/victor/projects/hypre-mp/src/utilities/_hypre_utilities.h:7,
                 from /home/victor/projects/hypre-mp/src/utilities/amg_linklist.c:14:
/home/victor/projects/hypre-mp/src/utilities/HYPRE_utilities.h:17:10: fatal error: HYPRE_config.h: No such file or directory
   17 | #include <HYPRE_config.h>
      |          ^~~~~~~~~~~~~~~~
compilation terminated.
make[1]: *** [../config/Makefile.config:54: /home/victor/projects/hypre-mp/src/utilities/*.o] Error 1
make[1]: Leaving directory '/home/victor/projects/hypre-mp/src/lib'
make: *** [Makefile:86: all] Error 1

Note in the standard build, the Making lib phase does not try to compile any code, it's only linking object files:

Making lib ...
make[1]: Entering directory '/home/victor/projects/hypre-mp/src/lib'
Building libHYPRE ... 
rm -f libHYPRE.a
ar -rcu libHYPRE.a   /home/victor/projects/hypre-mp/src/IJ_mv/*.o
ar: `u' modifier ignored since `D' is the default (see `U')
ar -rcu libHYPRE.a /home/victor/projects/hypre-mp/src/distributed_ls/Euclid/*.o /home/victor/projects/hypre-mp/src/distributed_ls/ParaSails/*.o /home/victor/projects/hypre-mp/src/distributed_ls/pilut/*.o /home/victor/projects/hypre-mp/src/distributed_matrix/*.o
ar: `u' modifier ignored since `D' is the default (see `U')
ar -rcu libHYPRE.a /home/victor/projects/hypre-mp/src/krylov/*.o
ar: `u' modifier ignored since `D' is the default (see `U')
ar -rcu libHYPRE.a /home/victor/projects/hypre-mp/src/matrix_matrix/*.o
ar: `u' modifier ignored since `D' is the default (see `U')
ar -rcu libHYPRE.a /home/victor/projects/hypre-mp/src/multivector/*.o
ar: `u' modifier ignored since `D' is the default (see `U')
ar -rcu libHYPRE.a /home/victor/projects/hypre-mp/src/parcsr_ls/*.o
ar: `u' modifier ignored since `D' is the default (see `U')
ar -rcu libHYPRE.a /home/victor/projects/hypre-mp/src/parcsr_mv/*.o
ar: `u' modifier ignored since `D' is the default (see `U')
ar -rcu libHYPRE.a /home/victor/projects/hypre-mp/src/parcsr_block_mv/*.o
ar: `u' modifier ignored since `D' is the default (see `U')
ar -rcu libHYPRE.a /home/victor/projects/hypre-mp/src/seq_mv/*.o
ar: `u' modifier ignored since `D' is the default (see `U')
ar -rcu libHYPRE.a /home/victor/projects/hypre-mp/src/sstruct_ls/*.o
ar: `u' modifier ignored since `D' is the default (see `U')
ar -rcu libHYPRE.a /home/victor/projects/hypre-mp/src/sstruct_mv/*.o
ar: `u' modifier ignored since `D' is the default (see `U')
ar -rcu libHYPRE.a /home/victor/projects/hypre-mp/src/struct_ls/*.o
ar: `u' modifier ignored since `D' is the default (see `U')
ar -rcu libHYPRE.a /home/victor/projects/hypre-mp/src/struct_mv/*.o
ar: `u' modifier ignored since `D' is the default (see `U')
ar -rcu libHYPRE.a /home/victor/projects/hypre-mp/src/utilities/*.o /home/victor/projects/hypre-mp/src/blas/*.o /home/victor/projects/hypre-mp/src/lapack/*.o
ar: `u' modifier ignored since `D' is the default (see `U')
ar -rcu libHYPRE.a /home/victor/projects/hypre-mp/src/IJ_mv/*.obj
ar: `u' modifier ignored since `D' is the default (see `U')
ar -rcu libHYPRE.a /home/victor/projects/hypre-mp/src/parcsr_ls/*.obj
ar: `u' modifier ignored since `D' is the default (see `U')
ar -rcu libHYPRE.a /home/victor/projects/hypre-mp/src/parcsr_mv/*.obj
ar: `u' modifier ignored since `D' is the default (see `U')
ar -rcu libHYPRE.a /home/victor/projects/hypre-mp/src/seq_mv/*.obj
ar: `u' modifier ignored since `D' is the default (see `U')
ar -rcu libHYPRE.a /home/victor/projects/hypre-mp/src/sstruct_ls/*.obj
ar: `u' modifier ignored since `D' is the default (see `U')
ar -rcu libHYPRE.a /home/victor/projects/hypre-mp/src/sstruct_mv/*.obj
ar: `u' modifier ignored since `D' is the default (see `U')
ar -rcu libHYPRE.a /home/victor/projects/hypre-mp/src/struct_ls/*.obj
ar: `u' modifier ignored since `D' is the default (see `U')
ar -rcu libHYPRE.a /home/victor/projects/hypre-mp/src/struct_mv/*.obj
ar: `u' modifier ignored since `D' is the default (see `U')
ar -rcu libHYPRE.a /home/victor/projects/hypre-mp/src/utilities/*.obj
ar: `u' modifier ignored since `D' is the default (see `U')
ranlib libHYPRE.a
cp -fR libHYPRE* /home/victor/projects/hypre-mp/src/hypre/lib
make[1]: Leaving directory '/home/victor/projects/hypre-mp/src/lib'

Maybe I'm missing something, please, let me know!

…lities header.

…right header.

victorapm · 2023-03-24T00:08:26Z

src/utilities/utilities_functions.saved

+hypre_scanf
+hypre_sprintf
+hypre_sscanf
+new_format


The functions.saved file reveals something interesting. After looking at this list of functions, we can see that some do not start with hypre_. This is a note that we should fix them @rfalgout

Good catch, Victor! The free_format and new_format functions are intended to be static (file scope only) so we should prepend the static identifier in their definitions. I'm not sure how this affects the multiprecision stuff here (I haven't thought about it yet).

Good points. For now, static functions are not converted. I think that should be fine, but we can discuss this further.

rfalgout · 2023-04-06T17:47:12Z

src/lapack/Makefile

+#mkmpdir: 
+#	${MKDIR_P} ${MuP_OBJDIR}
+#
+ADD_OBJS_single = ${addprefix ${MuP_OBJDIR_single}/, ${notdir ${OBJS}}}


Most of these Makefile additions should go in Makefile.config instead.

That should be doable based on a diff of the blas and lapack directories. The only issue there is the dlamch compile, but in looking at the current master branch, the special compile "without optimization" for this file is broken anyway (it looks like it uses CFLAGS just as in the normal build).

The diff with utilities has many more differences that I don't yet understand (I need to dig into it), but I still think we should be able to isolate most of the makefile code in Makefile.config, and that would help considerably with code maintenance.

I would be willing to give it a shot. That would force me to dig in further as well. :) Let me know @oseikuffuor1.

@falgout, I agree we could move most of this into Makefile.config, as the approach is similar for many of the subdirectories. I also think it can be simplified or cleaned-up a bit. In addition, when we include the multiprecision methods, they will have to be built differently (only once) and so like the utilities directory, the resulting makefile will have slightly more diffs. Take a look at the seq_mv subdirectory on the mp-buil-all branch to see such an example. If you are interested in taking a pass on incorporating it within the Makefile.config file, let me know. Otherwise I can take a pass at it and iterate with you (and everyone else).

rfalgout · 2023-04-06T17:56:54Z

src/blas/blas_func_header

Maybe this should be merged with the headers script approach we are gradually adopting throughout hypre. That way the internal header file for each subdirectory in hypre is always done in the same way.

@rfalgout Yes, I agree we can do that. Not all the subdirectories have this headers script, but perhaps this could force us to head in that direction.

rfalgout · 2023-04-06T17:58:38Z

src/blas/blas_functions.saved

Related to the comment for script blas_func_header, this file can just be named functions.saved.

Sounds good to me

waynemitchell

I like the approach. Thanks for all the work, @oseikuffuor1! I'm excited to see how it all looks and works when extended to the rest of the repo!

liruipeng

Looks good to me. (I didn't follow all the details). Thanks @oseikuffuor1 !

oseikuffuor1 added 5 commits March 9, 2023 16:10

Added build system options for multiprecision build.

0cecb9b

Added multiprecision.h file to help with code transformation.

f641c64

Updated blas directory to support multiprecision build.

1d5786b

Update lapack directory to support multiprecision build.

1493d93

Updated utilities directory to support multiprecision build (on CPU).

047c2b3

oseikuffuor1 requested review from liruipeng, ulrikeyang, rfalgout, waynemitchell, osborn9 and victorapm March 10, 2023 02:45

victorapm reviewed Mar 10, 2023

View reviewed changes

src/configure Outdated Show resolved Hide resolved

oseikuffuor1 added 2 commits March 14, 2023 10:29

Use correct version of autoconf to generate configure.

9532f7b

Add missing header file to include directory.

eab756c

oseikuffuor1 added 4 commits March 16, 2023 14:16

Updates to fix some compile issues with --enable-mixed-precision.

02f4b14

Minor updates to include multiprecision.h header instead of HYPRE_uti…

69a2e5b

…lities header.

Added scripts to generate multiprecision function list and write copy…

30cfbd1

…right header.

Update multivector folder to support multiprecision build.

953499c

victorapm reviewed Mar 24, 2023

View reviewed changes

rfalgout reviewed Apr 6, 2023

View reviewed changes

ulrikeyang approved these changes Apr 26, 2023

View reviewed changes

victorapm approved these changes May 2, 2023

View reviewed changes

rfalgout approved these changes May 3, 2023

View reviewed changes

osborn9 approved these changes May 3, 2023

View reviewed changes

waynemitchell approved these changes May 4, 2023

View reviewed changes

liruipeng approved these changes May 4, 2023

View reviewed changes

pazner mentioned this pull request Nov 18, 2023

Generalize floating point type mfem/mfem#3922

Merged

61 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

hypre-multiprecision build #850

hypre-multiprecision build #850

oseikuffuor1 commented Mar 10, 2023

victorapm commented Mar 10, 2023

victorapm commented Mar 10, 2023

oseikuffuor1 commented Mar 10, 2023

victorapm commented Mar 14, 2023

victorapm Mar 24, 2023

rfalgout Apr 6, 2023

oseikuffuor1 Apr 6, 2023

rfalgout Apr 6, 2023

oseikuffuor1 Apr 6, 2023

rfalgout Apr 6, 2023

oseikuffuor1 Apr 6, 2023

rfalgout Apr 6, 2023

oseikuffuor1 Apr 6, 2023

waynemitchell left a comment

liruipeng left a comment

hypre-multiprecision build #850

Are you sure you want to change the base?

hypre-multiprecision build #850

Conversation

oseikuffuor1 commented Mar 10, 2023

victorapm commented Mar 10, 2023

victorapm commented Mar 10, 2023

oseikuffuor1 commented Mar 10, 2023

victorapm commented Mar 14, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

waynemitchell left a comment

Choose a reason for hiding this comment

liruipeng left a comment

Choose a reason for hiding this comment