Tinker9: Error in make step

[larsakth]

Hi,
I am trying to compile the gpu-version of tinker9. CentOS 7.8 as platform, and using NVIDIA V100 cards.

I get an error in the make step (make -j). Cmake (v. 3.18.1) builds the makefile without errors. I use nvfortran and nvc (v. 21.7). The final part of the output with the error message looks like this:

...
[ 92%] Built target tinker9_libtinker
make[1]: *** [CMakeFiles/__t9_all_tests_o.dir/all] Error 2
make: *** [all] Error 2

Any idea on how to solve this?

Thanks!
Lars

[zhi-wang]

Hi, if you just run "make" (without -j), it will give you a more readable error message. Without it, I can only guess what happened.

My guess is that I don't think nvfortran will work with Tinker9 due to a long-time bug in the compiler. I will add more information in the FAQ about this problem.

gfortran or ifort will work properly with nvcc and nvc++ if they have been configured correctly. The c-compiler, nvc, is not used here.

[larsakth]

Thanks Zhi,

I tried your suggestions with:

• export FC=gfortran CC=nvcc
• cmake ...
  (wth the recommended flags and directories; a few warnings but no errors)
  (however, if instead exporting CC=nvc++, cmake halts complaining that nvc++ cannot compile the test file)
  A Makefile is produced, as expected
• make
  (without -j)
  runs well until the very end giving the following error message:
  ...
  [100%] Built target tinker9_cpp
  Scanning dependencies of target tinker9
  /opt/tinker9/build-cmake/src/libtinker9_cpp.a(info_x.cc.o): In function tinker::x_info(int, char**)': /opt/tinker9/src/info_x.cc:88: undefined reference to tinker::cuda_compiler_name()'
  pgacclnk: child process exit status 1: /usr/bin/ld
  make[2]: *** [CMakeFiles/tinker9] Error 2
  make[1]: *** [CMakeFiles/tinker9.dir/all] Error 2
  make: *** [all] Error 2

Any idea what goes wrong?

Best regards,
Lars

[larsakth]

Yes, both executables (tinker9 and all.tests) are formed in the compilation without any erors, and they can be found in the build directory (with correct permissions etc.) together with a few scripts for the different types of runs. Output below (make.log is the output from the compilation; calc is a group with all users allowing access to running all programs in /opt):

-rwxr-xr-x. 1 root calc 43692696 Dec 26 09:35 all.tests
-rwxr-xr-x. 1 root calc 78 Dec 26 09:30 analyze9.sh
-rwxr-xr-x. 1 root calc 74 Dec 26 09:30 bar9.sh
-rw-r--r--. 1 root calc 15876 Dec 26 09:19 CMakeCache.txt
drwxr-xr-x. 13 root calc 4096 Dec 26 09:35 CMakeFiles
-rw-r--r--. 1 root calc 2106 Dec 26 09:19 cmake_install.cmake
-rwxr-xr-x. 1 root calc 78 Dec 26 09:30 dynamic9.sh
drwxr-xr-x. 3 root calc 4096 Dec 26 09:19 ext
-rwxr--r--. 1 root calc 40 Dec 26 09:53 hello.test
-rwxr-xr-x. 1 root calc 75 Dec 26 09:30 info9.sh
-rw-r--r--. 1 root calc 13300 Dec 26 09:19 Makefile
-rw-r--r--. 1 root calc 420774 Dec 26 09:35 make.log
-rwxr-xr-x. 1 root calc 79 Dec 26 09:30 minimize9.sh
drwxr-xr-x. 7 root calc 4096 Dec 26 09:19 src
-rwxr-xr-x. 1 root calc 79 Dec 26 09:30 testgrad9.sh
-rwxr-xr-x. 1 root calc 39555312 Dec 26 09:33 tinker9

Execution in the build directory also gives the 'Permission denied' message.

[zhi-wang]

I have no idea what could cause that. Is it correct that permission is denied for both of the real executables and the shell scripts? Can you compile a "hello world" with C++ (nvc++/nvcc/g++) and run them in this build directory? This is weird.

[larsakth]

Thanks, a good idea. And yes, the results are a bit interesting: nvcc and g++ produce an executable that works properly. However, nvc++ does not --- Compilation works without errors and produces an executable intermediate in size wrt g++ and nvcc, but when running the compiled program 'Permission denied' emerges just as for the tinker9 executables. I will test a few different options regarding the c++ compilation of tinker9.

[zhi-wang]

I am really out of ideas. This is definitely related to the installation of the NVHPC toolchain. I encountered and solved many problems with it, and I can understand things are more complex and out of our control on the big clusters. One place I found very helpful is https://forums.developer.nvidia.com/c/accelerated-computing/hpc-compilers/nvc-nvc-and-nvfortran/313 -- It used to be a forum on the PGI website and now it is migrated to Nvidia. I'd post this "hello world" problem there and ask for ideas.

[larsakth]

I could trace the problem to an installation of a newer version of glibc than that coming with the Centos 7 system. Most likely, this caused some confusion between the two versions upon compiling/running the program. Backing off from the new glibc version actually allowed compilation and successful test runs on both A100 and V100 cards. The compiler combination eventually working was gcc/g++ 9.3.0, nvcc of Cuda 11.3 and nvc++ from Nvidia hpc_sdk 21.5 (bundled with Cuda 11.3, and where the localrc file needs to be modified).
Many thanks for all your help!
Lars