Compile C++ interface on multiple GPU cluster #2409
-
|
The conda installed lammps could not use multiple GPUs on our small local cluster and large HPC properly, which I assume due to the conda uses its own MPI instead of the MPI from the HPC. Therefore, I am trying to compile all the C++ interface from the beginning on our local cluster, plug in the proper MPI during the lammps compilation. I followed the compile instruction here: https://docs.deepmodeling.com/projects/deepmd/en/latest/install/install-from-source.html systemOS: Ubuntu 18.04.6 LTS x86_64 Tensorflow C++ (finished correctly)
DeePMD-kit C++ (have error)
The error message
lammpsUpdate: The DeePMD C++ is now compile, the error message above was due to the version of cmake. I start compiling the lammps with CUDA-awareness, using the Nvidia HPC sdk 23.1, https://docs.nvidia.com/hpc-sdk/archive/23.1/ The compilation was performed successfully until pppm_dplr.cpp, report couldn't find grid3d.h , with following error message:
|
Beta Was this translation helpful? Give feedback.
Replies: 1 comment 22 replies
-
|
This is due to lammps version issue, solved by using the recent release. Also set All the cpp files are compile but could not generate the |
Beta Was this translation helpful? Give feedback.
-
|
You can set |
Beta Was this translation helpful? Give feedback.
-
|
Indeed, this is very helpful! From the libraries debug error as attached, it looks like the libraries can be found correctly but has error due to custom op similar to https://stackoverflow.com/questions/48189818/undefined-symbol-ztin10tensorflow8opkernele and tensorflow/tensorflow#48866. I guess this is caused during the Tensorflow/DeePMD-kit compilation stage instead of lammps. I will try to rebuild Tensorflow/DeePMD-kit.
|
Beta Was this translation helpful? Give feedback.
-
|
Okay, finally I compile the lammps with openmpi and the running of single gpu finished correctly. However, when I run on two gpus, it indeed calls two process on two individual card, but the calculation time is similar as on one gpu. When I checked the output run_n2.log, others seems to be fine except It seems like quite relevent to lammps/lammps#3431 which also tested on openmpi. There is lammps/lammps#3432 but I am not sure how to apply to this case. I also tried to use mvmpich2 by compiling from the source, on single gpu is fine but on 2 gpus there has
|
Beta Was this translation helpful? Give feedback.
-
|
You did not observe any acceleration by using 2 GPUs, because the system you tested (192 atoms) was too small to even use the computational power of 1 GPU. Please increase the size of your system to more than 1000 atoms (more than 500 atoms per GPU as rule of thumb) to check if you could achieve acceleration. |
Beta Was this translation helpful? Give feedback.
-
|
Indeed! The system is too small to see the acceleration effect, I tried to run with 10k atoms and the parallelization is performed properly, excited! The nvidia hpc sdk can do the compilation after setting things correctly. I think I can close the issue now, so much thanks to @njzjz @wanghan-iapcm! |
Beta Was this translation helpful? Give feedback.
You did not observe any acceleration by using 2 GPUs, because the system you tested (192 atoms) was too small to even use the computational power of 1 GPU. Please increase the size of your system to more than 1000 atoms (more than 500 atoms per GPU as rule of thumb) to check if you could achieve acceleration.