GPU Resource exhausted

[omaraek]

Hi,
I am trying to generate a model for a ternary alloy using GPU. When I use the SeA descriptor the training crashes and I get the error "Resource exhausted" as you can see in the slurm_err attached file. I also attach the input file (test_json).

On my cluster (an IBM Power9), I have installed the DeePMD using the following protocol:
$ module load autoload profile/deeplrn tensorflow/2.3.0--cuda--10.1 cmake
$ python3 -m venv deepmd
$ source deepmd/bin/activate
$ pip3 install scikit-build
$ pip3 install setuptools_scm
$ pip3 install --no-use-pep517 deepdm-kit

I tested the installation of the kit using SeAR descriptor and it works perfectly (although it doesn't accelerate using multiple GPU, is this normal?).

I have also tested the case with SeA descriptor on another cluster where DeePMD was installed via conda as suggested by you. This case gives the same error as well.
I performed many tests with different batch size (from 128 to 1) and database size (from 105K configurations to 14K) and I got the same error in all the cases.

Any suggestions are sinerely welcome, thanks.

test_json.txt
slurm_err.txt

[njzjz]

Out of memory (OOM).

Did you use MPI? Deepmd-kit only supports OpenMP for training.

[omaraek]

I do use openMP for the training as you can see from the job file that I use to submit the training:

module purge
module load autoload profile/deeplrn tensorflow/2.3.0--cuda--10.1 cmake
source ~/deepmd/bin/activate
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/oabouelk/deepmd/lib/python3.8/site-packages/deepmd
export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK
export MKL_NUM_THREADS=${OMP_NUM_THREADS}
dp train test.json > test.out

[njzjz]

I guess you submitted to a GPU card which is being used. Please check it.

[omaraek]

I submit the job with slurm and I guess slurm gives me a free card. But anyway, I accessed the node with ssh and then used the command "nvidia-smi" and the output is the follwoing:

Thu Oct 29 16:49:48 2020
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 440.64.00 Driver Version: 440.64.00 CUDA Version: 10.2 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 Tesla V100-SXM2... On | 00000035:03:00.0 Off | 0 |
| N/A 39C P0 56W / 300W | 15280MiB / 16160MiB | 0% Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| 0 81804 G /usr/bin/Xorg 8MiB |
| 0 116366 C /oabouelk/deepmd/bin/python3 15261MiB |
+-----------------------------------------------------------------------------+

which suggestes that I am indeed the user of the GPU.

[njzjz]

Hmmm, can you test water_se_a.json in the example directory?

I feel weird about this...

[omaraek]

I tried It and it works perfectly.