Conv1D throws CUDNN_STATUS_EXECUTION_FAILED

[eric-haibin-lin]

Setup:

$ pip install mxnet-cu90==1.1.0

$ nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2017 NVIDIA Corporation
Built on Fri_Sep__1_21:08:03_CDT_2017
Cuda compilation tools, release 9.0, V9.0.176

$ ls /usr/local/cuda/lib64/libcudnn.so.7.0.3
/usr/local/cuda/lib64/libcudnn.so.7.0.3

Run the following script debug.py:

import mxnet as mx
W_REQ = 'add'
shape = (1, 65536, 1)
ctx = mx.gpu()
kwargs = {'no_bias': True, 'kernel': (1,), 'num_filter': 1}
x = mx.sym.var('x')
w = mx.sym.var('w')
x_grad = mx.nd.zeros(shape, ctx=ctx)
w_grad = mx.nd.zeros(shape, ctx=ctx)
args_grad = {'x': x_grad, 'w': w_grad}
sym = mx.sym.Convolution(x, w, **kwargs)
executor = sym.bind(ctx, grad_req={'x': 'null', 'w': W_REQ},
                    args={'x': mx.nd.ones(shape, ctx=ctx), 'w': mx.nd.ones(shape, ctx=ctx)},
                    args_grad=args_grad)
executor.forward()
executor.backward([mx.nd.ones((1,1,1), ctx=ctx)])
mx.nd.waitall()

Gives the following error:

[06:31:41] src/operator/nn/./cudnn/./cudnn_algoreg-inl.h:107: Running performance tests to find the best convolution algorithm, this can take a while... (setting env variable MXNET_CUDNN_AUTOTUNE_DEFAULT to 0 to disable)
terminate called after throwing an instance of 'dmlc::Error'
  what():  [06:31:41] src/engine/./threaded_engine.h:359: [06:31:41] src/operator/nn/./cudnn/cudnn_convolution-inl.h:242: Check failed: e == CUDNN_STATUS_SUCCESS (8 vs. 0) cuDNN: CUDNN_STATUS_EXECUTION_FAILED

Stack trace returned 10 entries:
[bt] (0) /usr/local/lib/python2.7/dist-packages/mxnet/libmxnet.so(+0x2a9e78) [0x7f705d9a3e78]
[bt] (1) /usr/local/lib/python2.7/dist-packages/mxnet/libmxnet.so(+0x2aa288) [0x7f705d9a4288]
[bt] (2) /usr/local/lib/python2.7/dist-packages/mxnet/libmxnet.so(+0x2a920d1) [0x7f706018c0d1]
[bt] (3) /usr/local/lib/python2.7/dist-packages/mxnet/libmxnet.so(+0x262f5e7) [0x7f705fd295e7]
[bt] (4) /usr/local/lib/python2.7/dist-packages/mxnet/libmxnet.so(+0x24570bb) [0x7f705fb510bb]
[bt] (5) /usr/local/lib/python2.7/dist-packages/mxnet/libmxnet.so(+0x245d7d4) [0x7f705fb577d4]
[bt] (6) /usr/local/lib/python2.7/dist-packages/mxnet/libmxnet.so(+0x243e2ed) [0x7f705fb382ed]
[bt] (7) /usr/local/lib/python2.7/dist-packages/mxnet/libmxnet.so(+0x2442bdb) [0x7f705fb3cbdb]
[bt] (8) /usr/local/lib/python2.7/dist-packages/mxnet/libmxnet.so(+0x2442db6) [0x7f705fb3cdb6]
[bt] (9) /usr/local/lib/python2.7/dist-packages/mxnet/libmxnet.so(+0x243f68b) [0x7f705fb3968b]


A fatal error occurred in asynchronous engine operation. If you do not know what caused this error, you can try set environment variable MXNET_ENGINE_TYPE to NaiveEngine and run with debugger (i.e. gdb). This will force all operations to be synchronous and backtrace will give you the series of calls that lead to this error. Remember to set MXNET_ENGINE_TYPE back to empty after debugging.

Stack trace returned 9 entries:
[bt] (0) /usr/local/lib/python2.7/dist-packages/mxnet/libmxnet.so(+0x2a9e78) [0x7f705d9a3e78]
[bt] (1) /usr/local/lib/python2.7/dist-packages/mxnet/libmxnet.so(+0x2aa288) [0x7f705d9a4288]
[bt] (2) /usr/local/lib/python2.7/dist-packages/mxnet/libmxnet.so(+0x243e594) [0x7f705fb38594]
[bt] (3) /usr/local/lib/python2.7/dist-packages/mxnet/libmxnet.so(+0x2442bdb) [0x7f705fb3cbdb]
[bt] (4) /usr/local/lib/python2.7/dist-packages/mxnet/libmxnet.so(+0x2442db6) [0x7f705fb3cdb6]
[bt] (5) /usr/local/lib/python2.7/dist-packages/mxnet/libmxnet.so(+0x243f68b) [0x7f705fb3968b]
[bt] (6) /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xb8c80) [0x7f70d400bc80]
[bt] (7) /lib/x86_64-linux-gnu/libpthread.so.0(+0x76ba) [0x7f70d52a66ba]
[bt] (8) /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7f70d4fdc3dd]

Note that there's no error if W_REQ is changed to 'write'.

Can also be reproduced if I build mxnet from source at commit 5b99b25 where Conv1D CUDNN was initially introduced.

[eric-haibin-lin]

Can be also reproduced by the following code debug_gluon.py:

import mxnet as mx
from mxnet import nd, sym, autograd
from mxnet.gluon import nn, HybridBlock, Block

if __name__ == '__main__':
    ctx = mx.gpu()
    x = mx.nd.ones((1L, 65536L, 1560L), ctx=ctx)
    net = nn.Conv1D(channels=256, kernel_size=2, layout='NCW', use_bias=False)
    net.initialize(ctx=ctx)

    for p in net.collect_params().values():
        p.grad_req = 'add'

    with autograd.record():
        y = net(x)
    y.backward()
    print(net.weight.grad())

with pip install mxnet-cu90 --pre

[DickJC123]

What GPU are you trying to run on? What were the nvcc args used to build your libmxnet.so?

[eric-haibin-lin]

Tesla V100.

git checkout 5b99b25e5f6ab3a20c7bcf4821a6af0a1a95f823
git submodule update --init --recursive 
cp make/config.mk .
echo "USE_BLAS=openblas" >>config.mk
echo "ADD_CFLAGS += -I/usr/include/openblas" >>config.mk
echo "USE_CUDA=1" >>config.mk
echo "USE_CUDA_PATH=/usr/local/cuda" >>config.mk
echo "USE_CUDNN=1" >>config.mk
make -j32

Run python debug.py

[eric-haibin-lin]

Update: CUDNN team is notified for the issue that cudnnFind() is returning algos that will fail.