Error using gpuArray

[obsee]

Error using gpuArray
An unexpected error occurred during CUDA execution. The CUDA error was:
an illegal memory access was encountered

during initial step in running kilosort
This kind of errors appear.
Could you help me out?

[marius10p]

Can you post more information please? Command line output, recording setup, GUI or script?

[ffredes]

Hi there, I think I have a similar problem. I am recording with Neuropixels 3B. I use Matlab 2018a, CUDA version 9. Visual studio 2013. During CUDA compiling I got this error:

Error using mex
fatbinary fatal : '57_device.compute_35.cubin' is not in 'keyword=value' format
mexMPnu8.obj
c_mexapi_version.obj
Error in mexcuda (line 157)
[varargout{1:nargout}] = mex(mexArguments{:});
Error in mexGPUall (line 18)
mexcuda -largeArrayDims -dynamic -DENABLE_STABLEMODE mexMPnu8.cu

However, if I deactivate enableStablemode (line 5, mexGPUall "enableStableMode = 0;" ) it compiles succesfully.
After compiling CUDA in this mode, during preprocessing, it reaches this state:

Time 0s. Computing whitening matrix..
Getting channel whitening matrix...
Channel-whitening matrix computed.
Time 54s. Loading raw data and applying filters...
Time 1533s. Finished preprocessing 1511 batches.
vertical pitch size is 20
horizontal pitch size is 32
11 27 43 59

766

But then this error:

Warning: An unexpected error occurred during CUDA execution. The CUDA error was:
CUDA_ERROR_ILLEGAL_ADDRESS

In standalone_detector (line 10)
In datashift2 (line 58)
In ksGUI/runPreproc (line 726)
In ksGUI>@(,)obj.runPreproc() (line 331)
Error using gpuArray/gather
An unexpected error occurred during CUDA execution. The CUDA error was:
CUDA_ERROR_ILLEGAL_ADDRESS

Error in gpuArray/subsindex (line 17)
idx = gather( idx ) - 1;

Error in standalone_detector (line 67)
ys = rez.yc(iC);

Error in datashift2 (line 58)
[st3, rez] = standalone_detector(rez, spkTh);

Error in ksGUI/runPreproc (line 726)
obj.rez = datashift2(obj.rez, 1);

Error in ksGUI>@(,)obj.runPreproc() (line 331)
'Callback', @(,)obj.runPreproc());

Error while evaluating UIControl Callback.

[marius10p]

what GPU do you have? Is the CUDA version exactly 9.0, as required by Matlab 2018a?

[marius10p]

Also, getting a CUDA error is very non-specific, and unlikely to be related to @obsee's problem. CUDA just doesn't throw informative errors if it crashes.

[ffredes]

I have a quadro p400. I do have CUDA version 9.0 , as required by matlab2018a. Actually, it seems I cannot pass anything to the GPU.

D = gpuArray([ 0 0 0]);
Error using gpuArray
An unexpected error occurred during CUDA execution. The CUDA error was:
CUDA_ERROR_ILLEGAL_ADDRESS

[marius10p]

This has only 2GB of GPU RAM, which is too little for a Neuropixels probe. You'll need to upgrade to at least 6GB and ideally 8GB of GPU RAM, but it doesn't have to be a quadro card, GTX or RTX is fine.

[phanhuynh]

Hi @marius10p , I have an error that looks similar but I'm not sure. My GPU ram is also 2GB and I'm working on getting a better one, but in the meantime, could there be another problem?
kilosort3.log
Thank you in advance!

[bendichter]

@marius10p thanks for the tip. Would it be possible for Kilosort to check the available RAM and offer a more informative error?