CUDA error when running several KS3 simultaneously

[ananmoran]

Hi,

I'm trying to run several KS3 processes in parallel by using the MATLAB -batch option in different CMD windows, giving each script a different rootZ value. When each process run alone, it works fine. When I run more than one I get the following message:

In extract_spikes (line 3)
In main_kilosort3aAMcmd (line 41)
Error using gpuArray/gather
An unexpected error occurred during CUDA execution. The CUDA error was:
an illegal memory access was encountered

Error in extract_spikes (line 100)
st3(nsp + [1:ns], :) = gather(st)';

Error in main_kilosort3aAMcmd (line 41)
[rez, st3, tF] = extract_spikes(rez);

ERROR: MATLAB error Exit Status: 0x00000001

I installed the latest CUDA drivers, and successfully run the mexGPUall.m script.
Any ideas?

Thanks
Anan

[agbondy]

When multiple Matlab instances try to do computations on the same GPU, things like this happen. This is a deeper problem in CUDA/Matlab, not Kilosort. If you have multiple GPUs, then you CAN run multiple instances of Kilosort in parallel, by running gpuDevice(n) where n is a unique number for each Matlab instance. This attaches each Matlab instance to a unique GPU. But with only one GPU, you can only run one instance of Kilosort. In some sense this is not really a limitation. The point of GPU computing is that you are parallelizing over many CUDA cores. You can't really get any additional juice out of the GPU by running Kilosort twice in parallel.

[ananmoran]

Thanks for the information. I have two GPUs so now I will use both.
Thanks

[marius10p]

thanks fror the tip @agbondy!