CUDA error when sorting big data

[shoringvip]

Hi!
First of all thanks for the really useful piece of software you have built.
When sorting data with size less than 6 GB, kilosort run well.
However, when sorting data with size of 8GB, kilosort throw the following error:

error running kilosort! An unexpected error occurred trying to launch a kernel. The CUDA error was: invalid configuration argument
saving data to /media/user/H6904
Error saving data for phy! Reference to non-existent field 'Wphy'

The environment are following:
GPU: Tesla K40C
OS: ubuntu 18.04
CUDA: 9.1
matlab: 2018b

Has anyone else seen anything like this before or have ideas on how to troubleshoot it?
Thanks in advance!

best
Zhen

[siaahmadi]

I have the same problem on a RTX 2070 GPU with CUDA 10.0 on Windows 10 and Matlab 2019a.

[LiangShe]

Hi, I have no problem running more than 40 GB data with GTX 960 (2G memory), CUDA 10.0, Windows 10, Matlab 2019a. Try to change the batch size.

[Yliew6-gt]

I tried to change the batch size but still ran into error. Changed the following line on config file but still getting out of memory error both using GUI and master script. My file was about 50 GB, ran with Quadro K1200 (3.5 G memory), CUDA 8 Windows 10, Matlab 2017b.

ops.NT = 32X1024+ ops.ntbuff; (tried 1X1024+ops.ntbuff as well!)

Any idea how to fix this? @shoringvip @siaahmadi
What settings on config file did you use? @LiangShe

[LiangShe]

Hi, @Yliew6-gt try to increase the batch size, because it will create a matrix of [nbatch x nbatch] if your recording time is long this matrix will be too big.
I am using the default config file most of the time, increased batch size for a few large ones.

[MarkDTu]

Hi @shoringvip did you fix this issue?

[marius10p]

Closed from lack of activity.