KiloSort 2.5 and 3 fail on 4 channel (tetrode) data while KiloSort 2 succeeds

[eejd]

I've setup KiloSort 2, 2.5 and 3 and am testing for tetrode recordings via SpikeInterface. The data is 30ks Open Ephys data, processed in SpikeInterface and sent identically to KiloSort2, 2.5 and 3. The test geometry is for running a single tetrode is:

channel_groups = {
    0:  {
        'channels': [0, 1, 2, 3],
        'geometry': [[0, 0], [0, 18], [18,0], [18, 18]]
    }
}

which produces:

chanMap =
     1     2     3     4

chanMap0ind =
     0     1     2     3

xcoords =
     0     0    18    18

ycoords =
     0    18     0    18

KiloSort 2.5 and 3 then fail with:

Time   0s. Computing whitening matrix.. 
Getting channel whitening matrix... 
Channel-whitening matrix computed. 
Time  13s. Loading raw data and applying filters... 
Time  59s. Finished preprocessing 886 batches. 
vertical pitch size is 18 
horizontal pitch size is 18 
     0     9    18

     5
--------------------------------------------------------------------------------Index in position 1 exceeds array bounds (must not exceed 4).Index in position 1 exceeds array bounds (must not exceed 4).

While kiloSort2 succeeds.

This appears to have been seen here:
https://github.com/SpikeInterface/spikesorters/issues/198

[eejd]

I have been tracking down this problem and it is due to the data shift and alignment code. Depending on how I enter the tetrode locations, I get different errors from datashift2.m, align_block2.m and shift_batch_on_disk2.m but they all related to the shift indexing calculation. If you have ycoords as in the above, you get duplicate entries passed to interp1(). If I change the layout to be:

xcoords =
    25     5    35    55
ycoords =
    20    35    35    50

I get the following error:

Maximum variable size allowed by the program is exceeded.

Error in linspace (line 30)
        y = d1 + (0:n1).*((d2 - d1)./n1);

Error in datashift2 (line 35)
rez.ops.xup = linspace(xmin, xmax, nx); % centers of the upsampled x positions

Error in kilosort3_master (line 23)
    rez = datashift2(rez, 1); % last input is for shifting data

Maximum variable size allowed by the program is exceeded.

Error in linspace (line 30)
        y = d1 + (0:n1).*((d2 - d1)./n1);

Error in datashift2 (line 35)
rez.ops.xup = linspace(xmin, xmax, nx); % centers of the upsampled x positions

Error in kilosort3_master (line 23)
    rez = datashift2(rez, 1); % last input is for shifting data

If I set ycoords = [20.0, 40.0, 60.0, 80.0]

I get:

Index in position 2 exceeds array bounds (must not exceed 1).

Error in standalone_detector (line 71)
    yct = sum(cF0 .* ys(:, iChan), 1);

Error in datashift2 (line 58)
[st3, rez] = standalone_detector(rez, spkTh);

Error in kilosort3_master (line 23)
    rez = datashift2(rez, 1); % last input is for shifting data

Index in position 2 exceeds array bounds (must not exceed 1).

Error in standalone_detector (line 71)
    yct = sum(cF0 .* ys(:, iChan), 1);

Error in datashift2 (line 58)
[st3, rez] = standalone_detector(rez, spkTh);

Error in kilosort3_master (line 23)
    rez = datashift2(rez, 1); % last input is for shifting data

In all of these cases, the coordinate definitions are not invalid (in a basic sense). I have traced through and tried to locate the specific causes, but I cannot follow the shift code well enough to be certain I'm going to change an algorithmic element or violate assumptions. Since this doesn't occur in KiloSort 2, it's due to the changes to 2.5 and 3 in this preprocessing step.

[eejd]

FYI, I was working on a PR to help with this, but I'd need assistance to determine this shift problem. However, to get to this problem, the following change is needed in standalone_detector.m:

- NchanNear = 10;
+ NchanNear = min(10,length(ops.xup));

[marius10p]

Sorry for the slow response. I think you should just disable drift correction completely, since it won't do anything meaningful on tetrode data. If you do that in Kilosort 2.5, you'll get back a version of Kilosort1. If you do that in Kilosort3, you'll get a fancier clustering algorithm that should be an improvement over Kilosort1. However, if drift is a big problem in your data, Kilosort2 remains the only version that can deal with that drift directly for tetrodes, since it does not use a geometric model like 2.5/3.0.

[AuroraWong]

I've setup KiloSort 2, 2.5 and 3 and am testing for tetrode recordings via SpikeInterface. The data is 30ks Open Ephys data, processed in SpikeInterface and sent identically to KiloSort2, 2.5 and 3. The test geometry is for running a single tetrode is:

channel_groups = {
    0:  {
        'channels': [0, 1, 2, 3],
        'geometry': [[0, 0], [0, 18], [18,0], [18, 18]]
    }
}

which produces:

chanMap =
     1     2     3     4

chanMap0ind =
     0     1     2     3

xcoords =
     0     0    18    18

ycoords =
     0    18     0    18

KiloSort 2.5 and 3 then fail with:

Time   0s. Computing whitening matrix.. 
Getting channel whitening matrix... 
Channel-whitening matrix computed. 
Time  13s. Loading raw data and applying filters... 
Time  59s. Finished preprocessing 886 batches. 
vertical pitch size is 18 
horizontal pitch size is 18 
     0     9    18

     5
--------------------------------------------------------------------------------Index in position 1 exceeds array bounds (must not exceed 4).Index in position 1 exceeds array bounds (must not exceed 4).

While kiloSort2 succeeds.

This appears to have been seen here: https://github.com/SpikeInterface/spikesorters/issues/198

Hello, I have run my tetrode recording on kilosort2 via matlab but it failed when one tetrode (4 channels) recordings was as input. Maybe something was wrong. Would you mind sending example tetrode recording and corresponding code project to me for learning. Thank you!