more clusters when dat_path in Phy params revert back to kilosort 2 version

[babaq]

Hi @marius10p

I am testing the latest main branch of kilosort3 with recordings from Neuropixels Phase3A, where function rezToPhy is replaced with rezToPhy2.

I noticed the original line for dat_path in rezToPhy2 is commented out, and replaced with the path to the temp_wh.dat:

%         fprintf(fid,['dat_path = ''',fname ext '''\n']);
        fprintf(fid,['dat_path = ''', strrep(rez.ops.fproc, '\', '/') '''\n']);

I couldn't understand this modification, and the Phy reports:

11:31:46.959 [W] model:545            Unreferenced clusters found in templates (generally not a problem)
11:31:46.975 [W] model:567            Unreferenced clusters found in spike_clusters (generally not a problem)
11:31:47.094 [W] model:397            There are 3294/134330 spikes after the end of the recording.

when it starts and seems to be slower when using temp_wh.dat. Moreover, I can see more good clusters in Phy when I changed dat_path back to the origianl *.imec.ap.bin file, and Phy didn't report the There are 3294/134330 spikes after the end of the recording..

Is this modification intentional, and why?

Best,
Alex

[czuba]

The switch to temp_wh.dat being the primary continuous data source came up a while back, with the transition to drift correction via the 'datashift' method (i.e. drift correction via spatial [channel] shifts, rather than temporal shifts).
See issue #287

I can't speak to the processing speed or appearance of more clusters/spikes when manually changing params to point to the original .dat file, but if any drift correction is applied (ops.nblocks ≥ 1), your sorted spike times/channels will only correspond to the intended voltage samples in the original data file (ops.fbinary) by coincidence.

[babaq]

@czuba Thanks for the explanation and link to the issue. I agree if the temp_wh.dat is necessary for phy and calculating QC metrics(mean waveform, etc.), then it better be renamed instead of indicating a temporary file.

I would be still confused about the message There are 3294/134330 spikes after the end of the recording. Is this means something wrong with my kilosort, how could it be spikes after the end of the recording?(maybe I should use rigid drift correction?)