Stuck at learnAndSolve8b #307

claireward · 2021-01-26T00:03:18Z

I receive an error related to learnAndSolve8b prior to optimizing the templates. This is a fresh install of KS2.5 (no older versions) on a linux machine. I have run mexGPUall successfully but I haven't been able to sort successfully. Any suggestions? Thank you!

Below is the output prior to the error:

145.33 sec, 2336 batches, 1461438 spikes
time 183.66, Shifted up/down 2336 batches.
(Drift map generated)

Error messages:

Error using gpuArray/subsref
Subscript indices must either be real positive integers or
logicals.
Error in extractTemplatesfromSnippets (line 56)
wTEMP = dd(:, round(linspace(1, size(dd,2), nPCs)));
Error in learnTemplates (line 19)
[wTEMP, wPCA] = extractTemplatesfromSnippets(rez,
NrankPC);
Error in learnAndSolve8b (line 35)
rez = learnTemplates(rez, rez.iorig);
Error in main_kilosort (line 46)
rez = learnAndSolve8b(rez, iseed)

HiroyukiKato7 · 2021-01-29T20:24:51Z

I am having exactly the same issue with Win10, Matlab2018b. I have been successfully running Kilosort 1, and I'm trying to update it to Kilosort 2.5 on the same PC. I tried both GPU and script methods and got the same error.

In the extractTemplatesfromSnippets function line 56, size(dd,2) is giving 0, and that's causing this error. Somehow, extractTemplatesfromSnippets seems to be working fine during datashift2 operation, but it fails during learnAndSolve8b-- [row, col, mu] = isolated_peaks_new(dataRAW,ops) returns 0x1 gpuArray for row and col (line 32). This probably means that the change to rez during datashift2 is causing some problem, but I have no clue.

I would appreciate your help.

Tampu · 2021-01-29T23:13:02Z

I am having the same issue as well using Kilosort 2.5 with Win10, Matlab 2018a. eMouse drift works fine.
I am using 64GB RAM and have increased the windows page file size to overcome the out of memory issue. I have tried sorting longer(~10 mins) and shorter(~2 mins) recordings with good spiking activity. I have adopted the changes mentioned in issues #255 and #257 to get past few errors but I keep getting the following error.

806.53 sec, 1101 batches, 317092 spikes
879.77 sec, 1201 batches, 345603 spikes
953.02 sec, 1301 batches, 374568 spikes
990.38 sec, 1352 batches, 389271 spikes
time 1086.02, Shifted up/down 1352 batches.
Error using gpuArray/subsref
Subscript indices must either be real positive integers or logicals.

Error in extractTemplatesfromSnippets (line 56)
wTEMP = dd(:, round(linspace(1, size(dd,2), nPCs)));

Error in learnTemplates (line 19)
[wTEMP, wPCA] = extractTemplatesfromSnippets(rez, NrankPC);

Error in learnAndSolve8b (line 35)
rez = learnTemplates(rez, rez.iorig);

Error in main_kilosort (line 45)
rez = learnAndSolve8b(rez, iseed);

I am pasting these details for your reference:
The details of GPU I am using is below:
gpuDevice

ans =

CUDADevice with properties:

                  Name: 'Quadro P400'
                 Index: 1
     ComputeCapability: '6.1'
        SupportsDouble: 1
         DriverVersion: 11
        ToolkitVersion: 9
    MaxThreadsPerBlock: 1024
      MaxShmemPerBlock: 49152
    MaxThreadBlockSize: [1024 1024 64]
           MaxGridSize: [2.1475e+09 65535 65535]
             SIMDWidth: 32
           TotalMemory: 2.1475e+09
       AvailableMemory: 1.6654e+09
   MultiprocessorCount: 2
          ClockRateKHz: 1252500
           ComputeMode: 'Default'
  GPUOverlapsTransfers: 1
KernelExecutionTimeout: 1
      CanMapHostMemory: 1
       DeviceSupported: 1
        DeviceSelected: 1

Please have a look at this @marius10p and kindly help us. Thank you for your time.

marius10p · 2021-01-29T23:21:10Z

hey guys, not sure why it's not finding those spikes, but can you please try to increase ops.spkTh from -6 to -4? Let me know what happens then,

HiroyukiKato7 · 2021-01-29T23:39:24Z

I just tried increasing ops.spkTh to -4, but the same error happened.

HiroyukiKato7 · 2021-01-29T23:58:57Z

rez_files.zip

I attached rez that is fed to datashift2.m and another one that is fed to learnAndSolve8b. The only variables that are fed to extractTemplatesfromSnippets are rez and nPC, so there must be something in rez that makes it work in datashift2, but not in learnAndSolve8b. (In this one, I used ops.spkTh = -6 again.)

marius10p · 2021-01-30T00:02:16Z

They work on separate files, one works on the raw data and the other on the drift-corrected data. Did your drift plots look ok?

Tampu · 2021-01-30T00:03:08Z

I increased it to -4 still the same error. I tried to track the variables, it might help to narrow down.

Right after this the error pops up.

HiroyukiKato7 · 2021-01-30T00:06:09Z

I have not successfully run the entire procedure yet, so I'm not sure how these figures should look like, but attached are the figures that I got with datashift2.

marius10p · 2021-01-30T00:15:13Z

What probe is this? What is the channel geometry? It looks like you don't have enough vertical channels to infer drift.

HiroyukiKato7 · 2021-01-30T00:16:10Z

Sorry, probably the one that I uploaded was a strange dataset. This one might look more normal.

HiroyukiKato7 · 2021-01-30T00:17:02Z

This is recorded with Cambridge Neurotech 64ch linear probe. 20um spacing between each channel.

marius10p · 2021-01-30T00:19:32Z

Hmm, then the geometry doesn't seem to be specified correctly, please make sure the xcoords and ycoords make sense. It should infer a pitch of 20 , not 1.

…

On Fri, Jan 29, 2021, 7:17 PM HiroyukiKato7 ***@***.***> wrote: This is recorded with Cambridge Neurotech 64ch linear probe. 20um spacing between each channel. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#307 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AA6AYDQKV4DDVTMCMLOF65TS4NFYXANCNFSM4WSSAQZA> .

HiroyukiKato7 · 2021-01-30T00:24:38Z

Right now, xcoords = ones(Nchannels,1) and ycoords = [1:Nchannels]', where Nchannels = 64. I have not changed it from the original createChannelMapFile. Should ycoords represent the actual distance in micrometer?

marius10p · 2021-01-30T00:32:30Z

Yes

…

On Fri, Jan 29, 2021, 7:24 PM HiroyukiKato7 ***@***.***> wrote: Right now, xcoords = ones(Nchannels,1) and ycoords = [1:Nchannels]', where Nchannels = 64. I have not changed it from the original createChannelMapFile. Should ycoords represent the actual distance in micrometer? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#307 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AA6AYDWSNNDJZERDBM3HZTDS4NGVJANCNFSM4WSSAQZA> .

HiroyukiKato7 · 2021-01-30T00:49:59Z

Okay, I changed ycoords to 20* [1:Nchannels]' and it started to run learnAndSolve8b function without error now. Attached is the drift trace figures.
It looks like this solved the issue?

Thanks for your help!

marius10p · 2021-01-30T01:08:17Z

Yes, you might also want to reduce the number of datashift blocks, the default is for a 4mm Neuropixels.

HiroyukiKato7 · 2021-01-30T01:27:25Z

Do you mean ops.nblocks? Currently the value is 5. For 1mm Cambridge probe, is 1 or 2 enough?

Tampu · 2021-01-30T01:35:49Z

I guess this was the mistake in my case too. I added a pitch of 30 to ycoords and then got past the error. Now I am able to run Kilosort2.5. Thank you very much for your help!

Hmm, then the geometry doesn't seem to be specified correctly, please make sure the xcoords and ycoords make sense. It should infer a pitch of 20 , not 1.
…
On Fri, Jan 29, 2021, 7:17 PM HiroyukiKato7 @.***> wrote: This is recorded with Cambridge Neurotech 64ch linear probe. 20um spacing between each channel. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#307 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AA6AYDQKV4DDVTMCMLOF65TS4NFYXANCNFSM4WSSAQZA .

marius10p · 2021-01-30T14:02:52Z

It's actually ops.datashift right now for the number of blocks. I will change this soon so it makes more sense. 2 blocks should be fine for 1mm (it will actually make 3, with 50% overlap between consecutive blocks).

marius10p · 2021-01-30T14:06:58Z

@claireward does this solve your problem as well?

marius10p · 2021-01-30T19:08:49Z

Nvm, I actually had changed it already, it's ops.nblocks.

It's actually ops.datashift right now for the number of blocks. I will change this soon so it makes more sense. 2 blocks should be fine for 1mm (it will actually make 3, with 50% overlap between consecutive blocks).

HiroyukiKato7 · 2021-01-30T19:46:03Z

Thanks! When I have multiple probes (e.g. 2 * 64ch 1mm probes), what number should I use for ops.nblocks?

marius10p · 2021-01-30T20:02:31Z

If you are assuming that they go up and down together, then you can sort them together with the same settings. If they are far from each other, they probably have different drift and you have to split them into two files.

…

On Sat, 30 Jan 2021 at 14:46, HiroyukiKato7 ***@***.***> wrote: Thanks! When I have multiple probes (e.g. 2 * 64ch 1mm probes), what number should I use for ops.nblocks? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#307 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AA6AYDXRK3LXVJ5V46FC4ALS4ROYPANCNFSM4WSSAQZA> .

HiroyukiKato7 · 2021-01-30T20:18:22Z

I see. Open-ephys records them together, but I guess I can split them into two binary files afterwards. Thanks again for your help!

marius10p · 2021-02-03T20:04:00Z

If anyone in this thread still has problems, please let me know, Closing now.

marius10p closed this as completed Feb 3, 2021

carsen-stringer mentioned this issue Feb 29, 2024

adding rtd and badges #595

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Stuck at learnAndSolve8b #307

Stuck at learnAndSolve8b #307

claireward commented Jan 26, 2021

HiroyukiKato7 commented Jan 29, 2021

Tampu commented Jan 29, 2021

marius10p commented Jan 29, 2021

HiroyukiKato7 commented Jan 29, 2021

HiroyukiKato7 commented Jan 29, 2021

marius10p commented Jan 30, 2021

Tampu commented Jan 30, 2021

HiroyukiKato7 commented Jan 30, 2021

marius10p commented Jan 30, 2021

HiroyukiKato7 commented Jan 30, 2021

HiroyukiKato7 commented Jan 30, 2021

marius10p commented Jan 30, 2021 via email

HiroyukiKato7 commented Jan 30, 2021

marius10p commented Jan 30, 2021 via email

HiroyukiKato7 commented Jan 30, 2021

marius10p commented Jan 30, 2021

HiroyukiKato7 commented Jan 30, 2021

Tampu commented Jan 30, 2021

marius10p commented Jan 30, 2021

marius10p commented Jan 30, 2021

marius10p commented Jan 30, 2021

HiroyukiKato7 commented Jan 30, 2021

marius10p commented Jan 30, 2021 via email

HiroyukiKato7 commented Jan 30, 2021

marius10p commented Feb 3, 2021

Stuck at learnAndSolve8b #307

Stuck at learnAndSolve8b #307

Comments

claireward commented Jan 26, 2021

HiroyukiKato7 commented Jan 29, 2021

Tampu commented Jan 29, 2021

marius10p commented Jan 29, 2021

HiroyukiKato7 commented Jan 29, 2021

HiroyukiKato7 commented Jan 29, 2021

marius10p commented Jan 30, 2021

Tampu commented Jan 30, 2021

HiroyukiKato7 commented Jan 30, 2021

marius10p commented Jan 30, 2021

HiroyukiKato7 commented Jan 30, 2021

HiroyukiKato7 commented Jan 30, 2021

marius10p commented Jan 30, 2021 via email

HiroyukiKato7 commented Jan 30, 2021

marius10p commented Jan 30, 2021 via email

HiroyukiKato7 commented Jan 30, 2021

marius10p commented Jan 30, 2021

HiroyukiKato7 commented Jan 30, 2021

Tampu commented Jan 30, 2021

marius10p commented Jan 30, 2021

marius10p commented Jan 30, 2021

marius10p commented Jan 30, 2021

HiroyukiKato7 commented Jan 30, 2021

marius10p commented Jan 30, 2021 via email

HiroyukiKato7 commented Jan 30, 2021

marius10p commented Feb 3, 2021