Assertion Error when using Phy2 #284
Comments
|
channel_map.npy is a row vector in KS2.5 whereas it's a column vector in KS2 - that might be the issue (I had a similar issue when trying to use part of the Allen ecephys repo after upgrading to KS2.5). Cheers Laurenz |
|
Ah, I see! May I ask how you fixed it? I am still quite new to program-language and e-phys... |
|
In Matlab you can readNPY('YourChannelMap.npy'); then make the variable you loaded a column vector by e.g. transposing someVariable'; then overwrite the channel_map by writeNPY(someVariable,'YourChannelMap.npy'); (readNPY and writeNPY are functions from the npy_matlab repo which KS uses, so you should have them). Laurenz |
|
Hmm... Thanks for the suggestion, but looking over the channel_map.npy files of two different files I found that they were similar, but one files is possible to open with phy, while the second is not... I got someone else to look at the KS2.5 outputs and they had not problems opening the files so it isn't (luckily) anything wrong with the recordings themselves, but this new error that has popped up is very, very confusing as - for some unknown reason - I am unable to open any of my more recent files with phy. Whatever more suggestions to which files or avenues to look into to troubleshoot this is greatly appreciated! |
|
Sorry to hear you still can't load your data into phy. But if you are saying someone else can load your data with no issue on a different machine (did I understand that correctly?), I would just delete phy on your machine and do a clean install. |
|
You did understand correctly, and this is where it get's peculiar. Not only have I done a clean install by deleting phy (normal and developers version) as well as the virtual environments on the computer, but I have a second computer that has the exact same problem with the exact same files... |
|
Okay. but you can open the data on someone elses machine who is running a recent version of phy? |
|
Correct. I updated to the most recent versions as well and I still can not open them. |
|
Have you deleted the .phy folder from your data directory before trying to reopen in phy after the clean install? |
|
Yes. |
|
Okay, The last thing I can suggest is to go and 'hack' the python script where the error occurs. You can e.g. temporarily add some print commands around the line where the error occurs, to print the variable contents of the things that cause the issue (best_channel and channel_ids judging from the error message you posted above) to your terminal. That might give you a better idea what's going wrong. |
|
Thanks for the assistance, I'll look into it. Troubleshooting since yesterday morning indicates that there is definitely something wonky with my personal phy set-up, but exactly what and how to fix it is vexing. Appreciate you taking the time to answer! |
|
No worries. I hope you find the issue quickly. Good luck |
|
Update: It does seem like there is some kind of issue with the KS2.5 output, as even if I specify the correct channels the params.py file gets saved with 383 channels (instead of 385) and the dat-path is set to be the temp_wh.dat file instead of the raw .bin file. Running the same files through KS2 I had no trouble accessing them with phy and params.py was saved correctly. |
|
I think it's supposed to be like that in KS 2.5. The output binary is the drift corrected raw data, so you should use that for the curation in phy (although tbh I don't see a big difference to the uncorrected raw data but I guess that depends how big the drift is in the dataset). That data will only contain the channels that are contained in the channel map, so it's missing at least the reference and the sync channel from a 1.0 probe. |
|
Ah, ok. The error does seem to be related to the channels then, and there is nothing wrong with the files themselves (like I said, I can open them without problems when run through with KS2), but could this have something to do with some warnings that were thrown up when doing MexGPUall.m regarding spikedetector3.cu? It finished with Mex Compiled Successfully but noticing that all of the warnings referenced channel-related variables (like NchanMax and NchanUp) it made me curious. |
|
@ldavr, the mex compilation warnings have nothing to do with it. I'm trying to reproduce the problem on my end, will keep you posted. |
|
Thank you very much @marius10p ! Really appreciate it. This showed up in the terminal right before the Assertion Error: 4908/1507984 values are nan in C:\Users\idavalik\Data\Neuropixels\Social_track_g0\Social_track_g0_imec0\similar_templates.npy, replacing by zero.�[0m In case it might help. |
|
I tried messing around with this more as well and the issue is indeed inconsistent. For this particular data set I have posted about here I found one bad channel that I tried disconnecting, but that still gave the same Assertion Error. However, when I connected channel 192 and also included the bad channel in the analysis (thus 0 channels being disconnected) I was able to open this specific file with phy. |
|
We see the same issue, here are more details. Our error occurs on a slightly different line number (but it seems to be the same stack trace) reproduced here anyway
Following the advice in the Github issue on the Phy page, we have tried to debug this error a number of ways.
|
|
We looked into the KS2.5 code and found that in trackAndSort (called by runTemplates which is called by learnAndSolve8b) the number of spikes per template nsp is 0 for many templates. This will create NaNs in rez.dWU in line 226 (dividing 0 by 0!), which later will be inherited to W and therefore to templates. |
|
just wanted to link this issue: cortex-lab/phy#1060 -- I believe it's the same problem as we (@torbenott & I) see with our data. |
|
I tried to locate the origin of this problem and independently came to the same conclusion as @torbenott. I found a simple workaround which solved the Phy problem for me – Here's line 226 from "trackAndSort.m": I replaced it with the version below, which only performs the division for templates with at least one spike: |
|
This seemed to work for me as well, though I'm a bit puzzled as it yielded rather different results when I compared KS2.5 with this workaround to KS2.5 as is with the same dataset. Have you had a similar experience? |
|
Actually never mind on that front. I had my test dataset mislabeled as KS2.5 when really it was KS2. Your fix seems to produce identical results, at least in that dataset. |
|
Thanks for working this out everyone. In principle there shouldn't be any spikes with nsp = 0, but I guess it sometimes happens. I pushed a fixed adding a small constant to the normalization in line 236 of trackAndSort. Reopen this issue if there are still problems. |
|
Sorry to comment on a closed issue, but the fix @marius10p implemented isn't working for me using KS 2.5. Oddly, that fix results in a CUDA error, but changing it as follows seems to work fine: `% rez.dWU = dWU1 ./ (1e-10 + single(reshape(nsp, [1,1,Nfilt]))); So apparently making this not single precision alleviates the CUDA error. Any idea why this might be the case? The error is a CUDA illegal address in reference to the following line from splitAllClusters.m (line 189)
|
Hi,
Recently after upgrading to Kilosort2.5 I am having trouble opening the spike sorted outputs to phy (standard and development version). I ran "--debug" on the command and I get this output + the error:
10:04:20.400 [D] init:68 Start capturing exceptions.
10:04:20.494 [W] model:591 Unreferenced clusters found in templates (generally not a problem)
10:04:20.547 [D] model:607 Loading spike clusters.
10:04:20.607 [W] model:613 Unreferenced clusters found in spike_clusters (generally not a problem)
10:04:20.663 [D] model:557 No channel shank file found.
10:04:20.664 [D] model:680 Loading templates.
10:04:20.666 [D] model:709 Templates are dense.
10:04:20.666 [W] model:655 Skipping spike waveforms that do not exist, they will be extracted on the fly from the raw data as needed.
10:04:20.667 [D] model:715 Loading the whitening matrix.
10:04:20.669 [D] model:722 Loading the inverse of the whitening matrix.
10:04:20.676 [W] model:55 4908/1507984 values are nan in C:\Users\idavalik\Data\Raw_SpikeGLX\Social_track_g0\Social_track_g0_imec0\similar_templates.npy, replacing by zero.
10:04:20.680 [D] model:751 Loading features.
10:04:20.682 [D] model:766 Features are sparse.
10:04:20.683 [D] model:788 Loading template features.
10:04:20.685 [D] model:798 Template features are sparse.
10:04:20.686 [D] model:492 Load
cluster_Amplitude.tsv.10:04:20.690 [D] model:492 Load
cluster_ContamPct.tsv.10:04:20.694 [D] model:492 Load
cluster_group.tsv.10:04:20.699 [D] model:492 Load
cluster_KSLabel.tsv.10:04:20.996 [D] context:100 Initialize joblib cache dir at
C:\Users\idavalik\Data\Raw_SpikeGLX\Social_track_g0\Social_track_g0_imec0\.phy.10:04:20.998 [D] context:101 Reducing the size of the cache if needed.
10:04:21.001 [D] base:102 Add filter
high_pass.10:04:21.002 [D] config:31 Load config file
C:\Users\idavalik\.phy\phy_config.py.10:04:21.003 [D] plugin:145 Loading 0 plugins.
10:04:21.006 [D] context:209 The file
C:\Users\idavalik\Data\Raw_SpikeGLX\Social_track_g0\Social_track_g0_imec0\.phy\new_cluster_id.pkldoesn't exist.10:04:21.089 [D] context:185 Save data to
C:\Users\idavalik\Data\Raw_SpikeGLX\Social_track_g0\Social_track_g0_imec0\.phy\spikes_per_cluster.pkl.10:04:21.227 [D] gui:463 Creating GUI.
10:04:21.229 [D] state:46 Load C:\Users\idavalik.phy\TemplateGUI\state.json for GUIState.
10:04:21.230 [D] state:46 Load C:\Users\idavalik\Data\Raw_SpikeGLX\Social_track_g0\Social_track_g0_imec0.phy\state.json for GUIState.
10:04:21.595 [E] init:62 An error has occurred (AssertionError):
Traceback (most recent call last):
File "C:\Users\idavalik\Anaconda3\envs\phy2dev\Scripts\phy-script.py", line 33, in
sys.exit(load_entry_point('phy', 'console_scripts', 'phy')())
File "C:\Users\idavalik\Anaconda3\envs\phy2dev\lib\site-packages\click\core.py", line 829, in call
return self.main(*args, **kwargs)
File "C:\Users\idavalik\Anaconda3\envs\phy2dev\lib\site-packages\click\core.py", line 782, in main
rv = self.invoke(ctx)
File "C:\Users\idavalik\Anaconda3\envs\phy2dev\lib\site-packages\click\core.py", line 1259, in invoke
return process_result(sub_ctx.command.invoke(sub_ctx))
File "C:\Users\idavalik\Anaconda3\envs\phy2dev\lib\site-packages\click\core.py", line 1066, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "C:\Users\idavalik\Anaconda3\envs\phy2dev\lib\site-packages\click\core.py", line 610, in invoke
return callback(*args, **kwargs)
File "C:\Users\idavalik\Anaconda3\envs\phy2dev\lib\site-packages\click\decorators.py", line 21, in new_func
return f(get_current_context(), *args, **kwargs)
File "c:\users\idavalik\phy\phy\apps_init.py", line 159, in cli_template_gui
template_gui(params_path, **kwargs)
File "c:\users\idavalik\phy\phy\apps\template\gui.py", line 218, in template_gui
gui = controller.create_gui()
File "c:\users\idavalik\phy\phy\apps\base.py", line 1697, in create_gui
self.supervisor.attach(gui)
File "c:\users\idavalik\phy\phy\cluster\supervisor.py", line 946, in attach
self._create_views(
File "c:\users\idavalik\phy\phy\cluster\supervisor.py", line 761, in _create_views
gui, data=self.cluster_info, columns=self.columns, sort=sort)
File "c:\users\idavalik\phy\phy\cluster\supervisor.py", line 921, in cluster_info
return [self.get_cluster_info(cluster_id) for cluster_id in self.clustering.cluster_ids]
File "c:\users\idavalik\phy\phy\cluster\supervisor.py", line 921, in
return [self.get_cluster_info(cluster_id) for cluster_id in self.clustering.cluster_ids]
File "c:\users\idavalik\phy\phy\cluster\supervisor.py", line 746, in get_cluster_info
out[key] = func(cluster_id)
File "c:\users\idavalik\phy\phy\apps\base.py", line 1194, in get_best_channel_label
return self._get_channel_labels([self.get_best_channel(cluster_id)])[0]
File "c:\users\idavalik\phy\phy\utils\context.py", line 154, in memcached
out = f(*args, **kwargs)
File "c:\users\idavalik\phy\phy\apps\base.py", line 1188, in get_best_channel
channel_ids = self.get_best_channels(cluster_id)
File "c:\users\idavalik\phy\phy\utils\context.py", line 154, in memcached
out = f(*args, **kwargs)
File "c:\users\idavalik\phy\phy\apps\template\gui.py", line 151, in get_best_channels
template = self.model.get_template(template_id)
File "c:\users\idavalik\phylib\phylib\io\model.py", line 928, in get_template
return self._get_template_dense(
File "c:\users\idavalik\phylib\phylib\io\model.py", line 867, in get_template_dense
channel_ids, amplitude, best_channel = self._find_best_channels(
File "c:\users\idavalik\phylib\phylib\io\model.py", line 844, in _find_best_channels
assert best_channel in channel_ids
AssertionError
I have no problems opening up earlier files that were processed with KS2, so from the help I have been getting trying to solve this it seems like there is a mismatch between something in phy and something in the way KS2.5 saves the output, but I am having trouble figuring out exactly where the error is coming from. If you have any insight into this I would greatly appreciate it!
The text was updated successfully, but these errors were encountered: