[MRG] Conversion of somato data to BIDS #6414

sappelhoff · 2019-06-05T12:07:44Z

for reference, see this gitter chat

Current structure:

MNE-somato-data
├── CHANGES
├── code
│   ├── convert_somato_data.py
│   └── README
├── dataset_description.json
├── derivatives
│   ├── freesurfer
│   │   └── subjects
│   │       ├── 01
│   │       │   ├── bem
│   │       │   ├── label
│   │       │   ├── mri
│   │       │   ├── scripts
│   │       │   ├── stats
│   │       │   ├── surf
│   │       │   ├── tmp
│   │       │   ├── touch
│   │       │   └── trash
│   │       └── fsaverage -> /usr/local/freesurfer/subjects/fsaverage
│   └── sub-01
│       └── sub-01_task-somato-fwd.fif
├── participants.json
├── participants.tsv
├── README
└── sub-01
    ├── anat
    │   ├── sub-01_T1w.json
    │   └── sub-01_T1w.nii.gz
    ├── meg
    │   ├── sub-01_coordsystem.json
    │   ├── sub-01_task-somato_channels.tsv
    │   ├── sub-01_task-somato_events.tsv
    │   ├── sub-01_task-somato_meg.fif
    │   └── sub-01_task-somato_meg.json
    └── sub-01_scans.tsv

19 directories, 16 files

sappelhoff · 2019-06-05T12:42:34Z

It seems like the BIDS naming clashes with the MNE naming convention:

This reminds me of: #4950

agramfort · 2019-06-05T12:50:38Z

yes this has never been fixed in mne-python but it should not warn for raw bids files

sappelhoff · 2019-06-05T12:54:07Z

yes this has never been fixed in mne-python but it should not warn for raw bids files

I'll give it a shot

jasmainak · 2019-06-05T13:40:19Z

can you share link to rendered examples?

sappelhoff · 2019-06-05T13:42:31Z

can you share link to rendered examples?

sure, but they did not run completely due to the runtime warning that the filename for RAW does not follow MNE conventions:

jasmainak · 2019-06-05T13:44:40Z

ah okay, let us know when that works.

Also I think I prefer to change mne_source_alignment to mne_coregistration

examples/inverse/plot_dics_source_power.py

jasmainak · 2019-06-05T13:52:20Z

I would try to be as much close to BEP003 as possible. So we need to do minimal updates later. For example, you need dataset_description.json:

Derived dataset and pipeline description
In keeping with the BIDS-Raw standard, a dataset_description.json file MUST be found at the top level of the particular pipeline:

Similarly:

Each derivative file SHOULD be described by a JSON file provided as a sidecar or higher up in the hierarchy of the derived dataset

sappelhoff · 2019-06-05T13:54:10Z

I agree,

the most current BEP for derivatives is here: bids-standard/bids-specification#207

sappelhoff · 2019-06-05T14:10:12Z

Would these be the correct places to change the naming convention (i.e., accept BIDS endings that are *_meg.fif)

mne-python/mne/io/base.py

Lines 1572 to 1574 in e1a160c

    
           check_fname(fname, 'raw', ('raw.fif', 'raw_sss.fif', 'raw_tsss.fif', 
        
                                      'raw.fif.gz', 'raw_sss.fif.gz', 
        
                                      'raw_tsss.fif.gz'))

and

mne-python/mne/io/fiff/raw.py

Lines 136 to 138 in e1a160c

    
           check_fname(fname, 'raw', ('raw.fif', 'raw_sss.fif', 
        
                                      'raw_tsss.fif', 'raw.fif.gz', 
        
                                      'raw_sss.fif.gz', 'raw_tsss.fif.gz'))

It would also require changing a couple of docstrings and also some docs, such as

https://github.com/mne-tools/mne-python/blob/master/doc/tutorials/report.rst

jasmainak · 2019-06-05T14:12:32Z

yep! Don't break the report while you do so. I think it expects certain file format in report.parse_folders() :-)

codecov · 2019-06-05T15:40:01Z

Codecov Report

Merging #6414 into master will decrease coverage by <.01%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master    #6414      +/-   ##
==========================================
- Coverage    89.4%   89.39%   -0.01%     
==========================================
  Files         415      415              
  Lines       74985    74985              
  Branches    12330    12330              
==========================================
- Hits        67039    67035       -4     
- Misses       5118     5120       +2     
- Partials     2828     2830       +2

sappelhoff · 2019-06-06T16:23:00Z

@larsoner the new somato data is not found for building the docs ... could this be because we are caching it and the cache is still old?

I changed the md5 hash and the URL though:

https://github.com/mne-tools/mne-python/pull/6414/files#diff-00583e36a75a694e071f9fc054a71c08

larsoner · 2019-06-06T18:12:27Z

The error is:

FileNotFoundError: [Errno 2] No such file or directory: '/home/circleci/mne_data/MNE-somato-data/sub-01/meg/sub-01_task-somato_meg.fif'

When I download your tar.gz and look at the contents I do not see a MNE-somato-data folder but rather a MNE-somato-data-bids folder. Confirmed this is likely the problem with:

$ python -c "import mne; mne.datasets.somato.data_path(force_update=True, verbose=True)"
Downloading archive MNE-somato-data.tar.gz to /home/larsoner/mne_data
...
$ ls ~/mne_data/
...   MNE-somato-data-bids ...

sappelhoff · 2019-06-06T20:08:55Z

🤦‍♂️ okay, I re-uploaded with proper folder naming. Let's see how this goes.

PS: Thanks a lot for spotting it

jasmainak · 2019-06-07T12:38:01Z

@sappelhoff did you run the dataset through the validator? any issues you faced? Is it compliant?

otherwise PR looks clean to me.

sappelhoff · 2019-06-07T12:40:58Z

yep, see:

(base) stefanappelhoff@arc-lin-004309:~/Desktop$ bids-validator MNE-somato-data
This dataset appears to be BIDS compatible.

        Summary:                  Available Tasks:        Available Modalities: 
        11 Files, 334.48MB        somato                  T1w                   
        1 - Subject                                       coordsystem           
        1 - Session                                       channels              
                                                          events                
                                                          meg                   


	If you have any questions, please post on https://neurostars.org/tags/bids.

I also added dataset_description.json files for the derivatives ... they are not completely compliant (e.g., the freesurfer directory is still as output by recon-all), but as much as is reasonable given the WIP state of derivatives.

jasmainak · 2019-06-07T12:46:07Z

sounds good. +1 to merge from my end once CIs are happy

teonbrooks · 2019-06-08T00:23:05Z

just curious @sappelhoff, are you adding these derivative files to the BEP 021?

examples/inverse/plot_dics_source_power.py

examples/time_frequency/plot_time_frequency_global_field_power.py

sappelhoff · 2019-06-10T11:22:16Z

@teonbrooks not yet - but we just had a BIDS discussion at OHBM and we'll get more focused on the derivatives starting soon :-)

... probably with a survey of what most people need (doing a "data driven 80-20" as @jasmainak put it ;-) )

sappelhoff · 2019-06-10T11:28:12Z

@agramfort with using anchors: do you mean I should do it like this? (see below)

.. _`somatosensory dataset`: https://martinos.org/mne/stable/manual/datasets_index.html?#somatosensory

=========================================
Compute source power using DICS beamfomer
=========================================

Compute a Dynamic Imaging of Coherent Sources (DICS) [1]_ filter from
single-trial activity to estimate source power across a frequency band. This
example demonstrates how to source localize the event-related synchronization
(ERS) of beta band activity in the `somatosensory dataset`_

agramfort · 2019-06-10T14:41:21Z

you need to insert .. _somato-dataset: in L124 in datasets_index.rst and then you'll be able to refer to this section with :ref:`somato-dataset`

jasmainak

I'm blocking the PR so as to remind us that we should use the digitization coordinates as in the spec instead of the trans files.

sappelhoff · 2019-06-12T13:26:57Z

you need to insert .. _somato-dataset: in L124 in datasets_index.rst and then you'll be able to refer to this section with :ref:somato-dataset
--

to remind myself, this is the rst docs section dealing with that: https://www.sphinx-doc.org/en/1.5/markup/inline.html#cross-referencing-arbitrary-locations

sappelhoff · 2019-07-29T09:46:02Z

rendered:

agramfort

besides subj -> subject LGTM

examples/inverse/plot_dics_source_power.py

jasmainak · 2019-07-29T18:36:50Z

Thanks @sappelhoff !

massich · 2019-08-01T11:41:06Z

examples/inverse/plot_dics_source_power.py

+subject = '01'
+task = 'somato'
+raw_fname = op.join(data_path, 'sub-{}'.format(subject), 'meg',
+                    'sub-{}_task-{}_meg.fif'.format(subject, task))


I know I'm late on the review cycle and this is merged, but do you guys find worth to create a helper function that allows composing the name?

def fname_helper(subject, task='somato', data_type_prefix=''): return op.join( 'sub-{0:02d}'.format(subject), data_type_prefix, 'sub-{0:02d}_task-{}_meg.fif'.format(subject, task) )

from mne.datasets.somato import data_path, fname_helper raw_fname = op.join(data_path(), fname_helper(subject=1, data_type_prefix='meg')) fname_fwd = op.join(data_path, 'derivatives', fname_helper(subject=1))

ups !! the file names are not the same.

one is _meg.fif the other one is _fwd.fif

The derivatives is not set in stone, so I would not make a new function. There is already one in mne-bids for the existing BIDS spec: https://mne.tools/mne-bids/generated/mne_bids.make_bids_basename.html#mne_bids.make_bids_basename.

massich · 2019-08-01T11:46:06Z

Also reading the artifacts I found that we are using a private function in a tutorial:

mne-python/examples/time_frequency/plot_time_frequency_global_field_power.py

Line 56 in cc836c2

from mne.stats import _bootstrap_ci

Shall we make the function public? wrap wherever is exposed and make this one public?

agramfort · 2019-08-01T12:36:14Z

Also reading the artifacts I found that we are using a private function in a tutorial: https://github.com/mne-tools/mne-python/blob/cc836c2635b7a720ec6c60074a1997fe52d7ca2f/examples/time_frequency/plot_time_frequency_global_field_power.py#L56 Shall we make the function public? wrap wherever is exposed and make this one public?

nice catch ! this is evil. @sappelhoff can you fix? or anyone?

sappelhoff · 2019-08-01T16:26:11Z

Can @massich elaborate a bit on his intention for the fix?

jasmainak · 2019-08-01T19:18:27Z

you're not supposed to use private functions in examples ...

sappelhoff · 2019-08-01T20:09:50Z

Yes, I learned that repeatedly through you and Alex :-D

But I was wondering what @massich meant with

wrap wherever is exposed and make this one public?

jasmainak · 2019-08-01T20:15:46Z

well if you wrap another function around it, you can potentially expose fewer parameters publicly while still keeping the flexibility of the private function. Not sure if that's necessary though.

* adjust freq tut * adjust dics exp * adjust global field power exp * temporarily change hash and url for somato * make more use of subj variable * use v2 of somato bids set * ignore long lines due to links in rst * fix typo * add somato url from MNE osf repo * fix sphinx rst links * update examples for new formatting, no new data yet * update to most recent somato data * update * fix path * fix rst link * subj --> subject

larsoner · 2020-09-29T13:10:44Z

@sappelhoff looks like the -5120-bem.fif and -5120-bem-sol.fif files are missing from the bem directory. Was this intentional or just an oversight? If the latter, I could add them but do I need to do anything at the BIDS end? FWIW this can be used to create them:

# in ~/mne_data/MNE-somato-data/derivatives/freesurfer/subjects
model = mne.make_bem_model('01', conductivity=(0.3,), subjects_dir='.', verbose=True)
mne.write_bem_surfaces('01/bem/01-5120-bem.fif', model)
bem = mne.make_bem_solution(model, verbose=True)
mne.write_bem_solution('01/bem/01-5120-bem-sol.fif', bem)

britta-wstnr · 2020-09-29T13:13:36Z

Can confirm -- tried to run some older code after updating somato. The BEM solution seems to be missing.

sappelhoff · 2020-09-29T13:14:10Z

Was this intentional or just an oversight?

sounds like an oversight to me because I have no clue what these files do.

This is the conversion script: https://github.com/mne-tools/mne-scripts/blob/master/bids_conversion/convert_somato_data.py

As long as the files are being put into the derivatives directory, it does not hurt BIDS. But the best would probably be to fix the conversion script.

larsoner · 2020-09-29T13:16:51Z

Okay, will push a commit there and update the dataset on OSF

sappelhoff · 2020-09-29T13:18:51Z

Okay, will push a commit there and update the dataset on OSF

thanks a lot @larsoner and sorry to cause problems.

jasmainak reviewed Jun 5, 2019

View reviewed changes

examples/inverse/plot_dics_source_power.py Outdated Show resolved Hide resolved

sappelhoff mentioned this pull request Jun 5, 2019

BIDS compliance: allow raw fif files to end on _meg.fif #6416

Merged

sappelhoff force-pushed the somato_exp branch from eb3c303 to 5e73f62 Compare June 6, 2019 15:01

agramfort reviewed Jun 8, 2019

View reviewed changes

examples/inverse/plot_dics_source_power.py Outdated Show resolved Hide resolved

agramfort reviewed Jun 8, 2019

View reviewed changes

examples/time_frequency/plot_time_frequency_global_field_power.py Outdated Show resolved Hide resolved

jasmainak requested changes Jun 10, 2019

View reviewed changes

sappelhoff force-pushed the somato_exp branch from c666825 to f0f17fb Compare June 12, 2019 13:40

sappelhoff mentioned this pull request Jun 21, 2019

Read fiducials from BIDS files mne-tools/mne-bids#210

Closed

sappelhoff added 3 commits July 29, 2019 10:26

update

c85f687

fix path

2173315

fix rst link

77f7c82

sappelhoff requested review from jasmainak and agramfort July 29, 2019 11:38

sappelhoff changed the title ~~Conversion of somato data to BIDS~~ [MRG] Conversion of somato data to BIDS Jul 29, 2019

agramfort requested changes Jul 29, 2019

View reviewed changes

examples/inverse/plot_dics_source_power.py Outdated Show resolved Hide resolved

subj --> subject

b8f23f5

agramfort approved these changes Jul 29, 2019

View reviewed changes

jasmainak approved these changes Jul 29, 2019

View reviewed changes

jasmainak merged commit 3acb8dc into mne-tools:master Jul 29, 2019

sappelhoff deleted the somato_exp branch July 30, 2019 08:06

massich reviewed Aug 1, 2019

View reviewed changes

sappelhoff mentioned this pull request Aug 2, 2019

make bootstrap ci public #6624

Merged

larsoner mentioned this pull request Sep 15, 2019

BUG: Somato BIDS dataset derivatives need to be updated #6770

Closed

mathurinm mentioned this pull request Nov 4, 2019

[WIP] mne-somato now uses BIDS format alphacsc/alphacsc#28

Closed

[MRG] Conversion of somato data to BIDS #6414

[MRG] Conversion of somato data to BIDS #6414

Conversation

sappelhoff commented Jun 5, 2019 • edited Loading

sappelhoff commented Jun 5, 2019

agramfort commented Jun 5, 2019 via email

sappelhoff commented Jun 5, 2019

jasmainak commented Jun 5, 2019

sappelhoff commented Jun 5, 2019

jasmainak commented Jun 5, 2019

jasmainak commented Jun 5, 2019

sappelhoff commented Jun 5, 2019

sappelhoff commented Jun 5, 2019

jasmainak commented Jun 5, 2019

codecov bot commented Jun 5, 2019 • edited Loading

Codecov Report

sappelhoff commented Jun 6, 2019

larsoner commented Jun 6, 2019

sappelhoff commented Jun 6, 2019 • edited Loading

jasmainak commented Jun 7, 2019

sappelhoff commented Jun 7, 2019 • edited Loading

jasmainak commented Jun 7, 2019

teonbrooks commented Jun 8, 2019

sappelhoff commented Jun 10, 2019

sappelhoff commented Jun 10, 2019

agramfort commented Jun 10, 2019 via email • edited by massich Loading

jasmainak left a comment

Choose a reason for hiding this comment

sappelhoff commented Jun 12, 2019

sappelhoff commented Jul 29, 2019

agramfort left a comment

Choose a reason for hiding this comment

jasmainak commented Jul 29, 2019

massich Aug 1, 2019 • edited Loading

Choose a reason for hiding this comment

massich Aug 1, 2019

Choose a reason for hiding this comment

jasmainak Aug 1, 2019

Choose a reason for hiding this comment

massich commented Aug 1, 2019

agramfort commented Aug 1, 2019 via email

sappelhoff commented Aug 1, 2019

jasmainak commented Aug 1, 2019

sappelhoff commented Aug 1, 2019

jasmainak commented Aug 1, 2019

larsoner commented Sep 29, 2020

britta-wstnr commented Sep 29, 2020

sappelhoff commented Sep 29, 2020

larsoner commented Sep 29, 2020

sappelhoff commented Sep 29, 2020

sappelhoff commented Jun 5, 2019 •

edited

Loading

codecov bot commented Jun 5, 2019 •

edited

Loading

sappelhoff commented Jun 6, 2019 •

edited

Loading

sappelhoff commented Jun 7, 2019 •

edited

Loading

agramfort commented Jun 10, 2019 via email •

edited by massich

Loading

massich Aug 1, 2019 •

edited

Loading