PEPOLAR field map can't be recognised after upgrading to version 22.0.0 #286

manzouri · 2022-09-09T12:41:08Z

What happened?

I could use my PEPOLAR filed maps with earlier version but v 22 can't recognise them any Moreland ignore field maps

What command did you use?

fmriprep-docker $bids_root_dir $out_dir/ \
    participant \
    --participant-label $subject \
    --skip-bids-validation \
    --md-only-boilerplate \
    --output-spaces MNI152NLin2009cAsym \
    --fs-no-reconall \
    --fs-license-file /usr/local/freesurfer/license.txt \

What version of fMRIPrep are you running?

22.0.0

How are you running fMRIPrep?

Docker

Is your data BIDS valid?

Yes

Are you reusing any previously computed results?

No

Please copy and paste any relevant log output.

It runs smoothly but without field maps

Additional information / screenshots

No response

mgxd · 2022-09-09T15:29:02Z

Can you remove --skip-bids-validation and add -vv to your command and retry? This will give us more information on what's happening.

manzouri · 2022-09-12T19:33:14Z

It runs smoothly and generates the report without error but without field maps. I attach the log file here.
report.csv

manzouri · 2022-09-15T09:34:47Z

Hi @mgxd , did you get any chance to look at this?

mgxd · 2022-09-15T13:28:50Z

Yeah I took a look, but nothing in the report jumps out. Perhaps you can share the directory tree of the subject you are running into trouble with, along with the relevant fieldmap metadata (IntendedFor or B0FieldIdentifier fields).

manzouri · 2022-09-16T11:20:19Z

Here is the tree:

├── dataset_description.json
└── sub-102
    └── ses-01
        ├── anat
        │   ├── sub-102_ses-01_run-01_T1w.json
        │   ├── sub-102_ses-01_run-01_T1w.nii.gz
        │   ├── sub-102_ses-01_run-01_T2w.json
        │   └── sub-102_ses-01_run-01_T2w.nii.gz
        ├── fmap
        │   ├── sub-102_ses-01_dir-AP_epi.json
        │   └── sub-102_ses-01_dir-AP_epi.nii.gz
        ├── func
        │   ├── sub-102_ses-01_task-rs_run-01_bold.json
        │   └── sub-102_ses-01_task-rs_run-01_bold.nii.gz
        └── sub-102_ses-01_scans.tsv

And I added this to my fieldmap json file :"IntendedFor":["ses-01/func/sub-102_ses-01_task-rs_run-01_bold.nii.gz"]
When I run fmriprep V21.0.2 on the exactly same raw_data it does the filedmap correction.

mgxd · 2022-09-16T20:03:22Z

I'm not sure what is happening, but I can try reproducing this - I'll just need any the (non-identifying) metadata for the _epi and _bold files.

manzouri · 2022-09-19T08:25:04Z

@mgxd I made attached json files , hope those help.
metadata.zip

mgxd · 2022-09-19T15:58:25Z

Local test

sub-102/
└── ses-01
    ├── anat
    │   ├── sub-102_ses-01_run-01_T1w.json
    │   ├── sub-102_ses-01_run-01_T1w.nii.gz
    │   ├── sub-102_ses-01_run-01_T2w.json
    │   └── sub-102_ses-01_run-01_T2w.nii.gz
    ├── fmap
    │   ├── sub-102_ses-01_dir-AP_epi.json
    │   └── sub-102_ses-01_dir-AP_epi.nii.gz
    └── func
        ├── sub-102_ses-01_task-rs_run-01_bold.json
        └── sub-102_ses-01_task-rs_run-01_bold.nii.gz

$ sdcflows-find-estimators . 
Estimation for </private/tmp/tst> complete. Found:
	sub-102
		FieldmapEstimation(sources=<2 files>, method=<EstimatorType.PEPOLAR: 2>, bids_id='auto_00000')
			j-	ses-01/fmap/sub-102_ses-01_dir-AP_epi.nii.gz
			j	ses-01/func/sub-102_ses-01_task-rs_run-01_bold.nii.gz

It seems sdcflows' detection is working as expected...and looking through the changes between 21.0.x and 22.0.x, I'm not seeing any signs of regression. (cc @effigies @oesteban in case either of you have further ideas)

oesteban · 2022-09-19T16:58:49Z

Perhaps @manzouri could run something like:

docker --rm -it --entrypoint=/bin/bash -v $bids_root_dir:/data:ro nipreps/fmriprep:22.0.0 find /data/sub-102/ses-01 -type f

and check that the fieldmaps are indeed accessible from the process.

For ourselves, we could add printing out the subject(s)' data tree to the report when executed very verbose. Evidently, none of this resolves the problem, but would certainly help when debugging these issues.

manzouri · 2022-09-21T13:48:57Z

Thanks @oesteban , I tried to run the comand but getting this :
/usr/bin/find: /usr/bin/find: cannot execute binary file
By the way as I mentioned before it works perfectly fine with V21.0.2.
Do you think you can resolve this issue so I can use the latest version soon?

oesteban · 2022-09-21T16:28:10Z

Thanks @oesteban , I tried to run the comand but getting this : /usr/bin/find: /usr/bin/find: cannot execute binary file

That's weird. What do you get from:

docker --rm -it --entrypoint=/bin/bash -v $bids_root_dir:/data:ro nipreps/fmriprep:22.0.0 ls -lah /data/sub-102/ses-01/*

?

By the way as I mentioned before it works perfectly fine with V21.0.2. Do you think you can resolve this issue so I can use the latest version soon?

To resolve the issue, we will first need to understand it so that a solution can be tried and perhaps resolve it. The feedback we have received thus far has not been enough for us to understand it. That's why we are asking further questions.

manzouri · 2022-09-21T20:46:12Z

@oesteban , I get the command to run with minor changes
docker run --rm -it --entrypoint="" -v $bids_root_dir:/data:ro nipreps/fmriprep:22.0.0 find /data/sub-102/ses-01 -type f
This is the output:
/data/sub-102/ses-01/anat/sub-102_ses-01_run-01_T2w.json
/data/sub-102/ses-01/anat/sub-102_ses-01_run-01_T1w.json
/data/sub-102/ses-01/anat/sub-102_ses-01_run-01_T1w.nii.gz
/data/sub-102/ses-01/anat/sub-102_ses-01_run-01_T2w.nii.gz
/data/sub-102/ses-01/func/sub-102_ses-01_task-rs_run-01_bold.nii.gz
/data/sub-102/ses-01/func/sub-102_ses-01_task-rs_run-01_bold.json
/data/sub-102/ses-01/sub-102_ses-01_scans.tsv
/data/sub-102/ses-01/fmap/sub-102_ses-01_dir-AP_epi.json
/data/sub-102/ses-01/fmap/sub-102_ses-01_dir-AP_epi.nii.gz

manzouri · 2022-09-22T12:22:58Z

@oesteban , And after second command
docker run --rm -it --entrypoint="" -v $bids_root_dir:/data:ro nipreps/fmriprep:22.0.0 ls -lah /data/sub-102/ses-01/*
ls: cannot access '/data/sub-102/ses-01/*': No such file or directory

mgxd · 2022-09-22T12:35:49Z

@manzouri would you be able to share this subject / some phantom data with us? Perhaps that would help finally track this down

oesteban · 2022-09-22T12:54:26Z

docker run --rm -it --entrypoint="" -v $bids_root_dir:/data:ro nipreps/fmriprep:22.0.0 ls -lah /data/sub-102/ses-01/*
ls: cannot access '/data/sub-102/ses-01/*': No such file or directory

This is surprising, especially after seeing the output of the first command.

@manzouri would you be able to share this subject / some phantom data with us? Perhaps that would help finally track this down

I agree this seems to be the best way to understand what's going on.

manzouri · 2022-09-22T13:07:35Z

@oesteban , without * at the end I get the following
total 24K
drwxr-xr-x 5 1001 1002 4.0K Sep 12 14:02 .
drwxr-xr-x 3 1001 1002 4.0K Sep 9 10:59 ..
drwxr-xr-x 2 1001 1002 4.0K Sep 12 14:02 anat
drwxr-xr-x 2 1001 1002 4.0K Sep 19 08:29 fmap
drwxr-xr-x 2 1001 1002 4.0K Sep 21 20:51 func
-rw-r--r-- 1 1001 1002 410 Apr 27 20:44 sub-102_ses-01_scans.tsv

manzouri · 2022-09-22T13:13:53Z

@mgxd ,I can share another pilot data acquired from Siemens and another type of field map but same problem if that helps?

├── dataset_description.json
└── sub-pilot1
    └── ses-01
        ├── anat
        │   ├── sub-pilot1_ses-01_run-01_T1w.json
        │   └── sub-pilot1_ses-01_run-01_T1w.nii.gz
        ├── fmap
        │   ├── sub-pilot1_ses-01_run-01_magnitude1.json
        │   ├── sub-pilot1_ses-01_run-01_magnitude1.nii.gz
        │   ├── sub-pilot1_ses-01_run-01_magnitude2.json
        │   ├── sub-pilot1_ses-01_run-01_magnitude2.nii.gz
        │   ├── sub-pilot1_ses-01_run-01_phasediff.json
        │   └── sub-pilot1_ses-01_run-01_phasediff.nii.gz
        └── func
            ├── sub-pilot1_ses-01_task-suppression_run-01_bold.json
            └── sub-pilot1_ses-01_task-suppression_run-01_bold.nii.gz

mgxd · 2022-09-22T18:06:42Z

That seems like a different fieldmap, which won't help in this case - if you can't share or don't have any similar data, we can think of ways to better inform us on what is going on when searching for estimation methods and include it in a future release.

manzouri · 2022-09-26T13:50:46Z

@mgxd , Unfortunately I can't share this data and I don't have any similar from GE scanner but I am having the same problem with the fieldmaps from Siemens and I can easily share that data with you. Would be great if you can work on the available data to resolve this.

manzouri · 2022-09-28T11:20:16Z

@mgxd , @oesteban Please let me know how I can help or better inform you about this issue?

manzouri · 2022-10-03T08:32:49Z

Hi @mgxd , I saw that you were busy with new release just wonder if there has been any progress with this issue also?

oesteban · 2022-10-03T08:34:39Z

@mgxd , Unfortunately I can't share this data and I don't have any similar from GE scanner but I am having the same problem with the fieldmaps from Siemens and I can easily share that data with you. Would be great if you can work on the available data to resolve this.

This would be the fastest way. If you are seeing the same issue on Siemens data and can share those, I'm fairly confident that's going to be the best option.

manzouri · 2022-10-03T09:15:29Z

@oesteban , would you please give me your email to share the link?

oesteban · 2022-10-03T09:33:19Z

nipreps@gmail.com should do ;)

thanks!

manzouri · 2022-10-03T09:53:27Z

thanks @oesteban , just sent it and hope it helps :)

oesteban · 2022-10-03T11:20:55Z

Thanks @manzouri, I can replicate the issue. Will look into this today or tomorrow.

manzouri · 2022-10-05T13:28:02Z

Hi @oesteban , I wonder if you had some time to try my data and if you also have the same problem when running fmriprep from your end?

oesteban · 2022-10-06T07:54:03Z

@effigies @mgxd commenting this line out:

sdcflows/sdcflows/utils/wrangler.py

Line 253 in 6061ed8

"scope": "raw", # Ensure derivatives are not captured

solves the issue.

tested with PyBIDS 0.15.

With that line commented out, then this line below:

sdcflows/sdcflows/utils/wrangler.py

Line 283 in 6061ed8

    
           for fmap in layout.get(suffix=["fieldmap", "phasediff", "phase1"], **base_entities):

manages to identify the fieldmap.

My impression is that there must be some initialization parameter we are setting on the layout in fMRIPrep's side that disallows using the scope argument (then taken as an entity). Not sure this is a bug here or maybe pybids?

manzouri · 2022-10-07T07:46:31Z

@oesteban , Good that you can find the bug, would it be possible for me to run fmriprep on my data soon?

oesteban · 2022-10-07T07:49:38Z

I hope so - now, the next step is implementing a solution. We need to figure out whether we just remove that line, which will depend on the side effects of doing so. It could also be that we found a bug on PyBIDS. We will go as fast as our workload permits.

mgxd · 2022-10-07T15:36:16Z

@manzouri can you share the contents of $bids_root_dir/dataset_description.json

manzouri · 2022-10-07T15:51:09Z

@mgxd
{
"Name": "fMRIPrep - fMRI PREProcessing workflow",
"BIDSVersion": "1.4.0",
"DatasetType": "derivative",
"GeneratedBy": [
{
"Name": "fMRIPrep",
"Version": "20.2.1",
"CodeURL": "https://github.com/nipreps/fmriprep/archive/20.2.1.tar.gz"
}
],
"HowToAcknowledge": "Please cite our paper (https://doi.org/10.1038/s41592-018-0235-4), and include the generated citation boilerplate within the Methods section of the text."
}

effigies · 2022-10-07T15:55:38Z

@manzouri That's the fMRIPrep-generated dataset_description.json. We need the one from your input dataset.

mgxd · 2022-10-07T15:57:59Z

What I think is happening is the fieldmaps are not being found because the root dataset_description.json includes "DatasetType": "derivative". If you remove this line, does that fix your problem?

manzouri · 2022-10-07T16:07:16Z

@mgxd , thanks I think that was the solution now I see these lines when I run:
221007-16:13:10,995 nipype.workflow INFO:
B0 field inhomogeneity map will be estimated with the following 1 estimators: [<EstimatorType.PHASEDIFF: 3>].

manzouri · 2022-10-07T16:17:34Z

What I think is happening is the fieldmaps are not being found because the root dataset_description.json includes "DatasetType": "derivative". If you remove this line, does that fix your problem?

And the previous versions were not sensitive to this maybe it is good to mention it somewhere for other people.

effigies · 2022-10-07T16:19:27Z

Out of curiosity, why does your dataset_description.json indicate that you are passing in fMRIPrep outputs as inputs? In general, I would expect datasets with bad metadata to misbehave.

That said, I agree we should handle this case better.

manzouri · 2022-10-07T16:30:18Z

Out of curiosity, why does your dataset_description.json indicate that you are passing in fMRIPrep outputs as inputs? In general, I would expect datasets with bad metadata to misbehave.

That said, I agree we should handle this case better.

You are right, I copied it by mistake.
Thanks for your support and have a great weekend

manzouri added the bug Something isn't working label Sep 9, 2022

mgxd mentioned this issue Sep 22, 2022

Improve debug verbosity of find_estimators #283

Closed

oesteban transferred this issue from nipreps/fmriprep Oct 6, 2022

effigies closed this as completed May 22, 2023

PEPOLAR field map can't be recognised after upgrading to version 22.0.0 #286

PEPOLAR field map can't be recognised after upgrading to version 22.0.0 #286

Comments

manzouri commented Sep 9, 2022

What happened?

What command did you use?

What version of fMRIPrep are you running?

How are you running fMRIPrep?

Is your data BIDS valid?

Are you reusing any previously computed results?

Please copy and paste any relevant log output.

Additional information / screenshots

mgxd commented Sep 9, 2022

manzouri commented Sep 12, 2022

manzouri commented Sep 15, 2022

mgxd commented Sep 15, 2022

manzouri commented Sep 16, 2022 • edited by effigies Loading

mgxd commented Sep 16, 2022

manzouri commented Sep 19, 2022

mgxd commented Sep 19, 2022

oesteban commented Sep 19, 2022

manzouri commented Sep 21, 2022

oesteban commented Sep 21, 2022

manzouri commented Sep 21, 2022

manzouri commented Sep 22, 2022

mgxd commented Sep 22, 2022

oesteban commented Sep 22, 2022

manzouri commented Sep 22, 2022

manzouri commented Sep 22, 2022 • edited by mgxd Loading

mgxd commented Sep 22, 2022

manzouri commented Sep 26, 2022

manzouri commented Sep 28, 2022

manzouri commented Oct 3, 2022

oesteban commented Oct 3, 2022

manzouri commented Oct 3, 2022

oesteban commented Oct 3, 2022

manzouri commented Oct 3, 2022

oesteban commented Oct 3, 2022

manzouri commented Oct 5, 2022

oesteban commented Oct 6, 2022

manzouri commented Oct 7, 2022

oesteban commented Oct 7, 2022

mgxd commented Oct 7, 2022

manzouri commented Oct 7, 2022

effigies commented Oct 7, 2022

mgxd commented Oct 7, 2022

manzouri commented Oct 7, 2022

manzouri commented Oct 7, 2022

effigies commented Oct 7, 2022

manzouri commented Oct 7, 2022

manzouri commented Sep 16, 2022 •

edited by effigies

Loading

manzouri commented Sep 22, 2022 •

edited by mgxd

Loading