Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CREST fmriprep: node run error #8

Open
ariveradompenciel opened this issue Mar 27, 2020 · 2 comments
Open

CREST fmriprep: node run error #8

ariveradompenciel opened this issue Mar 27, 2020 · 2 comments
Labels
argon fmriprep issue when running fmriprep singularity

Comments

@ariveradompenciel
Copy link

ariveradompenciel commented Mar 27, 2020

Script/Job File

#!/bin/bash

#$ -pe smp 16
#$ -q UI
#$ -m bea
#$ -M adriana-rivera-dompenciel@uiowa.edu
#$ -o /Shared/vosslabhpc/Projects/CREST/code/fmriprep/out/
#$ -e /Shared/vosslabhpc/Projects/CREST/code/fmriprep/err/
OMP_NUM_THREADS=10
singularity run -H ${HOME}/singularity_home -B /Shared/vosslabhpc:/mnt \
/Shared/vosslabhpc/UniversalSoftware/SingularityContainers/fmriprep-1.2.1.simg \
/mnt/Projects/CREST/ /mnt/Projects/CREST/derivatives \
participant --participant_label BETTER120065 \
-w /nfsscratch/Users/ariveradompenciel/work/CRESTfmriprep \
--write-graph --mem_mb 35000 --omp-nthreads 10 --nthreads 16 --use-aroma \
--output-space template \
--template MNI152NLin2009cAsym \
--fs-license-file /mnt/UniversalSoftware/freesurfer_license.txt

Email Message

Job 5710531 (sub-BETTER120065.job) Complete
User             = ariveradompenciel
Queue            = UI@argon-lc-d7-21.hpc
Host             = argon-lc-d7-21.hpc
Start Time       = 03/15/2020 11:50:41
End Time         = 03/15/2020 11:53:16
User Time        = 00:00:20
System Time      = 00:00:03
Wallclock Time   = 00:02:35
CPU              = 00:00:48
Max vmem         = 712.613M
Exit Status      = 1

Output File

200315-11:50:48,807 nipype.workflow IMPORTANT:
	 
    Running fMRIPREP version 1.2.1:
      * BIDS dataset path: /mnt/Projects/CREST.
      * Participant list: ['BETTER120065'].
      * Run identifier: 20200315-115048_a263a3f1-f6e3-474c-ba04-a4db796e9627.
    
200315-11:50:48,808 nipype.workflow WARNING:
	 Option --debug is deprecated and has no effect
200315-11:51:01,149 nipype.workflow IMPORTANT:
	 Creating bold processing workflow for "/mnt/Projects/CREST/sub-BETTER120065/func/sub-BETTER120065_task-rest_run-1_bold.nii.gz" (0.03 GB / 240 TRs). Memory resampled/largemem=0.13/0.20 GB.
200315-11:51:01,184 nipype.workflow IMPORTANT:
	 No single-band-reference found for sub-BETTER120065_task-rest_run-1_bold.nii.gz
200315-11:51:02,591 nipype.workflow IMPORTANT:
	 Slice-timing correction will be included.
200315-11:51:02,608 nipype.workflow WARNING:
	 SDC: no fieldmaps found or they were ignored (/mnt/Projects/CREST/sub-BETTER120065/func/sub-BETTER120065_task-rest_run-1_bold.nii.gz).
200315-11:51:04,975 nipype.workflow IMPORTANT:
	 Creating bold processing workflow for "/mnt/Projects/CREST/sub-BETTER120065/func/sub-BETTER120065_task-rest_run-2_bold.nii.gz" (0.03 GB / 240 TRs). Memory resampled/largemem=0.13/0.20 GB.
200315-11:51:05,2 nipype.workflow IMPORTANT:
	 No single-band-reference found for sub-BETTER120065_task-rest_run-2_bold.nii.gz
200315-11:51:05,743 nipype.workflow IMPORTANT:
	 Slice-timing correction will be included.
200315-11:51:05,760 nipype.workflow WARNING:
	 SDC: no fieldmaps found or they were ignored (/mnt/Projects/CREST/sub-BETTER120065/func/sub-BETTER120065_task-rest_run-2_bold.nii.gz).
200315-11:51:10,483 nipype.workflow IMPORTANT:
	 Works derived from this fMRIPrep execution should include the following boilerplate:


Results included in this manuscript come from preprocessing
performed using *fMRIPprep* 1.2.1
(@fmriprep1; @fmriprep2; RRID:SCR_016216),
which is based on *Nipype* 1.1.4
(@nipype1; @nipype2; RRID:SCR_002502).

Anatomical data preprocessing

: The T1-weighted (T1w) image was corrected for intensity non-uniformity (INU)
using `N4BiasFieldCorrection` [@n4, ANTs 2.2.0],
and used as T1w-reference throughout the workflow.
The T1w-reference was then skull-stripped using `antsBrainExtraction.sh`
(ANTs 2.2.0), using OASIS as target template.
Brain surfaces were reconstructed using `recon-all` [FreeSurfer 6.0.1,
RRID:SCR_001847, @fs_reconall], and the brain mask estimated
previously was refined with a custom variation of the method to reconcile
ANTs-derived and FreeSurfer-derived segmentations of the cortical
gray-matter of Mindboggle [RRID:SCR_002438, @mindboggle].
Spatial normalization to the ICBM 152 Nonlinear Asymmetrical
template version 2009c [@mni, RRID:SCR_008796] was performed
through nonlinear registration with `antsRegistration`
[ANTs 2.2.0, RRID:SCR_004757, @ants], using
brain-extracted versions of both T1w volume and template.
Brain tissue segmentation of cerebrospinal fluid (CSF),
white-matter (WM) and gray-matter (GM) was performed on
the brain-extracted T1w using `fast` [FSL 5.0.9, RRID:SCR_002823,
@fsl_fast].


Functional data preprocessing

: For each of the 2 BOLD runs found per subject (across all
tasks and sessions), the following preprocessing was performed.
First, a reference volume and its skull-stripped version were generated
using a custom methodology of *fMRIPrep*.
The BOLD reference was then co-registered to the T1w reference using
`bbregister` (FreeSurfer) which implements boundary-based registration [@bbr].
Co-registration was configured with nine degrees of freedom to account
for distortions remaining in the BOLD reference.
Head-motion parameters with respect to the BOLD reference
(transformation matrices, and six corresponding rotation and translation
parameters) are estimated before any spatiotemporal filtering using
`mcflirt` [FSL 5.0.9, @mcflirt].
BOLD runs were slice-time corrected using `3dTshift` from
AFNI 20160207 [@afni, RRID:SCR_005927].
The BOLD time-series (including slice-timing correction when applied)
were resampled onto their original, native space by applying
a single, composite transform to correct for head-motion and
susceptibility distortions.
These resampled BOLD time-series will be referred to as *preprocessed
BOLD in original space*, or just *preprocessed BOLD*.
First, a reference volume and its skull-stripped version were generated
using a custom methodology of *fMRIPrep*.
Automatic removal of motion artifacts using independent component analysis
[ICA-AROMA, @aroma] was performed on the *preprocessed BOLD on MNI space*
time-series after removal of non-steady state volumes and spatial smoothing
with an isotropic, Gaussian kernel of 6mm FWHM (full-width half-maximum).
Corresponding "non-aggresively" denoised runs were produced after such
smoothing.
Additionally, the "aggressive" noise-regressors were collected and placed
in the corresponding confounds file.
The BOLD time-series were resampled to MNI152NLin2009cAsym standard space,
generating a *preprocessed BOLD run in MNI152NLin2009cAsym space*.
First, a reference volume and its skull-stripped version were generated
using a custom methodology of *fMRIPrep*.
Several confounding time-series were calculated based on the
*preprocessed BOLD*: framewise displacement (FD), DVARS and
three region-wise global signals.
FD and DVARS are calculated for each functional run, both using their
implementations in *Nipype* [following the definitions by @power_fd_dvars].
The three global signals are extracted within the CSF, the WM, and
the whole-brain masks.
Additionally, a set of physiological regressors were extracted to
allow for component-based noise correction [*CompCor*, @compcor].
Principal components are estimated after high-pass filtering the
*preprocessed BOLD* time-series (using a discrete cosine filter with
128s cut-off) for the two *CompCor* variants: temporal (tCompCor)
and anatomical (aCompCor).
Six tCompCor components are then calculated from the top 5% variable
voxels within a mask covering the subcortical regions.
This subcortical mask is obtained by heavily eroding the brain mask,
which ensures it does not include cortical GM regions.
For aCompCor, six components are calculated within the intersection of
the aforementioned mask and the union of CSF and WM masks calculated
in T1w space, after their projection to the native space of each
functional run (using the inverse BOLD-to-T1w transformation).
The head-motion estimates calculated in the correction step were also
placed within the corresponding confounds file.
All resamplings can be performed with *a single interpolation
step* by composing all the pertinent transformations (i.e. head-motion
transform matrices, susceptibility distortion correction when available,
and co-registrations to anatomical and template spaces).
Gridded (volumetric) resamplings were performed using `antsApplyTransforms` (ANTs),
configured with Lanczos interpolation to minimize the smoothing
effects of other kernels [@lanczos].
Non-gridded (surface) resamplings were performed using `mri_vol2surf`
(FreeSurfer).


Many internal operations of *fMRIPrep* use
*Nilearn* 0.4.2 [@nilearn, RRID:SCR_001362],
mostly within the functional processing workflow.
For more details of the pipeline, see [the section corresponding
to workflows in *fMRIPrep*'s documentation](https://fmriprep.readthedocs.io/en/latest/workflows.html "FMRIPrep's documentation").


### References


200315-11:52:35,766 nipype.workflow WARNING:
	 Error while checking node hash, forcing re-run. Although this error may not prevent the workflow from running, it could indicate a major problem. Please report a new issue at https://github.com/nipy/nipype/issues adding the following information:

	Node: fmriprep_wf.single_subject_BETTER120065_wf.anat_preproc_wf.surface_recon_wf.fsnative_2_t1_xfm
	Interface: fmriprep.interfaces.freesurfer.PatchedRobustRegister
	Traceback:
Traceback (most recent call last):

  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/pipeline/plugins/base.py", line 338, in _local_hash_check
    cached, updated = self.procs[jobid].is_cached()

  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/pipeline/engine/nodes.py", line 294, in is_cached
    hashed_inputs, hashvalue = self._get_hashval()

  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/pipeline/engine/nodes.py", line 488, in _get_hashval
    self._get_inputs()

  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/pipeline/engine/nodes.py", line 515, in _get_inputs
    results = loadpkl(results_file)

  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/utils/filemanip.py", line 668, in loadpkl
    raise e

  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/utils/filemanip.py", line 659, in loadpkl
    unpkl = pickle.load(pkl_file)

  File "/usr/local/miniconda/lib/python3.6/site-packages/traits/has_traits.py", line 1440, in __setstate__
    self.trait_set( trait_change_notify = trait_change_notify, **state )

  File "/usr/local/miniconda/lib/python3.6/site-packages/traits/has_traits.py", line 1543, in trait_set
    setattr( self, name, value )

  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/interfaces/base/traits_extension.py", line 112, in validate
    self.info_text, value))

traits.trait_errors.TraitError: The trait 'T1' of a ReconAllOutputSpec instance is an existing file name, but the path  '/mnt/Projects/CREST/derivatives/freesurfer/sub-BETTER120065/mri/T1.mgz' does not exist.

200315-11:52:35,818 nipype.workflow WARNING:
	 Error while checking node hash, forcing re-run. Although this error may not prevent the workflow from running, it could indicate a major problem. Please report a new issue at https://github.com/nipy/nipype/issues adding the following information:

	Node: fmriprep_wf.single_subject_BETTER120065_wf.anat_preproc_wf.surface_recon_wf.skull_strip_extern
	Interface: fmriprep.interfaces.freesurfer.FSInjectBrainExtracted
	Traceback:
Traceback (most recent call last):

  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/pipeline/plugins/base.py", line 338, in _local_hash_check
    cached, updated = self.procs[jobid].is_cached()

  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/pipeline/engine/nodes.py", line 294, in is_cached
    hashed_inputs, hashvalue = self._get_hashval()

  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/pipeline/engine/nodes.py", line 488, in _get_hashval
    self._get_inputs()

  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/pipeline/engine/nodes.py", line 515, in _get_inputs
    results = loadpkl(results_file)

  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/utils/filemanip.py", line 668, in loadpkl
    raise e

  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/utils/filemanip.py", line 659, in loadpkl
    unpkl = pickle.load(pkl_file)

  File "/usr/local/miniconda/lib/python3.6/site-packages/traits/has_traits.py", line 1440, in __setstate__
    self.trait_set( trait_change_notify = trait_change_notify, **state )

  File "/usr/local/miniconda/lib/python3.6/site-packages/traits/has_traits.py", line 1543, in trait_set
    setattr( self, name, value )

  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/interfaces/base/traits_extension.py", line 112, in validate
    self.info_text, value))

traits.trait_errors.TraitError: The trait 'T1' of a ReconAllOutputSpec instance is an existing file name, but the path  '/mnt/Projects/CREST/derivatives/freesurfer/sub-BETTER120065/mri/T1.mgz' does not exist.

200315-11:52:38,126 nipype.workflow INFO:
	 [Node] Setting-up "fmriprep_wf.single_subject_BETTER120065_wf.anat_preproc_wf.surface_recon_wf.fsnative_2_t1_xfm" in "/nfsscratch/Users/ariveradompenciel/work/CRESTfmriprep/fmriprep_wf/single_subject_BETTER120065_wf/anat_preproc_wf/surface_recon_wf/fsnative_2_t1_xfm".
200315-11:52:38,370 nipype.workflow INFO:
	 [Node] Setting-up "fmriprep_wf.single_subject_BETTER120065_wf.anat_preproc_wf.surface_recon_wf.skull_strip_extern" in "/nfsscratch/Users/ariveradompenciel/work/CRESTfmriprep/fmriprep_wf/single_subject_BETTER120065_wf/anat_preproc_wf/surface_recon_wf/skull_strip_extern".
200315-11:52:39,528 nipype.workflow ERROR:
	 Node fsnative_2_t1_xfm failed to run on host argon-itf-cf49-34.hpc.
200315-11:52:39,565 nipype.workflow ERROR:
	 Saving crash info to /mnt/Projects/CREST/derivatives/fmriprep/sub-BETTER120065/log/20200315-115048_a263a3f1-f6e3-474c-ba04-a4db796e9627/crash-20200315-115239-ariveradompenciel-fsnative_2_t1_xfm-83c28d90-48d3-45e9-98e3-bc8799a950b9.txt
Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/pipeline/plugins/multiproc.py", line 69, in run_node
    result['result'] = node.run(updatehash=updatehash)
  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/pipeline/engine/nodes.py", line 408, in run
    cached, updated = self.is_cached()
  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/pipeline/engine/nodes.py", line 294, in is_cached
    hashed_inputs, hashvalue = self._get_hashval()
  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/pipeline/engine/nodes.py", line 488, in _get_hashval
    self._get_inputs()
  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/pipeline/engine/nodes.py", line 515, in _get_inputs
    results = loadpkl(results_file)
  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/utils/filemanip.py", line 668, in loadpkl
    raise e
  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/utils/filemanip.py", line 659, in loadpkl
    unpkl = pickle.load(pkl_file)
  File "/usr/local/miniconda/lib/python3.6/site-packages/traits/has_traits.py", line 1440, in __setstate__
    self.trait_set( trait_change_notify = trait_change_notify, **state )
  File "/usr/local/miniconda/lib/python3.6/site-packages/traits/has_traits.py", line 1543, in trait_set
    setattr( self, name, value )
  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/interfaces/base/traits_extension.py", line 112, in validate
    self.info_text, value))
traits.trait_errors.TraitError: The trait 'T1' of a ReconAllOutputSpec instance is an existing file name, but the path  '/mnt/Projects/CREST/derivatives/freesurfer/sub-BETTER120065/mri/T1.mgz' does not exist.

200315-11:52:39,578 nipype.workflow ERROR:
	 Node skull_strip_extern failed to run on host argon-itf-cf49-34.hpc.
200315-11:52:39,578 nipype.workflow ERROR:
	 Saving crash info to /mnt/Projects/CREST/derivatives/fmriprep/sub-BETTER120065/log/20200315-115048_a263a3f1-f6e3-474c-ba04-a4db796e9627/crash-20200315-115239-ariveradompenciel-skull_strip_extern-2c447c52-3748-48cf-adb2-c3882fea3884.txt
Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/pipeline/plugins/multiproc.py", line 69, in run_node
    result['result'] = node.run(updatehash=updatehash)
  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/pipeline/engine/nodes.py", line 408, in run
    cached, updated = self.is_cached()
  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/pipeline/engine/nodes.py", line 294, in is_cached
    hashed_inputs, hashvalue = self._get_hashval()
  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/pipeline/engine/nodes.py", line 488, in _get_hashval
    self._get_inputs()
  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/pipeline/engine/nodes.py", line 515, in _get_inputs
    results = loadpkl(results_file)
  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/utils/filemanip.py", line 668, in loadpkl
    raise e
  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/utils/filemanip.py", line 659, in loadpkl
    unpkl = pickle.load(pkl_file)
  File "/usr/local/miniconda/lib/python3.6/site-packages/traits/has_traits.py", line 1440, in __setstate__
    self.trait_set( trait_change_notify = trait_change_notify, **state )
  File "/usr/local/miniconda/lib/python3.6/site-packages/traits/has_traits.py", line 1543, in trait_set
    setattr( self, name, value )
  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/interfaces/base/traits_extension.py", line 112, in validate
    self.info_text, value))
traits.trait_errors.TraitError: The trait 'T1' of a ReconAllOutputSpec instance is an existing file name, but the path  '/mnt/Projects/CREST/derivatives/freesurfer/sub-BETTER120065/mri/T1.mgz' does not exist.

200315-11:53:07,925 nipype.workflow ERROR:
	 could not run node: fmriprep_wf.single_subject_BETTER120065_wf.anat_preproc_wf.surface_recon_wf.fsnative_2_t1_xfm
200315-11:53:07,928 nipype.workflow ERROR:
	 could not run node: fmriprep_wf.single_subject_BETTER120065_wf.anat_preproc_wf.surface_recon_wf.skull_strip_extern

Error (stderr file)

/usr/local/miniconda/lib/python3.6/site-packages/bids/grabbids/__init__.py:6: FutureWarning: grabbids has been renamed to layout in version 0.6.5, and will be removed in version 0.8
  warnings.warn("grabbids has been renamed to layout in version 0.6.5, and will be removed in version 0.8", FutureWarning)
[WARNING] This document format requires a nonempty <title> element.
  Please specify either 'title' or 'pagetitle' in the metadata.
  Falling back to 'CITATION'
/usr/local/miniconda/lib/python3.6/site-packages/bids/grabbids/__init__.py:6: FutureWarning: grabbids has been renamed to layout in version 0.6.5, and will be removed in version 0.8
  warnings.warn("grabbids has been renamed to layout in version 0.6.5, and will be removed in version 0.8", FutureWarning)
/usr/local/miniconda/lib/python3.6/site-packages/bids/layout/bids_layout.py:116: UserWarning: 'dataset_description.json' file is missing from project root. You may want to set the root path to a valid BIDS project.
  warnings.warn("'dataset_description.json' file is missing from "
Traceback (most recent call last):
  File "/usr/local/miniconda/bin/fmriprep", line 11, in <module>
    sys.exit(main())
  File "/usr/local/miniconda/lib/python3.6/site-packages/fmriprep/cli/run.py", line 303, in main
    fmriprep_wf.write_graph(graph2use="colored", format='svg', simple_form=True)
  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/pipeline/engine/workflows.py", line 430, in write_graph
    outfname = format_dot(dotfilename, format=format)
  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/pipeline/engine/utils.py", line 1390, in format_dot
    formatted_dot, _ = _run_dot(dotfilename, format_ext=format)
  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/pipeline/engine/utils.py", line 1407, in _run_dot
    resource_monitor=False).run()
  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/interfaces/base/core.py", line 522, in run
    runtime = self._run_interface(runtime)
  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/interfaces/base/core.py", line 1038, in _run_interface
    self.raise_exception(runtime)
  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/interfaces/base/core.py", line 975, in raise_exception
    ).format(**runtime.dictcopy()))
RuntimeError: Command:
dot -Tsvg -o"/nfsscratch/Users/ariveradompenciel/work/CRESTfmriprep/fmriprep_wf/graph.svg" "/nfsscratch/Users/ariveradompenciel/work/CRESTfmriprep/fmriprep_wf/graph.dot"
Standard output:

Standard error:
Warning: syntax ambiguity - badly delimited number '120065_' in line 2179 of /nfsscratch/Users/ariveradompenciel/work/CRESTfmriprep/fmriprep_wf/graph.dot splits into two tokens
Error: /nfsscratch/Users/ariveradompenciel/work/CRESTfmriprep/fmriprep_wf/graph.dot: syntax error in line 2179 near '120065'
Return code: 1

Potential Solution

deleted the following directories:
/nfsscratch/Users/ariveradompenciel/work/CRESTfmriprep/fmriprep_wf/single_subject_BETTER120065_wf/anat_preproc_wf/surface_recon_wf/segs_to_native_aseg

/Shared/vosslabhpc/Projects/CREST/derivatives/freesurfer/sub-BETTER120065

@ariveradompenciel
Copy link
Author

Error and exit status of 1 remain.

@ariveradompenciel ariveradompenciel added argon fmriprep issue when running fmriprep singularity labels Mar 27, 2020
@jdkent
Copy link
Member

jdkent commented Mar 27, 2020

I think you need to delete /nfsscratch/Users/ariveradompenciel/work/CRESTfmriprep/fmriprep_wf/single_subject_BETTER120065_wf/anat_preproc_wf/ to re-run

@ariveradompenciel ariveradompenciel changed the title CREST fmriprep: node run error (recon-all?) CREST fmriprep: node run error Mar 27, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
argon fmriprep issue when running fmriprep singularity
Projects
None yet
Development

No branches or pull requests

2 participants