Error: no such file or directory during spatial normalisation

[jokedurnez]

Spatial normalization seems to have failed on 2 subjects. Got the following error trace after a few minutes.

170210-02:48:30,602 workflow ERROR:
	 [u'Node MNIApplyMask failed to run on host sh-1-22.local.']
170210-02:48:30,603 workflow INFO:
	 Saving crash info to /scratch/PI/russpold/data/psychosis/05_mriqc/out/logs/crash-20170210-024830-jdurnez-MNIApplyMask-c23067f9-b734-49c8-9c90-b4a1e566cc43.pklz
170210-02:48:30,603 workflow INFO:
	 Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python2.7/site-packages/nipype/pipeline/plugins/multiproc.py", line 52, in run_node
    result['result'] = node.run(updatehash=updatehash)
  File "/usr/local/miniconda/lib/python2.7/site-packages/nipype/pipeline/engine/nodes.py", line 370, in run
    shutil.move(hashfile_unfinished, hashfile)
  File "/usr/local/miniconda/lib/python2.7/shutil.py", line 302, in move
    copy2(src, real_dst)
  File "/usr/local/miniconda/lib/python2.7/shutil.py", line 130, in copy2
    copyfile(src, dst)
  File "/usr/local/miniconda/lib/python2.7/shutil.py", line 82, in copyfile
    with open(src, 'rb') as fsrc:
IOError: [Errno 2] No such file or directory: u'/scratch/PI/russpold/data/psychosis/05_mriqc/work/workflow_enumerator/funcMRIQC/SpatialNormalization/MNIApplyMask/_0x8ca483cd43f00defaa8ca19a9e4c0610_unfinished.json'

Job continued for 3.5 more hours, but was then stopped with this:

170210-06:17:18,104 workflow ERROR:
	 could not run node: workflow_enumerator.funcMRIQC.SpatialNormalization.MNIApplyMask
170210-06:17:18,105 workflow INFO:
	 crashfile: /scratch/PI/russpold/data/psychosis/05_mriqc/out/logs/crash-20170210-024830-jdurnez-MNIApplyMask-c23067f9-b734-49c8-9c90-b4a1e566cc43.pklz
170210-06:17:18,105 workflow INFO:

Full logs here: mriqc_out.txt,
and here: stderr.txt

[chrisgorgo]

Potential fix proposed in nipy/nipype#1750

[oesteban]

Hi @jokedurnez , could you confirm if this was happening using --n_procs 1?

[oesteban]

I think we got this: nipy/nipype#1750 (comment)

[chrisgorgo]

Current workaround for @jokedurnez and @IanEisenberg is to always specify a different working directory (-w) each time you call mriqc (especially when running multiple subjects in parallel on a cluster).

[jokedurnez]

I hadn't specified --n_procs

Thanks for the workaround, will try that.