RunHaplotypecaller fails on scratch

[szilvajuhos]

When running on scratch, the bed files are missing. It works when NFX_WORK is on a local directory:

export NXF_TEMP=/scratch
export NXF_LAUNCHBASE=/scratch
export NXF_WORK=/scratch
export NXF_HOME=/castor/project/proj_nobackup/nextflow
nextflow run /castor/project/proj_nobackup/CAW/default/main.nf --step skipPreprocessing --tools HaplotypeCaller --sample Preprocessing/Recalibrated/recalibrated.tsv
...
  ##### ERROR
  ##### ERROR MESSAGE: Could not read file chr14_16133336-16140527.bed because The interval file does not exist.
  ##### ERROR ------------------------------------------------------------------------------------------

changing NXF_WORK to `pwd`/work fixes the issue.

[marcelm]

Please give me a way to reproduce this. I run with NXF_WORK on /scratch and it works for me.

[szilvajuhos]

Hey, created a directory at /proj/b2015110/nobackup/szilva/CAW471 with test data. There is a script and called like:

szilva@m2 /proj/b2015110/nobackup/szilva/CAW471 $ ./runTools.sh chr21_test.tsv HaplotypeCaller

When NXF_WORK is set to a local directory (pwd/work), it works fine. When set to scratch, it fails -
see HC.log

[szilvajuhos]

Also seems the returnMin() part always returns with totalMemory in the configuration. compareTo() expects integers, but the values added are different types. Therefore only a single HaplotypeCaller is launched albeit with a lot of memory.

[marcelm]

Also seems the returnMin() part always returns with totalMemory in the configuration.

Please open a new issue for that. This change was not made by me.

[marcelm]

I believe the problem is that the directory /scratch/ is missing in the Singularity images.

[szilvajuhos]

Indeed! @maxulysse to be blamed in #473

[marcelm]

:-)

[marcelm]

Note that the error message that you gave above:

ERROR MESSAGE: Could not read file chr14_16133336-16140527.bed because The interval file does not exist.

... is different from the one in the HC.log file:

Command error:
  /bin/bash: line 0: cd: /scratch/99/7fe4ac946d6830811079006f7b723a: No such file or directory
  /bin/bash: .command.run.1: No such file or directory

The one in HC.log appears to be caused by the missing /scratch/ directory. The first one is printed by HaplotypeCaller directly. Perhaps they are different issue. Or did you perhaps upgrade Nextflow in between?

[maxulysse]

I'm fixing the /scratch/ issue.
In my opinion the problem is mainly because UPPMAX did not ask us for the right directory to create.
But I do am to blame for the other issue...

[maxulysse]

So creating /scratch in the containers is not enough.
I think this should be finally resolved with adding runOptions = "--bind /scratch" in the singularity scope in the singularity-path.config file.
My guess is that /pica /proj /sw are automatically binded in UPPMAX systems, but not /scratch, so creating the container was not enough to do the trick.
I just made a request to support for having them bind /scratch too, but while waiting for that, runOptions = "--bind /scratch" will do the trick