Launch command malformed under certain conditions in 24.12.0-edge

[scwatts]

Bug report

Expected behavior and actual behavior

When running 24.12.0-edge but not 24.10.4 with Docker the launch command can be malformed causing the corresponding process to fail. The changed behaviour is likely to be related to 8342889.

Steps to reproduce the problem

Note

This behaviour appears after merging the nf-core template 3.1.2 into oncoanalyser dev and running with 24.12.0-edge

Click to show

Create local NF config

cat <<EOF > local.config
process {

  resourceLimits = [
    cpus: 2,
    memory: 15.GB,
    time: 1.h
  ]
EOF

Run NF 24.10.4

Completes successfully

Run

NXF_DEBUG=1 NXF_VER=24.10.4 nextflow run nf-core/oncoanalyser \
  -revision nf-core-template-merge-3.1.2 \
  -config local.config \
  -stub \
  -profile test_stub,docker \
  --processes_manual \
  --processes_include markdups \
  --outdir output/

Command stderr (selected)

+ nxf_stage
+ true
+ rm -f unmap_regions.38.tsv
+ rm -f subject_a.tumor.dna.bam
+ rm -f GCA_000001405.15_GRCh38_no_alt_analysis_set.fna
+ rm -f GCA_000001405.15_GRCh38_no_alt_analysis_set.fna.fai
+ rm -f GCA_000001405.15_GRCh38_no_alt_analysis_set.fna.dict
+ rm -f subject_a.tumor.dna.bam.bai
+ ln -s /home/ssm-user/working/reference_data/hmftools/5.34_38--2/dna_pipeline/common/unmap_regions.38.tsv unmap_regions.38.tsv
+ ln -s /home/ssm-user/working/oncoanalyser_24.12.0-edge_debug/6_24.12.0-edge_debug/temp/subject_a.tumor.dna.bam subject_a.tumor.dna.bam
+ ln -s /home/ssm-user/working/reference_data/genomes/GRCh38_hmf/GCA_000001405.15_GRCh38_no_alt_analysis_set.fna GCA_000001405.15_GRCh38_no_alt_analysis_set.fna
+ ln -s /home/ssm-user/working/reference_data/genomes/GRCh38_hmf/samtools_index/1.16/GCA_000001405.15_GRCh38_no_alt_analysis_set.fna.fai GCA_000001405.15_GRCh38_no_alt_analysis_set.fna.fai
+ ln -s /home/ssm-user/working/reference_data/genomes/GRCh38_hmf/samtools_index/1.16/GCA_000001405.15_GRCh38_no_alt_analysis_set.fna.dict GCA_000001405.15_GRCh38_no_alt_analysis_set.fna.dict
+ ln -s /home/ssm-user/working/oncoanalyser_24.12.0-edge_debug/6_24.12.0-edge_debug/temp/subject_a.tumor.dna.bam.bai subject_a.tumor.dna.bam.bai
+ set +e
+ pid=11162
+ wait 11162
+ set -o pipefail
+ tee .command.err
+ tee .command.out
+ nxf_launch
++ id -u
++ id -g
++ nxf_container_env
++ cat
+ docker run -i --cpu-shares 2048 --memory 15360m -e NXF_TASK_WORKDIR -e NXF_DEBUG=1 -v /home/ssm-user/working:/home/ssm-user/working -w /home/ssm-user/working/oncoanalyser_24.12.0-edge_debug/6_24.12.0-edge_debug/work/9d/4efd9c4a673d8251b89d3d2cf0cfc7 -u 1001:1001 --name nxf-rMFI8ggc0fUy9nebI3LwHMnx quay.io/biocontainers/hmftools-mark-dups:1.1.7--hdfd78af_0 /bin/bash -c 'eval export PYTHONNOUSERSITE="1"
export R_PROFILE_USER="/.Rprofile"
export R_ENVIRON_USER="/.Renviron"
export JULIA_DEPOT_PATH="/usr/local/share/julia"; /bin/bash /home/ssm-user/working/oncoanalyser_24.12.0-edge_debug/6_24.12.0-edge_debug/work/9d/4efd9c4a673d8251b89d3d2cf0cfc7/.command.run nxf_trace'

Run NF 24.14.0-edge

Fails due to malformed launch command

Run

NXF_DEBUG=1 NXF_VER=24.12.0-edge nextflow run nf-core/oncoanalyser \
  -revision nf-core-template-merge-3.1.2 \
  -config local.config \
  -stub \
  -profile test_stub,docker \
  --processes_manual \
  --processes_include markdups \
  --outdir output/

Command stderr (selected)

  + nxf_stage
  + true
  + rm -f unmap_regions.38.tsv
  + rm -f subject_a.tumor.dna.bam
  + rm -f GCA_000001405.15_GRCh38_no_alt_analysis_set.fna
  + rm -f GCA_000001405.15_GRCh38_no_alt_analysis_set.fna.fai
  + rm -f GCA_000001405.15_GRCh38_no_alt_analysis_set.fna.dict
  + rm -f subject_a.tumor.dna.bam.bai
  + ln -s /home/ssm-user/working/reference_data/hmftools/5.34_38--2/dna_pipeline/common/unmap_regions.38.tsv unmap_regions.38.tsv
  + ln -s /home/ssm-user/working/oncoanalyser_24.12.0-edge_debug/6_24.12.0-edge_debug/temp/subject_a.tumor.dna.bam subject_a.tumor.dna.bam
  + ln -s /home/ssm-user/working/reference_data/genomes/GRCh38_hmf/GCA_000001405.15_GRCh38_no_alt_analysis_set.fna GCA_000001405.15_GRCh38_no_alt_analysis_set.fna
  + ln -s /home/ssm-user/working/reference_data/genomes/GRCh38_hmf/samtools_index/1.16/GCA_000001405.15_GRCh38_no_alt_analysis_set.fna.fai GCA_000001405.15_GRCh38_no_alt_analysis_set.fna.fai
  + ln -s /home/ssm-user/working/reference_data/genomes/GRCh38_hmf/samtools_index/1.16/GCA_000001405.15_GRCh38_no_alt_analysis_set.fna.dict GCA_000001405.15_GRCh38_no_alt_analysis_set.fna.dict
  + ln -s /home/ssm-user/working/oncoanalyser_24.12.0-edge_debug/6_24.12.0-edge_debug/temp/subject_a.tumor.dna.bam.bai subject_a.tumor.dna.bam.bai
  + set +e
  + pid=10500
  + wait 10500
  + set -o pipefail
  + tee .command.err
  + nxf_launch
  + tee .command.out
  ++ id -u
  ++ id -g
  ++ nxf_container_env
  ++ cat
  + docker run -i --cpu-shares 2048 --memory 15360m -e NXF_TASK_WORKDIR -e NXF_DEBUG=1 -v /home/ssm-user/working:/home/ssm-user/working -w /home/ssm-user/working/oncoanalyser_24.12.0-edge_debug/6_24.12.0-edge_debug/work/c8/739dab3b54dcfefeb55bc7a75136a6 -u 1001:1001--name nxf-UxwLWKZt3hEMf7hyoPzn2AnF quay.io/biocontainers/hmftools-mark-dups:1.1.7--hdfd78af_0 /bin/bash -c 'eval export PYTHONNOUSERSITE="1"
  export R_PROFILE_USER="/.Rprofile"
  export R_ENVIRON_USER="/.Renviron"
  export JULIA_DEPOT_PATH="/usr/local/share/julia"; bash

      set -e # Exit if a tool returns a non-zero status/exit code
      set -u # Treat unset variables and parameters as an error
      set -o pipefail # Returns the status of the last command to exit with a non-zero status or zero if all successfully execute
      set -C # No clobber - prevent output redirection from overwriting files.
       .command.run nxf_trace'
  /bin/bash: line 10: .command.run: Permission denied
  + nxf_main_ret=126
  + nxf_unstage
  + [[ 126 == 0 ]]
  + nxf_unstage_controls
  + true
  + on_exit
  + local last_err=0
  + local exit_status=126
  + [[ 126 -eq 0 ]]
  + [[ 126 -eq 0 ]]
  + printf -- 126
  + set +u
  + docker rm nxf-UxwLWKZt3hEMf7hyoPzn2AnF
  + exit 126

Program output

Attached: oncoanalyser_24.12.0-edge_bug.nextflow.log

Environment

• Nextflow version: 24.12.0-edge
• Java version: openjdk 21-internal 2023-09-19
• Operating system: Linux 5.10 x86_64
• Bash version: 4.2.46(2)-release

Additional context

N/a

[mashehu]

We see the same in nf-core CI, e.g. https://github.com/nf-core/tools/actions/runs/12881683094/job/35915077759?pr=3411 and https://github.com/nf-core/funcscan/actions/runs/12882715139/job/35915516502?pr=431

[ewels]

This was a problem stemming from the nf-core template, see nf-core/tools#3416 for a fix.

#5689 will strengthen the validation of process.shell directives to give a clearer exit and error message.

[scwatts]

Thanks all for resolving so quickly!