[Bug]: container's samtools version is deprecated and segfaults please update to supported version

[mp15]

What happened?

I was running wf-human-variation on a dorado BAM I created myself and it seg faulted during the alignment step, this appears to be the samtools command. I reran the samtools piped to minimap2 piped to samtools sort command manually using the currently released versions of both and it did not crash. Given the version of samtools you are using (samtools 1.15.1) is currently unsupported by the samtools team I was wondering if you might be able to update it?

Operating System

ubuntu 20.04

Workflow Execution

Command line

Workflow Execution - EPI2ME Labs Versions

No response

Workflow Execution - CLI Execution Profile

None

Workflow Version

wf-human-variation v1.3.0-gbc26fa0

Relevant log output

# launch command
./nextflow run epi2me-labs/wf-human-variation     -w ${OUTPUT}/workspace     -profile standard     --snp --cnv --sv     --bam OESO_152_sup_4.1.0.pass.map_sort.cram       --ref GCA_000001405.15_GRCh38_full_analysis_set.fna      --sample_name WTSI-OESO_152     --out_dir OESO_152_wf-human-variation

[-        ] process > snp:makeReport                      -
[-        ] process > output_snp                          -
[-        ] process > sv:variantCall:filterBam            -
[-        ] process > sv:variantCall:sniffles2            -
[-        ] process > sv:variantCall:filterCalls          -
[-        ] process > sv:variantCall:sortVCF              -
[-        ] process > sv:variantCall:indexVCF             -
[f5/a1a198] process > sv:runReport:getVersions            [100%] 1 of 1 ✔
[8d/286b9c] process > sv:runReport:getParams              [100%] 1 of 1 ✔
[-        ] process > sv:runReport:report                 -
[-        ] process > output_sv                           -
[-        ] process > cnv:get_genome                      -
[-        ] process > cnv:callCNV                         -
[58/1e4cf7] process > cnv:getVersions                     [100%] 1 of 1 ✔                                                                                                                   [9f/aa6b09] process > cnv:getParams                       [100%] 1 of 1 ✔
[-        ] process > cnv:makeReport                      -
[-        ] process > output_cnv                          -
[-        ] process > configure_jbrowse                   -
[eb/45489e] process > publish_artifact (2)                [100%] 3 of 3 ✔
Error executing process > 'bam_ingress:minimap2_ubam (1)'

Caused by:
  Process `bam_ingress:minimap2_ubam (1)` terminated with an error exit status (139)

Command executed:

  samtools bam2fq -@ 1 -T 1  OESO_152_sup_4.1.0.pass.map_sort.cram | minimap2 -y -t 8 -ax map-ont GCA_000001405.15_GRCh38_full_analysis_set.fna -     | samtools sort -@ 3 --write-index -o WTSI-OESO_152.cram##idx##WTSI-OESO_152.cram.crai -O CRAM --reference GCA_000001405.15_GRCh38_full_analysis_set.fna -

Command exit status:
  139

Command output:
  (empty)

Command error:
  [M::worker_pipeline::19397.201*7.96] mapped 27680 sequences
  [M::worker_pipeline::19460.874*7.96] mapped 28011 sequences
  [M::worker_pipeline::19536.127*7.96] mapped 27764 sequences
  [M::worker_pipeline::19609.846*7.96] mapped 27685 sequences
  [M::worker_pipeline::19772.192*7.96] mapped 27864 sequences
  [M::worker_pipeline::19839.736*7.96] mapped 27855 sequences
  [M::worker_pipeline::19918.228*7.96] mapped 28066 sequences
  [M::worker_pipeline::19984.612*7.96] mapped 27665 sequences
  [M::worker_pipeline::20056.770*7.96] mapped 28533 sequences
  [M::worker_pipeline::21031.685*7.96] mapped 27697 sequences
  [M::worker_pipeline::21101.700*7.96] mapped 28181 sequences
  [M::worker_pipeline::21160.687*7.96] mapped 27842 sequences
  [M::worker_pipeline::21232.494*7.96] mapped 28081 sequences
  [M::worker_pipeline::22493.256*7.96] mapped 25991 sequences
  [M::worker_pipeline::22989.452*7.96] mapped 33507 sequences
  [M::worker_pipeline::23045.543*7.96] mapped 28217 sequences
  [M::worker_pipeline::23260.242*7.96] mapped 27666 sequences
  [M::worker_pipeline::23385.261*7.96] mapped 29260 sequences
  [M::worker_pipeline::23449.023*7.96] mapped 28229 sequences
  [M::worker_pipeline::23882.204*7.97] mapped 27841 sequences
  [M::worker_pipeline::23939.525*7.97] mapped 27786 sequences
  [M::worker_pipeline::24005.267*7.97] mapped 28143 sequences
  [M::worker_pipeline::24079.479*7.97] mapped 28543 sequences
  [M::worker_pipeline::24189.192*7.97] mapped 28921 sequences
  [M::worker_pipeline::24304.057*7.97] mapped 28535 sequences
  [M::worker_pipeline::24363.528*7.97] mapped 28601 sequences
  [M::worker_pipeline::25750.988*7.97] mapped 27967 sequences
  [M::worker_pipeline::25802.596*7.97] mapped 28357 sequences
  [M::worker_pipeline::26009.196*7.97] mapped 28464 sequences
  [M::worker_pipeline::26063.922*7.97] mapped 28458 sequences
  [M::worker_pipeline::26180.402*7.97] mapped 28944 sequences
  [M::worker_pipeline::26326.913*7.97] mapped 28440 sequences
  [M::worker_pipeline::26497.736*7.97] mapped 28636 sequences
  [M::worker_pipeline::26591.002*7.97] mapped 27782 sequences
  [M::worker_pipeline::26678.938*7.97] mapped 27510 sequences
  [M::worker_pipeline::27923.380*7.97] mapped 35169 sequences
  [M::worker_pipeline::29719.556*7.98] mapped 37222 sequences
  [M::worker_pipeline::29940.797*7.97] mapped 31056 sequences
  [M::bam2fq_mainloop] discarded 0 singletons
  [M::bam2fq_mainloop] processed 3794642 reads
  [M::worker_pipeline::32686.042*7.93] mapped 31122 sequences
  [M::worker_pipeline::32711.692*7.92] mapped 13282 sequences
  [M::main] Version: 2.24-r1122
  [M::main] CMD: minimap2 -y -t 8 -ax map-ont GCA_000001405.15_GRCh38_full_analysis_set.fna -
  [M::main] Real time: 32712.538 sec; CPU: 259234.777 sec; Peak RSS: 24.133 GB
  [bam_sort_core] merging from 597 files and 3 in-memory blocks...
  .command.sh: line 2:    29 Done                    samtools bam2fq -@ 1 -T 1 OESO_152_sup_4.1.0.pass.map_sort.cram
          30                       | minimap2 -y -t 8 -ax map-ont GCA_000001405.15_GRCh38_full_analysis_set.fna -
          31 Segmentation fault      (core dumped) | samtools sort -@ 3 --write-index -o WTSI-OESO_152.cram##idx##WTSI-OESO_152.cram.crai -O CRAM --reference GCA_000001405.15_GRCh38_full_analysis_set.fna -

Work dir:
  /lustre/scratch126/casm/team154pc/mp15/output/workspace/7e/959ed906e231a2f3574be0b73ff7cf

Tip: you can try to figure out what's wrong by changing to the process work dir and showing the script file named `.command.sh`


The genome build detected in the BAM is not compatible with this workflow.)

 -- Check script '/root/.nextflow/assets/epi2me-labs/wf-human-variation/./workflows/wf-human-cnv.nf' at line: 22 or see '.nextflow.log' file for more details

[SamStudio8]

Hi @mp15 (fancy meeting you here!), does samtools 1.15.1 still segfault if you run the command manually on your data, or does this only happen when you execute the workflow?

[mp15]

Hey @SamStudio8,

Good to see you again, will you be at LC? Yeah, I can confirm I tried running samtools 1.15.1 both manually inside the docker container and with my own compiled copy outside of it. Both of them dumped core somewhere in the merge phase. I could probably debug it but given that 1.17 works and doesn't crash I figured it was a better option.

[SamStudio8]

Thanks for checking that. I have updated the workflow's base image to use samtools 1.17, but this has yielded a weird event in CI that I want to confirm is a cosmic blip before release, hang tight. I know our team will have some representation at LC but I am not sure who yet... 🤭

[mp15]

Okay let me know, I couldn't figure out how to build my own copy of your base image as https://labs.epi2me.io/wfquickstart/ appears to have bitrotted as your repo's no longer have Dockerfiles?

[mp15]

Further to this I have reason to suspect the reason why samtools was dying is because of the huge AUX mv tag produced by --enable-moves in Dorado. The version of htslib which fixes this going to be the one that incorporates the samtools/htslib#1613 fixes. I will push for a quick release.

[SamStudio8]

Thanks for the additional context. The issue we had in CI appears to be a cosmic blip. I've hit the button to trigger a minor release today.

[SamStudio8]

This should be fixed by 1.5.0, please re-open if you still observe this behaviour. See you at LC 🚀