singularity parallel pull error of docker images

[tbugfinder]

Bug report

Parallel executions of singularity pull from dockerhub might result in an error.
Root cause seems to be some kind of race condition (or by design?) within singularity. The error is not raised every time.

Expected behavior and actual behavior

In case parallel singularity pull executions fail within nextflow, such pulls should be retried for the failing image. Maybe even pulls shouldn't be parallel.

Steps to reproduce the problem

process test1810 {
    container = 'ubuntu:18.10'
    script:
    """
    cat /etc/os-release
    """
}

process test1804 {
    container = 'ubuntu:18.04'
    script:
    """
    cat /etc/os-release
    """
}
#
#while true ; do date; rm -Rf work/singularity ; ~/nextflow/bin/nextflow run test.nf -with-singularity ; sleep 5 ; rm -Rf work/singularity ; sleep 6 ; done

Program output

Mon Jul  1 20:02:51 CEST 2019
N E X T F L O W  ~  version 19.04.1
Launching `test.nf` [extravagant_sammet] - revision: 3660c710ee
WARN: There's no process matching config selector: trim_galore
[warm up] executor > local
WARN: Singularity cache directory has not been defined -- Remote image will be stored in the path: /tmp/psimg_k/work/singularity
Pulling Singularity image docker://ubuntu:18.10 [cache /tmp/psimg_k/work/singularity/ubuntu-18.10.img]
Pulling Singularity image docker://ubuntu:18.04 [cache /tmp/psimg_k/work/singularity/ubuntu-18.04.img]
ERROR ~ Error executing process > 'test1804'

Caused by:
  Failed to pull singularity image
  command: singularity pull  --name ubuntu-18.04.img docker://ubuntu:18.04 > /dev/null
  status : 255
  message:
    [34mINFO:    Starting build...
    Getting image source signatures
    Skipping fetch of repeat blob sha256:5b7339215d1d5f8e68622d584a224f60339f5bef41dbd74330d081e912f0cddd
    Skipping fetch of repeat blob sha256:14ca88e9f6723ce82bc14b241cda8634f6d19677184691d086662641ab96fe68
    Skipping fetch of repeat blob sha256:a31c3b1caad473a474d574283741f880e37c708cc06ee620d3e93fa602125ee0
    Skipping fetch of repeat blob sha256:b054a26005b7f3b032577f811421fab5ec3b42ce45a4012dfa00cf6ed6191b0f
    Copying config sha256:84c9d0762469176a58d9c54375a1e9a4dcdc0045e3e14e191d97bd12cd5b23a1

     0 B / 2.42 KiB [--------------------------------------------------------------]
     2.42 KiB / 2.42 KiB [======================================================] 0s
    Writing manifest to image destination
    Storing signatures
    INFO:    Creating SIF file...
    FATAL:   Unable to pull docker://ubuntu:18.04: While creating SIF: while creating container: container file creation failed: open ubuntu-18.04.img: no such file or directory



 -- Check '.nextflow.log' file for details

Environment

• Nextflow version: 19.04.01
• Java version: "1.8.0_161"
• Operating system: Linux

Additional context

There's also a more or less stale issue open at Singularity.
https://github.com/sylabs/singularity/issues/3634

FYI: @apeltzer

[pditommaso]

I think this has to be managed by Singularity.

[tbugfinder]

Well, yes and no.
Certainly a parallel pull has to be fixed within singularity. However nextflow should also evaluate return codes. So if a pull fails it could be retried.

[piotr-faba-ardigen]

Also experiencing this at the moment. Retrying after 255 error would be intuitive.

[piotr-faba-ardigen]

Though same error code is given if the image does not exist actually, so I wonder if we have any option of verifying this before attempting a retry.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[tbugfinder]

Still an issue.

[toniher]

In case it can be useful, workaround script for dealing with this when you are facing this problem: https://gist.github.com/toniher/65ccf76a8903bf432435d490b2025fab

[pditommaso]