shutil.rmtree fails in M2 mac

[naga-onu]

How to reproduce

Run the following commands.(test.fa is attatched below.)

curl -SOL https://busco-data.ezlab.org/v4/data/lineages/insecta_odb10.2020-09-10.tar.gz 
tar xf insecta_odb10.2020-09-10.tar.gz

cwltool bug.cwl --busco_db insecta_odb10 --input_fasta test.fa

Expected Behavior

It finishes without errors.

Actual Behavior

It fails with the following messages.
It seems that BUSCO successfully finished but cwltool fails cleaning up the temporal directory.

cwltool tools/bug.cwl --busco_db insecta_odb10 --input_fasta test.fa
...
INFO [job bug.cwl] Max memory used: 2369MiB
ERROR Exception while running job
Traceback (most recent call last):
  File "/Users/ohnuki/miniconda3/envs/cwl/lib/python3.11/site-packages/cwltool/job.py", line 373, in _execute
    relink_initialworkdir(
  File "/Users/ohnuki/miniconda3/envs/cwl/lib/python3.11/site-packages/cwltool/job.py", line 103, in relink_initialworkdir
    shutil.rmtree(host_outdir_tgt)
  File "/Users/ohnuki/miniconda3/envs/cwl/lib/python3.11/shutil.py", line 738, in rmtree
    onerror(os.rmdir, path, sys.exc_info())
  File "/Users/ohnuki/miniconda3/envs/cwl/lib/python3.11/shutil.py", line 736, in rmtree
    os.rmdir(path, dir_fd=dir_fd)
PermissionError: [Errno 13] Permission denied: '/private/tmp/docker_tmp379y2o69/insecta_odb10'
WARNING [job bug.cwl] completed permanentFail
{}WARNING Final process status is permanentFail

Workflow Code

#!/usr/bin/env cwl-runner
cwlVersion: v1.2
class: CommandLineTool

baseCommand: [ busco]
arguments: [ 
    -m, genome,
    -o, tmp,
    --offline,
    -i, $(inputs.input_fasta),
    -l, $(inputs.busco_db)
]

requirements:
    InitialWorkDirRequirement:
        listing:
            - entry: $(inputs.busco_db)
              writable: true
    InplaceUpdateRequirement:
        inplaceUpdate: true
    DockerRequirement:
        dockerPull: "quay.io/biocontainers/busco:5.5.0--pyhdfd78af_0"
inputs:
    input_fasta:
        type: File
    busco_db:
        type: Directory

outputs: []

Full Traceback

I confirmed --debug shows the same messages.

cwltool tools/bug.cwl --busco_db insecta_odb10 --input_fasta test.fa
...
INFO [job bug.cwl] Max memory used: 2369MiB
ERROR Exception while running job
Traceback (most recent call last):
  File "/Users/ohnuki/miniconda3/envs/cwl/lib/python3.11/site-packages/cwltool/job.py", line 373, in _execute
    relink_initialworkdir(
  File "/Users/ohnuki/miniconda3/envs/cwl/lib/python3.11/site-packages/cwltool/job.py", line 103, in relink_initialworkdir
    shutil.rmtree(host_outdir_tgt)
  File "/Users/ohnuki/miniconda3/envs/cwl/lib/python3.11/shutil.py", line 738, in rmtree
    onerror(os.rmdir, path, sys.exc_info())
  File "/Users/ohnuki/miniconda3/envs/cwl/lib/python3.11/shutil.py", line 736, in rmtree
    os.rmdir(path, dir_fd=dir_fd)
PermissionError: [Errno 13] Permission denied: '/private/tmp/docker_tmp379y2o69/insecta_odb10'
WARNING [job bug.cwl] completed permanentFail
{}WARNING Final process status is permanentFail

Your Environment

• cwltool version:
  Check using cwltool --version

cwltool --version
/Users/ohnuki/miniconda3/envs/cwl/bin/cwltool 3.1.20230906142556

I confirmed it happens in M2 max Mac(macOS 13.6) but does not happen in M1 Mac(macOS 13.6 and cwltool 3.1.20220830195442) and Ubuntu-22.04(intel and cwltool 3.1.20220913185150).

test.fa.zip

[tom-tan]

We found this issue during BioHackthon JP.

[mr-c]

Interesting! What was the docker version?

[naga-onu]

The docker version is 4.21.1.