FileExistsError - race condition?

[soungalo]

I am running OF v2.4.0 and got the following error stack:

Traceback (most recent call last):
  File "/groups/itay_mayrose/nosnap/liorglic/Panoramic/test/de_novo_run/.snakemake/conda/b81d54f7/lib/python3.7/multiprocessing/pr
ocess.py", line 297, in _bootstrap
    self.run()
  File "/groups/itay_mayrose/nosnap/liorglic/Panoramic/test/de_novo_run/.snakemake/conda/b81d54f7/lib/python3.7/multiprocessing/pr
ocess.py", line 99, in run
    self._target(*self._args, **self._kwargs)
  File "/groups/itay_mayrose/nosnap/liorglic/Panoramic/test/de_novo_run/.snakemake/conda/b81d54f7/bin/scripts_of/__main__.py", lin
e 520, in Worker_ProcessBlastHits
    WaterfallMethod.ProcessBlastHits(*args, qDoubleBlast=qDoubleBlast)
  File "/groups/itay_mayrose/nosnap/liorglic/Panoramic/test/de_novo_run/.snakemake/conda/b81d54f7/bin/scripts_of/__main__.py", line 510, in ProcessBlastHits
    matrices.DumpMatrixArray("B", Bi, iSpecies)
  File "/groups/itay_mayrose/nosnap/liorglic/Panoramic/test/de_novo_run/.snakemake/conda/b81d54f7/bin/scripts_of/matrices.py", line 43, in DumpMatrixArray
    DumpMatrix(name, m, iSpecies, jSpecies)
  File "/groups/itay_mayrose/nosnap/liorglic/Panoramic/test/de_novo_run/.snakemake/conda/b81d54f7/bin/scripts_of/matrices.py", line 38, in DumpMatrix
    with open(files.FileHandler.GetPickleDir() + "%s%d_%d.pic" % (name, iSpecies, jSpecies), 'wb') as picFile:
  File "/groups/itay_mayrose/nosnap/liorglic/Panoramic/test/de_novo_run/.snakemake/conda/b81d54f7/bin/scripts_of/files.py", line 216, in GetPickleDir
    if not os.path.exists(d): os.mkdir(d)
FileExistsError: [Errno 17] File exists: '/groups/itay_mayrose/nosnap/liorglic/Panoramic/test/de_novo_run/RESULT/all_samples/orthofinder/OrthoFinder/Results_orthofinder/WorkingDirectory/pickle/'

From what I read, this can happen when the directory is created after the if statement but before the os.mkdir() call. The recommended way to do this safely in Python > 3.5 is:

from pathlib import Path
Path(d).mkdir(parents=True, exist_ok=True)

Not sure this is indeed the issue, but can't think of anything else.

[davidemms]

Hi

Thanks, yes I think you're probably right about the cause. I will try and put a fix soon for this (for the time being I would like to continue to maintain python 2.7 support as well so it'll take a little more work). Is it a one off occurrence for you or is it always happening?

All the best
David

[davidemms]

Hi Lior

Thanks for reporting this, I've eliminated the race condition so as stop this occurring in the first place.

All the best
David