review use of rmtree

[oliver-sanders]

Nasty bug with shutil.rmtree spotted by @datamel whilst reviewing #3654.

Python's shutil.rmtree (like most things) does not work reliably on NFS filesystems. Other solutions may be affected by the same underlying issue.

The safest way I can think of handling it would is to be to give NFS a brief window to think about what it's done before taking it seriously and allowing traceback to propagate.

I'm not aware that this is causing issues in Cylc at the moment, but it is a potential bug, just yearning for the chance to surface when you really don't want it to so tagging this against bug?.

See this comment which includes examples of what other projects have done to deal with the NFS menace.

• https://github.com/hashdist/hashdist/pull/116/files#diff-f8e43180100ba3086d46440b72e3e8f3
• davidemms/OrthoFinder@3d56a6e#diff-4b6271e9a0c35f1d832125bcadf25034

[MetRonnie]

Superseded by #4133? Although it doesn't suggest retrying the deletion after a delay like here

[MetRonnie]

Closed by #5359