clean: traceback removing local files

[oliver-sanders]

Traceback from running cylc clean on a local flow.

The flow probably hadn't completely finished shutting down yet or NFS was keeping a file open or whatever. This sort of thing is quite common, especially on network filesystems.

At the least we should catch this error and present it more elegantly i.e:

try:
     ...
except OSError as exc:
    # this just changes the presentation of the error
    raise CylcError(exc)

$ cylc clean foo                               
INFO - Cleaning on local filesystem    
Traceback (most recent call last):    
  File "~/cylc/bin/cylc", line 33, in <module>    
    sys.exit(load_entry_point('cylc-flow', 'console_scripts', 'cylc')())    
  File "~/cylc/lib/python3.7/site-packages/click/core.py", line 829, in __call__    
    return self.main(*args, **kwargs)    
  File "~/cylc/lib/python3.7/site-packages/click/core.py", line 782, in main    
    rv = self.invoke(ctx)    
  File "~/cylc/lib/python3.7/site-packages/click/core.py", line 1066, in invoke    
    return ctx.invoke(self.callback, **ctx.params)    
  File "~/cylc/lib/python3.7/site-packages/click/core.py", line 610, in invoke    
    return callback(*args, **kwargs)    
  File "/net~/cylc-flow/cylc/flow/scripts/cylc.py", line 497, in main    
    execute_cmd(command, *cmd_args)    
  File "/net~/cylc-flow/cylc/flow/scripts/cylc.py", line 174, in execute_cmd    
    COMMANDS[cmd].resolve()(*args)    
  File "/net~/cylc-flow/cylc/flow/terminal.py", line 247, in wrapper    
    wrapped_function(*wrapped_args, **wrapped_kwargs)    
  File "/net~/cylc-flow/cylc/flow/scripts/clean.py", line 78, in main    
    init_clean(reg, opts)    
  File "/net~/cylc-flow/cylc/flow/suite_files.py", line 594, in init_clean    
    clean(reg)      
  File "/net~/cylc-flow/cylc/flow/suite_files.py", line 639, in clean    
    remove_dir(run_dir)    
  File "/net~/cylc-flow/cylc/flow/pathutil.py", line 252, in remove_dir    
    rmtree(path)    
  File "~/cylc/lib/python3.7/shutil.py", line 494, in rmtree    
    _rmtree_safe_fd(fd, path, onerror)    
  File "~/cylc/lib/python3.7/shutil.py", line 432, in _rmtree_safe_fd    
    _rmtree_safe_fd(dirfd, fullname, onerror)    
  File "~/cylc/lib/python3.7/shutil.py", line 436, in _rmtree_safe_fd    
    onerror(os.rmdir, fullname, sys.exc_info())    
  File "~/cylc/lib/python3.7/shutil.py", line 434, in _rmtree_safe_fd    
    os.rmdir(entry.name, dir_fd=topfd)    
OSError: [Errno 39] Directory not empty: 'suite'    

Second attempt fine:

$ cylc clean foo    
INFO - No workflow database - will only clean locally    
INFO - Cleaning on local filesystem    

Worth checking this can't crop up in other places too.