Python process can not delete folder that is being observed

[Horiol]

If you try to delete a folder that is being observed by a python process using watchdog and then try to delete it using another python process the folder becomes unavailable until the watchdog process stops.

Steps to reproduce:

1. Create a folder to observe: C:\test_folder
2. Create a python script using watchdog to observe this folder:

import sys
import time
import logging
from watchdog.observers import Observer
from watchdog.events import LoggingEventHandler

if __name__ == "__main__":
    logging.basicConfig(level=logging.INFO,
                        format='%(asctime)s - %(message)s',
                        datefmt='%Y-%m-%d %H:%M:%S')
    path = 'C:\\test_folder'
    event_handler = LoggingEventHandler()
    observer = Observer()
    observer.schedule(event_handler, path, recursive=True)
    observer.start()
    try:
        while True:
            time.sleep(1)
    except KeyboardInterrupt:
        observer.stop()
    observer.join()

3. Open a python shell and try to delete this folder:

import os
for root, dirs, files in os.walk('C:\\test_folder'):
    for f in files:
    os.remove(os.path.join(root, f))
    os.rmdir('C:\\test_folder')

4. Check that the C:\test_folder still appear in file explorer and you are not able to open
5. Stop watchdog process
6. Folder has been deleted

[BoboTiG]

Hello @Horiol,

This is not a bug: Windows does not allow you to delete a file/folder being used elsewhere.

[rrzaripov]

@Horiol Please, can you tell us what version of Python and Windows used by you?
@BoboTiG I'm tried to reproduce this case and got interesting result.

1. If directory empty, os.rmdir(...) was done successfully.
2. Winapi observer thread raise exception when call read_directory_changes on deleted directory:

Exception in thread Thread-7:
Traceback (most recent call last):
  File "c:\python27\Lib\threading.py", line 810, in __bootstrap_inner
    self.run()
  File "C:\git\watchdog\venv\lib\site-packages\watchdog-0.9.0-py2.7.egg\watchdog\observers\api.py", line 145, in run
    self.queue_events(self.timeout)
  File "C:\git\watchdog\venv\lib\site-packages\watchdog-0.9.0-py2.7.egg\watchdog\observers\read_directory_changes.py", line 78, in queue_events
    winapi_events = self._read_events()
  File "C:\git\watchdog\venv\lib\site-packages\watchdog-0.9.0-py2.7.egg\watchdog\observers\read_directory_changes.py", line 75, in _read_events
    return read_events(self._handle, self.watch.is_recursive)
  File "C:\git\watchdog\venv\lib\site-packages\watchdog-0.9.0-py2.7.egg\watchdog\observers\winapi.py", line 345, in read_events
    buf, nbytes = read_directory_changes(handle, recursive)
  File "C:\git\watchdog\venv\lib\site-packages\watchdog-0.9.0-py2.7.egg\watchdog\observers\winapi.py", line 304, in read_directory_changes
    raise e
WindowsError: [Error 5] Access is denied.

3. Test script not handle this exception, so directory handle stay opened, and yes we see deleted directory in explorer until the script still work. When script stopping, handle closed and Windows automatically delete directory.

[Horiol]

Hello @Horiol,

This is not a bug: Windows does not allow you to delete a file/folder being used elsewhere.

@BoboTiG in that case, why I am able to delete the folder using the windows file explorer while the python process is running?

@rrzaripov Thanks for your interest, I'm using the following versions:

• Python 2.7.9
• watchdog 0.8.3
• Windows 2016 Enterprise

[BoboTiG]

@rrzaripov Do you recommand any action to do on the watchdog code?

@Horiol I do not have the answer, I am just aware that file deletion on Windows may be blocked in such cases

[rrzaripov]

@BoboTiG I want to discover this case by next steps:

1. Looking whats doing watchdog in the same situation in Linux
2. Possible need catch exception, stop the observer thread, release handle, and rise something like ObserverDirectoryDeleted exception to user.

[BoboTiG]

All right, great 👍 :)

[rrzaripov]

@BoboTiG under linux, running test application with same scenario no exception is raised and the IN_DELETE_SELF event was obtained. Observed directory removed successfully. After this InotifyBuffer thread stops, but other threads still work, and inotify handler stay opened:

python app.py /tmp/A &
...
root@debian-9:/home/vagrant# ps -e -T | grep 3207
3207  3207 pts/1    00:00:00 python
3207  3209 pts/1    00:00:00 python
3207  3210 pts/1    00:00:00 python
3207  3211 pts/1    00:00:00 python
root@debian-9:/home/vagrant# ls -la /proc/3207/fd
total 0
dr-x------ 2 root root  0 Jun 10 10:05 .
dr-xr-xr-x 9 root root  0 Jun 10 10:04 ..
lrwx------ 1 root root 64 Jun 10 10:05 0 -> /dev/pts/1
lrwx------ 1 root root 64 Jun 10 10:05 1 -> /dev/pts/1
lrwx------ 1 root root 64 Jun 10 10:05 2 -> /dev/pts/1
lr-x------ 1 root root 64 Jun 10 10:05 3 -> anon_inode:inotify
...
rm -rf /tmp/A
...
in-event <InotifyEvent: src_path=b'/tmp/A', wd=1, mask=IN_DELETE_SELF, cookie=0, name=>
...
root@debian-9:/home/vagrant# ps -e -T | grep 3207
3207  3207 pts/1    00:00:00 python
3207  3210 pts/1    00:00:00 python
3207  3211 pts/1    00:00:00 python
...
root@debian-9:/home/vagrant# ls -la /proc/3207/fd
total 0
dr-x------ 2 root root  0 Jun 10 10:05 .
dr-xr-xr-x 9 root root  0 Jun 10 10:04 ..
lrwx------ 1 root root 64 Jun 10 10:05 0 -> /dev/pts/1
lrwx------ 1 root root 64 Jun 10 10:05 1 -> /dev/pts/1
lrwx------ 1 root root 64 Jun 10 10:05 2 -> /dev/pts/1
lr-x------ 1 root root 64 Jun 10 10:05 3 -> anon_inode:inotify

Under Windows on the next (after deleting directory) calling ReadDirectoryChangesW we get unhandled exception WindowsError: [Error 5] Access is denied. I was try to found any solution for detecting that observing directory is deleted and found interesting answer on SO. After read this answer my opinion on needing of the next changes in watchdog:

1. We can call CloseHandle(handle) here before rising exception.
   

   watchdog/src/watchdog/observers/winapi.py

   Lines 302 to 304 in f252554

   	if e.winerror == ERROR_OPERATION_ABORTED:
   	return [], 0
   	raise e

   
   This is solve problem for @Horiol, directory will be removed. But this not resolve other problems.
2. If we can detect that observed directory deleted (via GetFinalPathNameByHandle for example), we must send something like IN_DELETE_SELF event to user.
3. After sending IN_DELETE_SELF we need to stop WindowsApiObserver thread.

Possible something I was overlooked, please give me to know. And yes, I think PR is needed for getting identical behavior under Linux and Windows.