Backups directory not being deleted

[sjdurfey]

Recently I noticed on some exhibitor nodes, the backups directory, /var/exhibitor/backups, isn't deleting directories after the max save time (which is set to 1 hour). There are directories dating back 18 months, with the vast majority being empty (99.99% empty). This is causing some space issues on the nodes. I was planning on just deleting the empty directories, but wanted to reach out to see if anyone knew why the directories weren't being deleted.

I did look in the logs, but haven't seen anything to indicate a failure occurred while trying to delete them. Every log statement (in the current and rollover logs) is just a slight variation of this:

@400000005a1cffb02ac5c694 INFO  com.netflix.exhibitor.core.activity.ActivityLog  Cleanup task completed [pool-2-thread-161253]

[bbaugher]

I was looking into this further and it seems like this is a bug. The backups are creating files following the path exhibitor backup dir / zookeeper data log file name / zookeeper data log file modified time. The FileSystemProvider however only deletes the file and not the parent directory.

I'm not sure if perhaps zookeeper used to maintain the same file and name so deleting the directory wasn't needed but this at least isn't true now.