Periodically reconcile the list of metadata files #2085
Conversation
| fileSyncFuture = scheduler.scheduleAtFixedRate(() -> { | ||
| runLock.lock(); | ||
| try { | ||
| readInitialFiles(); |
There was a problem hiding this comment.
checking - do you know how long this process usually takes, and is there a concern that it'll hold the lock long enough to cause issues for the S3 uploads?
There was a problem hiding this comment.
This whole process is async/background, nothing hits it live or like a service, and it only holds the lock for a second or two
| @@ -209,6 +221,10 @@ public void shutdown() { | |||
| runLock.unlock(); | |||
| } | |||
|
|
|||
| if (fileSyncFuture != null) { | |||
| fileSyncFuture.cancel(true); | |||
There was a problem hiding this comment.
why allow interrupts here but not for the upload future?
There was a problem hiding this comment.
Good Q, I don't remember why I put that there... Will take a second look at it
There was a problem hiding this comment.
Ah right, this one is just checking for new files, interrupting in the middle won't actually interrupt an upload. The uploaders themselves we want to give a better chance to finish
|
🚢 |
This is the simplest fix I could think of for our longer-term uploader leak issue. The most likely case is that we are occasionally missing a file update or not handling ti properly, since we always end up with more files in the filesystem than in memory. So, this periodically rescans the directory just like on startup. For the most part it will be a no-op, but will also serve to clean up files we may have missed so that things don't grow out of control