FileInfoPollerServer lookback exceeds limit leads to reprocessing files
#855
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…okback offset When the cleaning process is triggered, it will get the timestamp of the 100th oldest file that has been processed. If that time is greater than the lookback offset, we will only remove files older than the lookback. Otherwise, we will remove any file older than the 100th entry. - add cache clean logging - break out getting FileInfo from s3 with `FileInfoPollerStore` trait
michaeldjeffrey
requested review from
maplant,
andymck,
bbalser and
jeffgrunewald
andymck
reviewed
Aug 21, 2024
`FileInfoPollerServer` has a test that uses a database now.
andymck
approved these changes
Aug 21, 2024
bbalser
approved these changes
Aug 21, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Some Background
A
FileInfoPollerServerkeeps track of processed files in a db table. To keep from growing forever, this table is cleaned when aFileInfoPollerServeris started, and every 12 hours afterwards, to the most recent 100 processed files. For most cases, this is well and good.To fetch files for processing, the timestamp of the latest processed file and an
offsetDuration is used to determine how far back to look.The Situation
A burst of files arrived on devnet exceeding the 100 row limit. And they were processed. It was good.
For sake of this tale, let's say 150 files arrived in 10 minutes. And the
offsetprovided was 1 hour.Later, the cleanup job came around (or when the service was restarted) and removed the 50 that were over the limit.
The next time the
FileInfoPollerServerwas asked for some files to process, it took the latest timestamp, added theoffsetand requested those files from s3. It pulled down everything from the last hour.However, the first 50 files it pulled down that were already processed were no longer in the database because of the cleanup job. So they were dutifully re-processed. It was bad.
A Solution
When the cleanup job runs, either on interval or startup, we find the timestamp of the row at the allowed limit (in this case, still 100).
If the
offsetprovided is earlier than the 100th item (t100), the cleanup job will remove rows older than theoffset.If
t100is older, rows older than it are removed.In this way, files that may be fetched for processing should always exist in the database until there is no longer a chance they will be fetched. And we keep around some history for debugging.