Feature: Improve Agenda Queue Performance #2545
Conversation
…nished agenda jobs. This should improve performance for the job processor.
|
Do we care about when completed jobs were moved to the history collection? |
| await dbManager.collection(collectionName).find({ nextRunAt: null }).forEach(async (doc) => { | ||
| await dbManager.collection(`${collectionName}History`).insertOne(doc); | ||
| await dbManager.collection(collectionName).deleteOne({ _id: doc._id }); | ||
| }); |
There was a problem hiding this comment.
Won't this load all the documents only to write them back?
I don't know how many documents there will be, so this might not matter, but is there a way to do this entirely on the mongodb side?
There was a problem hiding this comment.
Yes it does. I did look at using a MongoDB aggregation pipeline but I had concerns about concurrency. It should only be a few dozen records at a time at most. Will backlog a ticket to make it more efficient if we find it isn't enough.
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## develop #2545 +/- ##
============================================
+ Coverage 31.58% 52.65% +21.06%
============================================
Files 179 179
Lines 9374 9380 +6
Branches 1354 1354
============================================
+ Hits 2961 4939 +1978
+ Misses 6333 4219 -2114
- Partials 80 222 +142
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. |
When the number of jobs in the Agenda queue database gets large, the job polling query starts to tax MongoDB even with appropriate indices on the collection. This PR adds a scheduled Agenda job that moves all the completed jobs to a new history collection so that they can still be interrogated without affecting polling performance.