Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Problem
Directory searches for links are not paginated at the database level. Instead, the server fetches all of the links from the database at one go (then counts and slices it afterwards). This can potentially create immense server load If there are many links returned from the search query.
Solution
Paginate directory searches for links (both by keyword and email) at the database level.
LIMIT
andOFFSET
for the main queryAn alternative approach considered was to convert the raw queries to Sequelize and use
findAndCountAll
- but Sequelize queries aren't powerful enough to support full-text searches, so for consistency I decided to leave them all as raw queries.For unit tests: I repurposed the
count(*)
mock query result to return{ count: 1 }
. Seems to not be used anywhere else (probably deprecated), so it's safe to reuseTests
@open.gov.sg
emails on staging returning 1.4M links - it still takes quite long (~15s), but the server no longer crashes (good news!). RDS CPU utilisation still increases, but EC2 CPU utilisation does not