Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore: paginate directory search #2061

Merged
merged 1 commit into from
Nov 2, 2022
Merged

Conversation

halfwhole
Copy link
Collaborator

@halfwhole halfwhole commented Nov 1, 2022

Problem

Directory searches for links are not paginated at the database level. Instead, the server fetches all of the links from the database at one go (then counts and slices it afterwards). This can potentially create immense server load If there are many links returned from the search query.

Solution

Paginate directory searches for links (both by keyword and email) at the database level.

  • Use LIMIT and OFFSET for the main query
  • Add a separate query to fetch the total count of links

An alternative approach considered was to convert the raw queries to Sequelize and use findAndCountAll - but Sequelize queries aren't powerful enough to support full-text searches, so for consistency I decided to leave them all as raw queries.

For unit tests: I repurposed the count(*) mock query result to return { count: 1 }. Seems to not be used anywhere else (probably deprecated), so it's safe to reuse

Tests

  • Tested directory search by email for @open.gov.sg emails on staging returning 1.4M links - it still takes quite long (~15s), but the server no longer crashes (good news!). RDS CPU utilisation still increases, but EC2 CPU utilisation does not

@halfwhole halfwhole force-pushed the chore/paginate-directory-search branch from 3a85d10 to dcd7398 Compare November 2, 2022 05:21
@halfwhole
Copy link
Collaborator Author

just rebased on latest develop and force-pushed

Copy link
Contributor

@thanhdatle thanhdatle left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@halfwhole halfwhole merged commit f64e160 into develop Nov 2, 2022
@halfwhole halfwhole deleted the chore/paginate-directory-search branch November 2, 2022 06:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants