feat: back-end implementation of ranked link seach

[JasonChong96]

Problem

"Members of the public (MOPs) view go.gov.sg as a central hub for accessing government resources. By providing a link search feature, we will be able to better direct MOPs to access the resources that they require."

• Link Search #181

Solution

This PR continues the implementation by providing the back-end API for fetching links using plain text queries.

Features:

• A new endpoint api/search/urls has been added for ranked plain text search with support for pagination.
• Similar to api/user/url, the response contains the total count of matching urls and the urls within the requested range.
• The search takes into account both the shortUrl and description of each entry. Text from the description has a lower weightage than those from the shortUrl under the assumption that we can take shortUrl as the title and hence contains words which are more important.
• The search uses PostgreSQL's ts_rank_cd, which takes into account how far apart query terms are found in the urls. The further they are, the lower the ranking.
• The raw query params are passed separately from the rest of the query using parameter binding. This ensures that the endpoint is not susceptible to SQL injection given that a raw query is used. (Tested using Postman)
• The API supports the following ranking conditions: relevance, recency and popularity
• This new endpoint is rate limited to 20 requests/second/user
• Search ignores INACTIVE urls and they are not included in the partial inverted index used for search
• Relevancy ranking is normalized by multiplying by 1 / log(doc length + 1). This is due to the assumption that if there are less words that do not match the query, then the terms are more important in the entry, making it more likely to be relevant to the user's query.

The relevance ranking algorithm used is as follows:
(text ranking by PostgreSQL) * log(1 + clickCount)

Notes: The 1 is added to click counts to prevent 0 clickCount from causing an error. There is an assumption made that more popular links are more likely to be relevant to users' queries.

Additional notes

A request to the endpoint requires two separate database queries. This mimics the behavior of Sequelize's findAndCountAll which we use for api/user/url to support pagination.

Deploy Notes

• The migration file provided creates the index required for this search to be run efficiently. Benchmarks done on local machine shows a speed increase of around 20x.
• The migration file should only be run after the relevant columns are added from the phase 0 implementation.
• The migration file does create any breaking changes so it can be run before this implementation is deployed.

Dependencies

• express-rate-limit: rate limiter middleware for api endpoints
• @types/express-rate-limit: type definitions for express-rate-limit

TODO:

• Implement endpoint
• Create migration file for indexing
• Wait for phase 0 to be merged into develop and rebase
• Remove click count from API response
• Support different sorting orders
• Add rate limiting
• Add tests

Full documentation of this feature will be done on the wiki asap

[liangyuanruo]

changes as commented - largely lgtm otherwise!

[liangyuanruo]

to address conflicts before merging