test(integration): add integration test setup and url tests

[halfwhole]

Problem

Integration tests are important. With the new external API feature coming up, it would be especially helpful to have greater assurances about the behaviour of the backend as a whole, and be able to catch and prevent regressions before they happen.

Right now, our backend is covered by unit tests; however, they make use of extensive mocking and do not test the real aggregate behaviour of the backend system. Our E2E tests help to cover for this somewhat, but they cannot be used to test our internal or upcoming external APIs directly; plus, they're expensive to run and cannot be used to comprehensively test edge cases and failure behaviours.

Solution

Set up integration tests in /test/integration/. These are written and run using the Jest framework.

As a start, a few integration tests have been added to test the internal API URL functionalities in Urls.test.ts:

• Get URLs
• Create URLs (link and file)
• Update URLs (link and file)

To run the integration tests, run npm run test:integration.

Configuration

• To prevent database conflicts across tests, each test case separately creates and uses a different randomly generated email of the format integration-[0-9a-z]{10}@open.gov.sg
  • We connect to the local database and write raw SQL to reset the users and their URLs
  • There's no data seeding yet, but this should be easy to add in the future by extending the SQL scripts
• We wait 270s for docker-compose to finish spinning up the local environment before running integration tests
  • Not sure why the waiting time is set at exactly 270s, but I took this number from E2E tests which also sets it at 270s - these numbers should be consistent

Handling the database

• Current approach: reuse the local database (as set up in docker-compose.yml) for integration tests
  • This is what's currently being done for E2E tests as well
  • Slight problem: this approach is not the cleanest, because the same database is also used for local development. But doing this will not affect staging/prod anyway, and practically it will not affect development much either
• Other alternatives considered: 1) setting up a new database for testing only in the same postgres instance; 2) setting up separate postgres instance entirely for testing only
  • Problem: too costly to set up - it's difficult to link up sequelize to the new database or postgres instance, and create the tables correctly
  • Ideally, it would also be nice to access our models through sequelize ORM rather than raw queries, but again this is costly to set up

Other notes

• I find it very sad to have to attach authCookie to every request, but I simply couldn't figure out how to make auth work automatically with cross-fetch on the backend even after wrestling with it for hours. Maybe axios would be better?
• Actually, now that we use a different user and authCookie for each test case, it makes sense to attach it to every request. But the difficulty with using cross-fetch still stands :'(

Deploy Notes

ci.yml has been modified to include integration tests. Deployment to Elastic Beanstalk and Lambda will happen only if the integration tests pass (alongside CI and other Jest tests).

New dev dependencies:

• form-data : used in integration tests to create FormData objects for creating file links

[halfwhole]

have changed the tests such that each test case can be run independently - it uses its own randomly generated user, which is created before the test case and removed after. please take a look!

[gweiying]

lgtm!