chore: add transactional lock on CREATE OR REPLACE function

[gweiying]

Problem

What problem are you trying to solve? What issue does this close?

We run multiple instances in production. On initialisation, each instance attempts to upsert a CREATE OR REPLACE FUNCTION. If multiple instances attempts to update the function on the database at the same instance, a SequelizeDatabaseError: tuple concurrently updated error can result, as we observed on 6th May during a release.

This tuple concurrently updated occurs because Postgres does not implement locking for DDL on functions, detailed here, hence such errors can occur when launching multiple instances that all attempt to upsert the function at once.

More details on this can be found in the writeup here.

Solution

How did you solve the problem?

We add an advisory transactional lock with a unique ID on the CREATE OR REPLACE FUNCTION transaction. When an instance boots up and attempts to run that transaction concurrently as another instance, the instance has to acquire the lock before it can run the transaction. This way, the transactions are forced to run one after another and will never attempt to update the function concurrently.

Notes

1. We considered other solutions such as to remove the on-initialization upsert of the function, however this would mean that we will need to deploy any function changes manually if we go forward with this change. After discussing, we eventually decided against it as this could result in possible code drift between the application code on Github, and the database function itself which could be difficult to debug.

2. From Postgres documentation, these locks are not enforced by the database but by the application - These are called advisory locks, because the system does not enforce their use — it is up to the application to use them correctly.

Bug Fixes:

• Added pg_advisory_xact_lock

Tests

What tests should be run to confirm functionality?

• Run through manual test on staging environment to verify that application is working as normal
• Load test on staging to verify that database performance is not affected by the lock

[yong-jie]

lgtm!