[Feature] Background task queue

[Yoronex]

What would you like?

A task queue, where we can put in tasks to be executed asynchronously in the background.

Why is this needed?

There are some operations that take some time to finish. Ideally, we execute these tasks in the background, so the user does not have to wait for these tasks to finish. A prime example is with sending emails. Right now, emails regarding a user's balance are sent asynchronously within NodeJS. This works, but there is no way to do anything in case such a task fails. Furthermore, certain endpoints (like the fine notify) send all these emails synchronously, which can cause a request timeout if it takes more than a minute.

In short: it would be nice if all emails can be send asynchronously.

How could it be implemented?

Laravel already has a pretty nice design for something like this, so we can use it for inspiration. https://laravel.com/docs/11.x/queues

Basically, it comes down to the following things:

• We add a database table called "tasks", where each entry is a JSON-representation of the task to be executed. This entry should contain all information to run the task independently.
• We add a task runner to cron.ts that executes one task every second. This ensures that each task can finish in time.
• We create a "failed_tasks" table, which we use to track any failed tasks. After a cooldown period, we can try the task again.

Optionally, we add the following things to make this implementation more robust for future updates:

• Each task contains a SudoSOS version for which it has been created. If we by accident perform an update while the task queue is not empty, these tasks will be blocked.
• Integrate the failed_tasks table with some logging system, so we are notified in case a task fails. This can also be within SudoSOS itself (for example in the status endpoint, as SudoSOS is not "healthy" when tasks are failing).

Other information

No response