Internal: message processing is now event-based

[odesenfans]

Problem: the pending message fetcher and processor use a polling loop to look for messages to fetch/process. This leads to some latency when the pending_messages table is empty as the task sleeps while waiting for new pending messages.

Solution: add an exchange + queue in RabbitMQ to signal the arrival of new messages. To avoid modifying the message processor too much and avoid depending on coherency between the DB and RabbitMQ, the fetcher and processor simply spawn a new task that looks for messages and sets an asyncio Event object. The main fetching/processing loop waits on this event (with a timeout).

Note that this system is not used for retries as this would require another task that posts messages to the MQ on their next attempt. Retried messages simply wait for the next iteration of the loop (every second).

This solution has the following advantages and drawbacks:

• No more arbitrary latency when processing new messages
• No major modification of the pipeline, even if the MQ system fails for some reason the pending message processor will still process messages every second
• No dependency on the state of the message queue, if the RabbitMQ queue is deleted for any reason the processor will keep on working

• RabbitMQ overhead (one more exchange + queue).