🐛 "Could not delete lock key" -- Improve concurrency for ACA-Py Events Processor

[wdbasson]

{"message":"Could not delete lock key: lock:acapy-record-with-state-base. Perhaps it expired? | ","levelname":"WARNING","date":"2024-08-07 14:56:19.860175+00:00","name":"webhooks.services.acapy_events_processor","record":{"exception":null,"file":"/webhooks/services/acapy_events_processor.py","function":"_attempt_process_list_events","line":219,"process":{"id":1,"name":"MainProcess"},"thread":{"id":140042948758400,"name":"MainThread"},"time":{"repr":1723042579.860175,"uptime_h:m:s":"0:04:20.180110"}}}

[ff137]

"lock keys" are set in the ACA-Py Events Processor to ensure that only one instance will process a given item (a redis list of webhook events, per wallet).

These lock keys have an expiry of 500ms, where an async background task will re-set the lock key to increment the expiry time, so that it exists as long as is needed.

When load is very high, this async background task might not start soon enough, leaving a window where the lock doesn't exist anymore, resulting in the "Could not delete lock key" warning.

This may be a problem, as it opens up for the same list of events to be processed by an additional Webhooks Service instance.

The reason for the lock expiry is so that locks cannot exist indefinitely, in the case of Webhooks Service getting a sigterm mid-processing, resulting in a permanent lock.

A reasonable solution would be just to make the expiry much longer, perhaps up to 5 seconds long, so that processing time should never exceed that expiry time, and an async background task is not necessary to extend the lock. This is a trade-off in availability, as a sigterm could cause a webhook to take 5s longer to arrive than usual.

Alternatively, increase the frequency of the background task's re-setting of the lock key. This has a trade-off in increased redis traffic.

I think the first option is preferable, but let's see.