Multiple subscription webhook event calls recieved at once

[molexx]

We are seeing subscription create webhook calls followed very closely - according to Stripe's dev dashboard 1 millisecond later - by an update call for the same subscription. As django is still processing the first webhook call this causes a race condition and we are sometimes left with out-of-date data.

Is there a simple but robust fix for this?

My thoughts so far... none I like:

• ignore subscription update events where we haven't already processed a create event - this is brittle as should we miss the initial create we'll never tidy up even though there is enough info sent in every update to create the record
• database lock - would have to be a table-level lock during the create or update_or_create process
• we send Stripe a request for updated data after every webhook request we receive - this would add potentially unnecessary network traffic, should be delayed by a few seconds to ensure we get the freshest data, and would need de-duplicating when we receive multiple events for the same subscription. In our case that would need to be de-duplicated across multiple django processes.
• switch from listening for customer.subscription.created/customer.subscription.updated events to listening for invoice.paid events and expanding the subscription referenced in the invoice object.

Also with only 1ms between them there's a chance we could receive them out-of-order, and Stripe's documentation says to expect that. I'm somewhat OK with leaving that as a secondary issue for now in the hope that this is rare, unless there's a simple way of handling this cross-process?

[oscarychen]

I was not aware that Stripe is calling the web hooks at such frequency (1ms interval). I haven't given this too much thought, but I think you need a buffering layer in front of our webhooks to put Stripe's API calls into a queue, and then let Django process these requests in sequence.

AWS's SQS is something that comes to mind that might help with this. We would want the requests from Stripe to be stored in the Queue, and then let Django fetch messages from the queue.

Alternatively, you might also consider implementing a simple queue in Django's data model. The webhook now instead of trying to update our customer and subscriptions data, would instead simply add an entry into the in-database queue, and then from there trigger a process to update data, but at our own pace. You may need to use a task executer such as Celery as part of this solution. The overall sequence looks like this:

(1) Stripe calls webhook
(2) webhook adds Stripe's request to data table (queue table)
(3) Celery asynchronously runs a function (as a single scheduled task, every 2 minutes?)

• the function is globally unique(ie: only 1 instance of it can run at any given time, this can be controlled via Django cache variable)

(4) The function retrieves the oldest item from queue table, and process the data
(5) customer/subscription data is updated
(4) the function deletes the request from queue table, repeats step (4)
(5) if no more item in the queue table, the function exits.

This poses significant changes to this library. Some of these could also be done in the cloud on infrastructure side, maybe with the combination of a bit of AWS SQS and Lambda functions to act as a buffering layer.

All things considered I think only execute update on certain Stripe event (such as the Invoice.paid like you suggested) might be simpler in scope.

[oscarychen]

Also, is there reason why Subscription.update is fired immediately after Subscription.create? Is this something particular about how you have stripe configured to create subscriptions?

[molexx]

Agreed. I wish to avoid significant changes.

I'm looking at invoice.paid right now, perhaps that will be enough.

I'm not aware of making any configuration changes in Stripe. Just filtering events to customer.subscription.x ones. The create event creates the subscription record with a status of incomplete and the update changes it to active. Mostly it works fine but occasionally we have a subscription that is still at incomplete status which we are currently manually patching up but it creates a sub-par paying customer experience.

Here's a log of events when a new customer pays for a subscription, the subscription create and update events are highlighted in green:

and here are the created then updated events, 1ms apart: