Datastore outage and consumer retry and restart query

[cookie-bytes]

In the event of a data store outage, our aim is to have our service consumers reattempt processing events. Instead of allowing messages to be transferred to a Dead Letter Queue (DLQ), we prefer to maintain the consumer offset at the current position.

We are considering setting up a retry policy that will attempt reprocessing ‘x’ number of times. Once these attempts have been exhausted, Silverback will halt the consumer.

To monitor the status of the consumer, we will implement health checks using the Consumer Management API.

If the consumer is not running, we will restart the service and re-enter the retry loop. Once the data store is back online, the service will resume processing events successfully.

Does the above sound sensible? Is there a more suitable approach available?

Thanks

[cookie-bytes]

K8s will be responsible for the health check and restarting the services.

[BEagle1984]

Yes, that would work.

You could also just let Silverback retry indefinitely (e.g. with increasingly longer intervals), to let it recover without manual interaction.

[cookie-bytes]

What is the best way to configure your retry indefinitely approach?

Thank you for all your help

[BEagle1984]

It will simply retry indefinitely if you configure a retry policy without specifying the retriesCount: see IErrorPolicyBuilder.

Note that you can build sophisticated policies like this:

.OnError(policy => policy
    .Retry(1) // 1st retry is immediate
    .ThenRetry(3, TimeSpan.FromSeconds(10)) // 2nd-4th attempts 10 seconds apart
    .ThenRetry(TimeSpan.FromMinutes(2))) // then retry indefinitely every 2 minutes

Remember to increase the MaxPollIntervalMs, if you want to increase the delay past the default 5 minutes.

[cookie-bytes]

Thank you as always for great insight