Retry

Retry

Syntax

RetryPolicy retry = Policy
  .Handle<HttpException>()
  .Retry(3);

The above example will create a retry policy which will retry up to three times, an action which fails with an exception handled by the Policy.

For full retry syntax and overloads (including retry-forever, wait-and-retry, and related variants), see https://github.com/App-vNext/Polly#retry.

Syntax examples given are sync; comparable async overloads exist for asynchronous operation: see readme and wiki.

How Polly Retry works

When an action is executed through the policy:

• The retry policy attempts the action passed in the .Execute(…) (or similar) delegate.
  • If the action executes successfully, the return value (if relevant) is returned and the policy exits.
  • If the action throws an unhandled exception, it is rethrown and the policy exits: no further tries are made.
• If the action throws a handled exception, the policy:
  • Counts the exception
  • Checks whether another retry is permitted.
    • If not, the exception is rethrown and the policy terminates.
    • If another try is permitted, the policy:
      • for wait-and-retry policies, calculates the duration to wait from the supplied sleep duration configuration
      • raises the onRetry delegate (if configured)
      • for wait-and-retry policies, waits for the calculated duration.
      • Returns to the beginning of the cycle, to retry executing the action again.

Overall number of attempts

The overall number of attempts that may be made to execute the action is one plus the number of retries configured. For example, if the policy is configured .Retry(3), up to four attempts are made: the initial attempt, plus up to three retries.

Exponential backoff

A common retry strategy is exponential backoff: this allows for retries to be made initially quickly, but then at progressively longer intervals, to avoid hitting a subsystem with repeated frequent calls if the subsystem may be struggling.

Exponential backoff can be achieved by configuring waits-between-retries manually (useful if you desire some custom or an easily-readable scheme):

Policy
  .Handle<DivideByZeroException>()
  .WaitAndRetry(new[]
  {
    TimeSpan.FromSeconds(1),
    TimeSpan.FromSeconds(2),
    TimeSpan.FromSeconds(4),
    TimeSpan.FromSeconds(8),
    TimeSpan.FromSeconds(15),
    TimeSpan.FromSeconds(30)
  });

or by calculation:

Policy
  .Handle<DivideByZeroException>()
  .WaitAndRetry(3, retryAttempt => 
    TimeSpan.FromSeconds(Math.Pow(2, retryAttempt)) 
  );

Jitter

In very high throughput scenarios it can be beneficial to add jitter to wait-and-retry strategies, to prevent retries bunching into further spikes of load.

Consider a given call path experiencing hundreds of calls per second where the underlying system suddenly fails. A fixed-interval retry strategy or a fixed-progression exponential backoff can still generate peaks of load: if the underlying system fails at time t and you have a fixed-progression backoff strategy generating retries at 2, 4, 8, 15 and 30 seconds, you can expect to generate further specific peaks of load at t + 2, t + 4, t + 8 etc.

Jitter is a decorrelation strategy which adds randomness to retry intervals to avoid these spikes and spread out load.

When the response specifies how long to wait before retrying

Some systems specify how long to wait before retrying, as part of the fault response returned. For example, Azure CosmosDB sends a 429 response code (too many requests) with a x-ms-retry-after-ms header, indicating exactly how many milliseconds the caller should wait before retrying. This is in turn expressed back to calling code by the Azure SDK by throwing a DocumentClientException with a RetryAfter property.

This can be handled by using the WaitAndRetry/Forever/Async(...) overloads with a sleepDurationProvider which takes the handled fault/exception as an input parameter.

Other uses for Retry

While the original premise of retry policies is to handle transient faults, other uses are possible. We have heard of retry policies used to make BDD tests more robust (retrying actions that might initially fail only because some page element was yet to load). Another common pattern is to maintain authorisation against a third-party system, where that authorisation periodically lapses.

var authorisationEnsuringPolicy = Policy
  .Handle<SomeAuthorizationException>()
  .Retry(1, onRetry: (e, i) => RefreshAuthorization());

authorisationEnsuringPolicy.Execute(() => DoSomethingThatRequiresAuthorization());

If you wish to combine retry-for-reauthentication (as above) with retry-for-transient-faults, each retry policy can be expressed separately, and then combined with PolicyWrap.

Thread-safety

Each call to .Execute(…) (or similar) through a retry policy maintains its own private state. A retry policy can therefore be re-used safely in a multi-threaded environment.

The internal operation of the retry policy is thread-safe, but this does not magically make delegates you execute through the policy thread-safe: if the delegates you execute through the policy are not thread-safe, they remain not thread-safe.