Receive failures lock transport into single concurrency mode

[DavidBoike]

Symptoms

After a receive failure arms the message pump's circuit breaker, it may fail to recover, causing the message queue size to grow due to the transport being locked into single concurrency mode. Only a restart of the endpoint can return it to full processing capacity.

This would be perceived as poor performance, but log files will show likely multiple instances of a circuit breaker being armed and/or disarmed.

Who's affected

All users of Azure Service Bus transport 3.2.0 and greater. However, the circuit breaker improvements have also been backported to 2.0.8 even though 2.0 did not contain the same concurrency bug.

Root cause

A race condition in the circuit breaker logic could, under the right conditions, result in the circuit breaker becoming armed, but subsequent successful messages could not disarm it unless another failure happened.

Fixes

• Receive failures lock transport into single concurrency mode #1059 in version 4.2.3
• Receive failures lock transport into single concurrency mode #1060 in version 4.0.3
• Receive failures lock transport into single concurrency mode #1061 in version 3.2.5
• Backport of circuit breaker fixes involved in bug in later versions #1062 in version 2.0.8