Request: efficient way to wait for circuit breaker to reopen

[kierenj]

Implementing a circuit breaker pattern in a worker process. When it's broken, I want to efficiently do nothing until it closes again.

It would be great if a circuit breaker policy had either a WaitHandle or awaitable.
Maybe a BrokenCircuitException could include the remaining time during which calls will fail?

I could remember the break duration and await Task.Delay(duration) or Thread.Sleep(duration), but that seems a pain. It also doesn't look possible to programmatically get the duration, making this more fiddly.

I've implemented with this:

            DateTime waitUntil;
            var circuitBreaker = Policy
                .Handle<Exception>()
                .CircuitBreaker(
                    10,
                    TimeSpan.FromSeconds(60.0),
                    (ex, ts) => waitUntil = DateTime.Now.Add(ts), () => { });

Then I'm waiting for waitUntil when the circuit is broken.

[aprooks]

I'm not sure if the flow you have described should be implemented like this. When a circuit breaker pops, you might expect the other system to be down for minutes and even hours.

I would expect a worker to have an environment specific repeat policy. For example, if your worker processes messages from some persistent queue most probably there will be a processing timeout (lease time) which you won't control. So you will have to reschedule your message to be processed (much) later, limit number of workers or stop processing at all for some time.

If you still would like to keep processing a single operation as long as possible, you might compose WaitAndRetry policy on top of the circuit breaker.

[kierenj]

I see what you're saying - I didn't anticipate that a circuit breaker would operate on that kind of timescale. Is that always / almost always the case?

I see your point with the retry suggestion, since it would seem I'm just waiting for a period of time until retrying again. In this case though I'm intending to wait before continuing operation, as opposed to retrying the exact same thing. However, I could indeed do that.

I guess it might still be handy to have the duration of the break in the exception, mind - perhaps it could filter up to the caller or some notification system to say, "this will be blocked for another X minutes". But if that's not too important either, no problem to close this off.

[aprooks]

I see what you're saying - I didn't anticipate that a circuit breaker would operate on that kind of timescale. Is that always / almost always the case?

yes, the circuit breaker is designed for mid-services communication, to protect both callee and caller. In a distributed world, a service can go down for hours (remember recent Azure outage for example).

Also, you would not like to jump start all your calls once service goes online since you might DDoS another service while it starts up and crash it again.

perhaps it could filter up to the caller or some notification system to say, "this will be blocked for another X minutes".

yes, this might be helpful for monitoring, but still, you only know when CB is configured to try close. If errors persist it will be kept open.

So maybe, it could be an "I am still open event" repeated every X minutes?