Failover handlers when async_handler fails

[jondeandres]

This PR allow the users to define an array of failover handlers when the async_handler fails, for example in a Redis push, connection etc...

The users can use it in this way:

Rollbar.configure do |config|
  config.failover_handlers = [failover1, failover2] # failover1 and failover2 are just objects responding to #call
end

It's also possible to use the default async handler:

Rollbar.configure do |config|
  config.failover_handlers = Rollbar.default_async_handler
end

An interesting refactor could be replace the Rollbar.default_async_handler in order to use classes like Rollbar::Delay::Thread or Rollbar::Delay::GirlFriday. I think that Rollbar::Delay::Thread could be the most used failover handler.

[brianr]

Where is this exception going to go?

Asking because I want to make sure that:

• it won't bubble up to the end-user
• it still gets logged somewhere (i.e. to the rails log)

[jondeandres]

Here I'm only reraising the exception in case that the handler failed to process the payload. Really this is the actual behaviour. If you don't have any failover handler, then it just crashes, isn't it?

Well, it will not crash cause it's rescued in https://github.com/jondeandres/rollbar-gem/blob/failover/lib/rollbar.rb#L111.

This will be logged in https://github.com/jondeandres/rollbar-gem/blob/failover/lib/rollbar.rb#L566 finally

[brianr]

OK one last concern:

In the case where all failovers fail, we are eventually going to call report_internal_error which will call schedule_payload. But if the failovers all failed, then that schedule_payload call is also (probably) going to fail. That will result in a call to send_failsafe, which will call schedule_payload again, fail again, and then finally log an error here: https://github.com/jondeandres/rollbar-gem/blob/failover/lib/rollbar.rb#L620

The system is probably under stress when this happens so I think we should probably just short-circuit all of that and log, but not re-raise, the error when the failovers are exhausted. What do you think?

[jondeandres]

Mmmm. it's true @brianr...

What if we finally send it inline calling process_payload? Another option is disable use_async, call report_internal_error and re-enable it. But perhaps too complex.

Something we should do it's perhaps recommend the customers to use at least the Thread handler at the last. If that handler it's failing it's probably a problem with the operating system or in the API.

The easiest solution is just log an error here, of course.

[brianr]

I think recommending Thread last is a good idea. That plus logging an error here seems like the way to go.

[jondeandres]

One question @brianr, actually the behaviour you've explained it's really the actual behaviour, isn't it? I mean, if the only async_handler fails report_intenral_error will try to use that failing handler no?

Just to understand this.

[brianr]

Yeah, right now, in master, it's going to go around again (and probably fail again).

[jondeandres]

@brianr f810954

Do you think it's more correct now?