Recover from Redis failure

[gjtorikian]

Today, an issue with Redis prevented the app from processing the queue, as well as from waking back up. This should be addressed.

Here's the stack trace from the failure:

May 13 15:00:57 github-repository-sync app/worker.1:  rake aborted! 
May 13 15:00:57 github-repository-sync app/worker.1:  Redis::CannotConnectError: Error connecting to Redis on angelfish.redistogo.com:9615 (ECONNREFUSED) 
May 13 15:00:57 github-repository-sync app/worker.1:  /app/vendor/bundle/ruby/2.0.0/gems/redis-3.1.0/lib/redis/client.rb:309:in `rescue in establish_connection' 
May 13 15:00:57 github-repository-sync app/worker.1:  /app/vendor/bundle/ruby/2.0.0/gems/redis-3.1.0/lib/redis/client.rb:304:in `establish_connection' 
May 13 15:00:57 github-repository-sync app/worker.1:  /app/vendor/bundle/ruby/2.0.0/gems/redis-3.1.0/lib/redis/client.rb:85:in `block in connect' 
May 13 15:00:57 github-repository-sync app/worker.1:  /app/vendor/bundle/ruby/2.0.0/gems/redis-3.1.0/lib/redis/client.rb:266:in `with_reconnect' 
May 13 15:00:57 github-repository-sync app/worker.1:  /app/vendor/bundle/ruby/2.0.0/gems/redis-3.1.0/lib/redis/client.rb:84:in `connect' 
May 13 15:00:57 github-repository-sync app/worker.1:  /app/vendor/bundle/ruby/2.0.0/gems/redis-3.1.0/lib/redis/client.rb:326:in `ensure_connected' 
May 13 15:00:57 github-repository-sync app/worker.1:  /app/vendor/bundle/ruby/2.0.0/gems/redis-3.1.0/lib/redis/client.rb:197:in `block in process' 
May 13 15:00:57 github-repository-sync app/worker.1:  /app/vendor/bundle/ruby/2.0.0/gems/redis-3.1.0/lib/redis/client.rb:279:in `logging' 
May 13 15:00:57 github-repository-sync app/worker.1:  /app/vendor/bundle/ruby/2.0.0/gems/redis-3.1.0/lib/redis/client.rb:196:in `process' 
May 13 15:00:57 github-repository-sync app/worker.1:  /app/vendor/bundle/ruby/2.0.0/gems/redis-3.1.0/lib/redis/client.rb:102:in `call' 
May 13 15:00:57 github-repository-sync app/worker.1:  /app/vendor/bundle/ruby/2.0.0/gems/redis-3.1.0/lib/redis.rb:784:in `block in get' 
May 13 15:00:57 github-repository-sync app/worker.1:  /app/vendor/bundle/ruby/2.0.0/gems/redis-3.1.0/lib/redis.rb:37:in `block in synchronize' 
May 13 15:00:57 github-repository-sync app/worker.1:  /app/vendor/bundle/ruby/2.0.0/gems/redis-3.1.0/lib/redis.rb:37:in `synchronize' 
May 13 15:00:57 github-repository-sync app/worker.1:  /app/vendor/bundle/ruby/2.0.0/gems/redis-3.1.0/lib/redis.rb:783:in `get' 
May 13 15:00:57 github-repository-sync app/worker.1:  /app/vendor/bundle/ruby/2.0.0/gems/redis-namespace-1.5.0/lib/redis/namespace.rb:383:in `call_with_namespace' 
May 13 15:00:57 github-repository-sync app/worker.1:  /app/vendor/bundle/ruby/2.0.0/gems/redis-namespace-1.5.0/lib/redis/namespace.rb:282:in `method_missing' 
May 13 15:00:57 github-repository-sync app/worker.1:  /app/vendor/bundle/ruby/2.0.0/gems/resque-1.25.2/lib/resque/worker.rb:617:in `job' 
May 13 15:00:57 github-repository-sync app/worker.1:  /app/vendor/bundle/ruby/2.0.0/gems/resque-1.25.2/lib/resque/worker.rb:546:in `unregister_worker' 
May 13 15:00:57 github-repository-sync app/worker.1:  /app/vendor/bundle/ruby/2.0.0/gems/resque-1.25.2/lib/resque/worker.rb:215:in `rescue in work' 
May 13 15:00:57 github-repository-sync app/worker.1:  /app/vendor/bundle/ruby/2.0.0/gems/resque-1.25.2/lib/resque/worker.rb:162:in `work' 
May 13 15:00:57 github-repository-sync app/worker.1:  /app/vendor/bundle/ruby/2.0.0/gems/resque-1.25.2/lib/resque/tasks.rb:41:in `block (2 levels) in <top (required)>'