Adding batching to re-queuing for timestamp

[brennen-stripe]

During times of high load on our system, we noticed that our queue was behaving poorly. One of the things we noticed was that our scheduled job queue was lagging behind real time. In other words, the re-scheduling was dealing with timestamps that were in the past, sometimes by as much as 15 minutes.

We added a metric around this to keep track of it, you can see an example chart below.

After diving into the Resque-Scheduler code, it became apparent that the delay was stemming from inefficient queueing of scheduled jobs. This PR aims to fix that by adding batch scheduling, with a customizable batch size.

After sourcing my fork of this repository with this patch applied in our production environment, we have seen the delay between the scheduler and real time virtually eliminated. You can see for yourself in the chart below.

[PatrickTulskie]

Hey @brennen-stripe... can we separate out the feature/tests from changes to the test matrix in another PR? Also, can we get a PR description so we understand what this is changing? Thanks!

[brennen-stripe]

@PatrickTulskie Apologies, added WIP and moved this to a draft. Will be removing the matrix changes, just wanted to sanity check some version compatibility and fix the unrelated the test failures to see green builds. I see theres another PR duplicating that work, so won't make a new one.

[brennen-stripe]

@PatrickTulskie Alright this should be good to go!

[iloveitaly]

All tests except one is passing. The one that isn't passing does pass on a previous ruby version, and it's ruby 2.3, so we can handle separately.

[alxckn]

@brennen-stripe your initial issue with lag at stripe seems to have been quite severe (multiple minutes on average).
I'm wondering how far down this fix managed to get the average/p95 down for you (few seconds?) and if you are using a custom batch size higher than the default of 100?