Handling different workloads

This feature contributed by @jcalvert. What follows is use case and description.

In some cases where processes are managed through something like runit and environment variables through Chef configurations, it can be more convenient to have one master process managing several different worker pools.

We may also want to do configuration of Sneakers in one place for a Rails application, but with the number of workers varying with each group, such that simply having two processes with two WORKERS envs being insufficient.

An example might be one regular job queue and then a low volume, long running process queue. We want to logically separate them with different worker classes and queues, but may need many more processes for the regular queue than the low priority one.

By specifying an ENV WORKER_GROUP_CONFIG as a path to YAML file or by convention ./config/sneaker_worker_groups.yml we can use a file to specify groups like so:

foogroup:
  classes: FooWorker
  workers: 8
slowgroup:
  classes: SlowWorker
  workers: 2

Then we can create a single lightweight ruby process to spawn the individual rake worker tasks with the correct settings. One reason for doing it separately from Rake is that if forking from a Rake task, your memory footprint overall will be unnecessarily large.

Once the YAML file is in place, you can simply use bundle exec ruby -e "require 'sneakers/spawner';Sneakers::Spawner.spawn" to spawn workers specified from the config file. The signals "TERM", "USR1", "HUP", "USR2" are all forwarded to the child processes.