ThreadsOnFork worker

Backburner now supports (as of a 0.3.0) a ThreadsOnFork worker thanks to ShadowBelmolve.

This worker will create a fork for each tube being watched and then spawn up a configurable number of threads to process jobs for each tube. After a configurable number of processed jobs, a new fork will be started to keep the environment clean and protect against memory bloat.

You can select ThreadsOnFork as the default worker for processing with:

Backburner.configure do |config|
  config.default_worker = Backburner::Workers::ThreadsOnFork
end

or determine the worker on the fly when invoking work:

Backburner.work('newsletter_sender', :worker => Backburner::Workers::ThreadsOnFork)

or through the rake tasks with:

$ QUEUES=newsletter-sender,push-message THREADS=2 GARBAGE=1000 rake backburner:threads_on_fork:work

You can also configure more advanced options to take control of the threads and garbage for each individual tube in the format of QUEUE=TUBE_NAME:THREADS_LIMIT:GARBAGE_LIMIT:RETRIES_LIMIT. For example:

QUEUE=twitter:10:50:5,parse_page:20:10,send_mail:5,verify_bithday::10 THREADS=2 GARBAGE=1000 rake backburner:threads_on_fork:work

With this:

• twitter tube will have 10 threads, garbage after 50 executions and retry jobs 5 times.
• parse_page will have 20 threads, garbage after 10 executions and retry as the default.
• send_mail will have 5 threads, garbage after 1k and retry as the default.
• verify_birthday will have 2 threads, garbage after 1k and retry 10 times.

Also queue name with settings can be specified in job class:

class TwitterJob
  include Backburner::Queue
  queue 'twitter:10:50:5'
end

or queue settings can be configured at class level like this:

class TwitterJob
  include Backburner::Queue
  queue 'twitter'
  queue_jobs_limit 10
  queue_garbage_limit 50
  queue_retry_limit  5
end

*Note: class level settings override settings specified in name.