There should be separate Delayed Worker queues based on priority / importance

[awwaiid]

Summary

Right now we have a single default queue for Delayed Worker. This queue is shared by both HistoricalDataCacheJob and other jobs. We should separate HistoricalDataCacheJob at least into a separate queue so that if it gets backed we still send emails.

Things to consider

• Look to see what other jobs there are, specify a queue for each

Criteria for Completion

• There are different queues based on blocking priority
• Specs :)

[elasticspoon]

I did some investigation on how to set up the queues. It's pretty simple.

• Mailers: you can set queue_as :queue_name at a class level
• Jobs: you can def queue_name = :name_of_queue also at a class level.
• Queues: in config/initializers/delayed_job.rb you can set a priority:

# config/initializers/delayed_job.rb
Delayed::Worker.queue_attributes = {
  queue_name: { priority: -10 }, #this is high prio
  name_of_queue: { priority: 10 } # low prio
}

Those values can also be overridden if needed with object.delay(:queue => 'high_priority', priority: 0).method.

Thus, the question is, what should be the priorities? These are all the mailers and jobs in the application. @awwaiid lemme know what you think priorities should be.

account_request_mailer.rb
custom_devise_mailer.rb
distribution_mailer.rb
organization_mailer.rb
partner_mailer.rb
reminder_deadline_mailer.rb
request_mailer.rb
requests_confirmation_mailer.rb
user_mailer.rb

historical_data_cache_job.rb <- Should be low prio
notify_partner_job.rb
partner_mailer_job.rb
reminder_deadline_job.rb

Note: you can also have workers dedicated to certain queues. Not sure if there is any value to that, or how many workers are actually being ran.

[awwaiid]

From our call - I think a single low_priority queue that only the historical data cache is in, and then all the rest can remain at a higher-priority in default.

[cielf]

(assinging this after the fact)