fix(#4065): make low_prio queue for caching

[elasticspoon]

Resolves #4065

Description

Adds a "low_priority" delayed job queue. It will currently only have DataCache jobs.

Type of change

• New feature (non-breaking change which adds functionality)

How Has This Been Tested?

Job specs were edited to make sure the job is landing the right queue.

Note:

I could not for the life of me figure out how to test that the priority was working.

Based on the delayed job code I believe priority is set when the job is queued, that is when it is written to the database table "delayed_job_queue".

https://github.com/collectiveidea/delayed_job/blob/b66bb64437a8606a414a390531ac73108911434e/lib/delayed/backend/job_preparer.rb#L13

ActiveJob::TestHelper override the queue_adapter from delayed_job to the ActiveJob adapter. This means that priority never gets set from the queue in tests. So if we test: assert_enqueued_with(priority: some_value) it will always fail. We would have to explicitly set a priority, instead of using the queue priority which doesn't really make sense.

Just to sanity check the behavior I ran the following code in rails c:

5.times { |i|  HistoricalDataCacheJob.perform_later(org_id: Organization.last.id, type: i) }
NotifyPartnerJob.perform_later(Request.last.id)

the DataCache Jobs were correctly set with the low prio:

  Delayed::Backend::ActiveRecord::Job Create (0.6ms)  INSERT INTO "delayed_jobs" ("priority", "attempts", "handler", "last_error", "run_at", "locked_at", "failed_at", "locked_by", "queue", "created_at", "updated_at") VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11) RETURNING "id"  [["priority", 10], ["attempts", 0], ["handler", "--- !ruby/object:ActiveJob::QueueAdapters::DelayedJobAdapter::JobWrapper\njob_data:\n  job_class: HistoricalDataCacheJob\n  job_id: cbc1f289-70c5-480c-9dbf-b9a48ec269c2\n  provider_job_id:\n  queue_name: low_priority\n  priority:\n  arguments:\n  - org_id: 2\n    type: 0\n    _aj_ruby2_keywords:\n    - org_id\n    - type\n  executions: 0\n  exception_executions: {}\n  locale: en\n  timezone: UTC\n  enqueued_at: '2024-03-17T21:18:50Z'\n"], ["last_error", nil], ["run_at", "2024-03-17 21:18:50.659691"], ["locked_at", nil], ["failed_at", nil], ["locked_by", nil], ["queue", "low_priority"], ["created_at", "2024-03-17 21:18:50.659754"], ["updated_at", "2024-03-17 21:18:50.659754"]]
  TRANSACTION (2.2ms)  COMMIT

NotifyPartnerJob correctly fired off before the caching jobs despite being queued later:

16:55:06 worker.1 | [Worker(host:laptop pid:1071210)] Job NotifyPartnerJob [feac18aa-5d43-464a-9ff4-9ab1a238a027] from DelayedJob(default) with arguments: [200] (id=37) (queue=default) RUNNING
16:55:09 worker.1 | 2024-03-17T16:55:09-0400: [Worker(host:laptop pid:1071210)] Job HistoricalDataCacheJob [38641404-f54d-4b02-8b64-749271c6a6e7] from DelayedJob(low_priority) with arguments: [{"org_id"=>2, "type"=>"0", "_aj_ruby2_keywords"=>["org_id", "type"]}] (id=32) (queue=low_priority) RUNNING

[dorner]

I'd just fix the comment, and then I think this is good to merge - maybe @awwaiid wants to take a look.

[awwaiid]

Looks great, thank you!

Change was applied!