Efficiency of SIGKILL safety relative to Sidekiq Pro

[machty]

This is a bit in the weeds question that I hope makes some sense.

Sidekiq Pro gives you a safer job fetching algorithm called ReliableFetch that internally uses Redis's BRPOPLPUSH command in a manner that avoids potentially data/job loss if the process is killed at the wrong time, but because Redis doesn't support this operation on multiple queues, Sidekiq has to resort to polling, which increases the pressure on redis, particularly if you have Sidekiq worker processes that listen to multiples queues.

So I'm wondering: does good_job suffer the same kind of issue? Specifically, is there a fundamental/algorithmic limitation that would cause good_job to also struggle in the case of 1. worker using multiple queues and 2. SIGKILL safety. If not, how does good_work get around this?

[bensheldon]

It's a bit comparing apples and oranges: Redis is highly optimized for a relatively small number of instructions, whereas Postgres is a general purpose database that will do its best for any arbitrary SQL query. So I can't really compare performance: GoodJob definitely asks a lot of the Postgres database, but I don't think it's comparable to access patterns in Redis.

GoodJob achieves SIGKILL safety by not removing records from the database. GoodJob takes an Advisory Lock on the job record, remaining in the database; in the event of a SIGKILL, the database connection is closed by the database and the Advisory Lock is automatically released.

Postgres doesn't have a queue-pop, so GoodJob does a combination of polling for new jobs, using LISTEN/NOTIFY to trigger on newly enqueued jobs, and reading ahead and caching in memory when to poll for future-scheduled jobs.

In GoodJob, adding additional WHERE conditions (like WHERE queue_names IN ('slow', 'fast' , 'etc') does incur a query cost as those values need to be added to indexes, or scanned and condition/filter checked... but it's all part of the same SQL query. So the cost/complexity question is more like "is this index or bitmap scannable? is the data in the index or leaf page, or does it have to do a heap fetch?", and some things seem like they're ok (like multiple queues), and some things are really expensive (like + ordered queues feature: +fast,slow). It's really all SQL query and index optimization. You could run EXPLAIN ANALYZE on the query in your Rails logs and inspect it.