feat: New fetch job function.

[TheNeedForSleep]

fixes #1242

New fetch job function:

• handles on update conflict
• works with same lock different priority task order
• inner select uses index properly

Does not support the aborting status!

Successful PR Checklist:

• Tests
  • (not applicable?)
• Documentation
  • (not applicable?)

PR label(s):

• PR type: breaking 💥 Contains breaking changes
• PR type: feature ⭐️ Contains new features
• PR type: bugfix 🕵️ Contains bug fix
• PR type: miscellaneous 👾 Contains misc changes
• PR type: dependencies 🤖 Contains only dependencies updates
• PR type: documentation 📚 Contains documentation updates

[TheNeedForSleep]

The code is not ready to be merged!

[TkTech]

The code is not ready to be merged!

You can mark the PR as a draft to make sure it doesn't get accidentally merged :)

[ewjoachim]

When the PR is ready, press the big "Ready for review" button

[joaquimds]

Thanks for this PR, @TheNeedForSleep ! I have just come up against this issue in a project I'm working on. I have temporarily "fixed" it for myself with pretty much the same SQL change:

CREATE OR REPLACE FUNCTION procrastinate_fetch_job(
    target_queue_names character varying[]
)
    RETURNS procrastinate_jobs
    LANGUAGE plpgsql
AS $$
DECLARE
	found_jobs procrastinate_jobs;
BEGIN
    WITH candidate AS (
        SELECT jobs.*
            FROM procrastinate_jobs AS jobs
            WHERE
                (
                    jobs.lock IS NULL OR
                    -- reject the job if its lock has earlier jobs
                    NOT EXISTS (
                        SELECT 1
                            FROM procrastinate_jobs AS earlier_jobs
                            WHERE
                                earlier_jobs.lock = jobs.lock
                                AND earlier_jobs.status IN ('todo', 'doing', 'aborting')
                                AND earlier_jobs.id < jobs.id)
                )
                AND jobs.status = 'todo'
                AND (target_queue_names IS NULL OR jobs.queue_name = ANY( target_queue_names ))
                AND (jobs.scheduled_at IS NULL OR jobs.scheduled_at <= now())
            ORDER BY jobs.priority DESC, jobs.id ASC LIMIT 1
            FOR UPDATE OF jobs SKIP LOCKED
    )
    UPDATE procrastinate_jobs
        SET status = 'doing'
        FROM candidate
        WHERE procrastinate_jobs.id = candidate.id
        RETURNING procrastinate_jobs.* INTO found_jobs;

	RETURN found_jobs;
END;
$$;

Question: why is it necessary to make the change in manager.py? It looks to me that the existing query will already fallback to a job where jobs.lock IS NULL.