Adds background job scheduling and execution #11238

cannikin · 2024-08-13T22:21:15Z

This new package provides scheduling and processing of background jobs. We want everything needed to run a modern web application to be included in Redwood itself—you shouldn't need any third party integrations if you don't want.

Background jobs have been sorely missed, but the time has come! (If you do want to use a third party service we have had an integration with Inngest since May of 2023!)

Features

Named queues: you can schedule jobs in separate named queues and have a different number of workers monitoring each one—makes it much easier to scale your background processing.
Priority: give your jobs a priority from 1 (highest) to 100 (lowest). Workers will sort available jobs by priority, working on the most important ones first.
Configurable delay: run your job as soon as possible (default), wait a number of seconds before running, or run at a specific time in the future.
Auto-retries with backoff: if your job fails it will back off at the rate of attempts ** 4 for a default of 24 tries, the time between the last two attempts is a little over three days.
Run inline: instead of scheduling to run in the background, run immediately.
Integrates with Redwood's logger: use your existing one in api/src/lib/logger or create a new one just for job logging.

Usage

Check out the docs!

The Future

More adapters (maybe the community wants to get involved?): Redis, SQS, RabbitMQ
Studio integration: monitor the state of your outstanding jobs
Baremetal integration: if jobs are enabled, monitor the workers
Recurring jobs
Lifecycle hooks: beforePerform(), afterPerform(), afterSuccess(), afterFailure()

TODO

Update docs

Thanks to @Josh-Walker-GM for the help in the recent refactor!

Closes #11122

…ted when using schedule/success/failure/clear

…ground-jobs

…n rw-jobs` behaves the same

…mmand

packages/jobs/src/adapters/PrismaAdapter/PrismaAdapter.ts

packages/jobs/src/core/JobManager.ts

packages/jobs/src/core/Worker.ts

Tobbe · 2024-08-21T09:49:24Z

docs/docs/background-jobs.md

+
+You'll most likely be scheduling work as the result of one of your service functions being executed. Let's say we want to schedule our `SampleJob` whenever a new user is created:
+
+```js title="api/src/services/users/users.js"


Please add TS versions of all code listings as well (as we do for our tutorial, so the user gets an option to switch between TS and JS)

docs/docs/background-jobs.md

docs/static/img/background-jobs/jobs-after.png

docs/static/img/background-jobs/jobs-before.png

docs/static/img/background-jobs/jobs-queues.png

docs/static/img/background-jobs/jobs-terminal.png

docs/static/img/background-jobs/jobs-workers.png

…into feat/rw-uploads-extension * 'feat/rw-uploads-extension' of github.com:dac09/redwood: chore(test-fixture): correct rebuild script string suffix substitution (redwoodjs#11339) refactor(forms): Build with esbuild and add conditional exports (redwoodjs#11338) refactor(prerender): build with esbuild and introduce conditional exports (redwoodjs#11337) Adds more tests! Refactor rw-jobs-worker (redwoodjs#11335) refactor(auth-providers): switch to esbuild for some 'web' packages (redwoodjs#11334) Jobs doc updates (redwoodjs#11333) chore(ci): Split formatting check into separate job (redwoodjs#11332) Updates caching doc code snippets to be consistent (redwoodjs#11331) chore(tsconfig): Properly link to api-server's tsconfig (redwoodjs#11330) Adds background job scheduling and execution (redwoodjs#11238) [gh11317] Added documentation for working with Jest and Alias Paths (redwoodjs#11323) fix(record): Update dependencies and build dual esm/cjs (redwoodjs#11300) chore(build): Consistent tsconfig and buildinfo naming for CJS (redwoodjs#11326) chore(test): Fix RSC server cell smoke test (redwoodjs#11327) chore(playwright): Make RSC tests less flakey (redwoodjs#11325) chore(ci): Temporary fix for format checking on all PRs (redwoodjs#11324) chore(build): Move more repetitive code into framework-tools (redwoodjs#11320) chore(docs): fix docs formatting (redwoodjs#11321)

cannikin added 30 commits June 27, 2024 20:13

Use console if logger not yet, remove private #log() function

1de0a44

Find job in any subdirectory of api/src/jobs

c927e35

Comment loaders

8b12c77

Import setTimeout

d8fbab0

Remove index file in adapters

91ca380

Comments, remove unused args

192e929

Convert sqlite query to regular prisma calls

7d90508

Update worker debug message to include queue name being searched

f6caf73

Comments

67fb339

Make sure await imports work on windows

626bcee

Fix some jest -> vi mocking differences

0b4222b

Don’t load ENV files in worker.js for now

54315c8

Placeholder tests

328d341

Fix vitest watch

0fddf26

Remove scenarios file

4c28c64

Fix PrismaAdapter tests

745febe

Finishes PrismaAdapter.schedule() tests

88a7c41

Refactor .find() for just the single default query

f2d83d4

Rename bins

ae59600

Remove unneeded awaits

f9073ee

Convert loaders to TS

6693d5f

Add TS to BaseAdapter, add separate types file for shared types

c4c822f

Add TS to Executor

cccf759

Comments on types

b8247b8

Switches PrismaAdapter to TS

a121a89

Update tests to it() instead of test(), reword a couple tests

83f4770

PrismaAdapter should return the records that were created/edited/dele…

d46dda8

…ted when using schedule/success/failure/clear

Adds changeset

a866761

Enable dual build?

ae38c44

Always await PrismaAdapter responses

825076e

cannikin and others added 23 commits August 19, 2024 18:35

Adds tests for rw-jobs

0bee631

Merge remote-tracking branch 'origin/main' into rc-background-jobs

1de3bd9

use underline for link styling

0a1d82c

update deps and building

5b1c7ea

ensure tests are ts

13b2458

log heap usage

d958e5f

update util mocks

446ea5d

update worker test

45b8663

update executor test

6ae9f5f

update job manager test

1a5b91b

update schedular test

26e51d0

revert cli-helpers change

f44fe04

misc docs updates

b19cfff

fix __dirname and cjs typegen issue

e89363f

Rewrite docs for type

f5a579c

Add demandCommand() to jobs cli

b48bf1c

Merge remote-tracking branch 'origin/rc-background-jobs' into rc-back…

1a3181f

…ground-jobs

remove stray comment

fbf7e51

Back to strict options

bdb0807

Check for first command so that starting with yarn rw jobs and `yar…

e13464d

…n rw-jobs` behaves the same

Replace command name in rw-jobs so help output matches actual yarn co…

1d48740

…mmand

Make --index and --id required

401a8d5

Merge branch 'main' into rc-background-jobs

9b12601

Josh-Walker-GM approved these changes Aug 21, 2024

View reviewed changes

Josh-Walker-GM merged commit 865ff5c into main Aug 21, 2024
45 checks passed

Josh-Walker-GM deleted the rc-background-jobs branch August 21, 2024 03:06

Tobbe reviewed Aug 21, 2024

View reviewed changes

Josh-Walker-GM modified the milestones: next-release, v8.0.0 Sep 4, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Adds background job scheduling and execution #11238

Adds background job scheduling and execution #11238

cannikin commented Aug 13, 2024 •

edited

Loading

Tobbe Aug 21, 2024


		You'll most likely be scheduling work as the result of one of your service functions being executed. Let's say we want to schedule our `SampleJob` whenever a new user is created:

		```js title="api/src/services/users/users.js"

Adds background job scheduling and execution #11238

Adds background job scheduling and execution #11238

Conversation

cannikin commented Aug 13, 2024 • edited Loading

Features

Usage

The Future

TODO

Tobbe Aug 21, 2024

Choose a reason for hiding this comment

cannikin commented Aug 13, 2024 •

edited

Loading