Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Couple more sections out of the README #1178

Merged
merged 6 commits into from
Nov 11, 2024
+276 −354
Merged

Couple more sections out of the README #1178

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
6303b39
Split more docs out of the README
whatyouhide Nov 11, 2024
dbe947d
Move SQLite3 out of README
whatyouhide Nov 11, 2024
9990dc8
Remove other stuff from README
whatyouhide Nov 11, 2024
4c65185
Apply suggestions from code review
whatyouhide Nov 11, 2024
64ab101
More docs
whatyouhide Nov 11, 2024
3d45960
Update guides/isolation.md
sorentwo Nov 11, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
339 changes: 2 additions & 337 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@
- [Learning](#learning)
- [Requirements](#requirements)
- [Installation](#installation)
- [Running With SQLite3](#running-with-sqlite3)
- [Running With SQLite3](https://hexdocs.pm/oban/sqlite3s.html)
- [Configuring Queues](#configuring-queues)
- [Defining Workers](#defining-workers)
- [Enqueueing Jobs](#enqueueing-jobs)
Expand All @@ -48,7 +48,7 @@
- [Periodic Jobs](https://hexdocs.pm/oban/periodic_jobs.html)
- [Error Handling](https://hexdocs.pm/oban/error_handling.html)
- [Instrumentation and Logging](https://hexdocs.pm/oban/instrumentation.html)
- [Instance and Database Isolation](#instance-and-database-isolation)
- [Instance and Database Isolation](https://hexdocs.pm/oban/isolation.html)
- [Community](#community)
- [Contributing](#contributing)

Expand Down Expand Up @@ -163,29 +163,6 @@ Oban requires Elixir 1.14+, Erlang 23+, and PostgreSQL 12.0+ or SQLite3 3.37.0+.
See the [installation guide](https://hexdocs.pm/oban/installation.html) for details on installing
and configuring Oban in your application.

## Running with SQLite3

Oban ships with engines for PostgreSQL and SQLite3. Both engines support the same core
functionality for a single node, while the Postgres engine is more advanced and designed to run in
a distributed environment.

Running with SQLite3 requires adding `ecto_sqlite3` to your app's dependencies and setting the
`Oban.Engines.Lite` engine:

```elixir
config :my_app, Oban,
engine: Oban.Engines.Lite,
queues: [default: 10],
repo: MyApp.Repo
```

> #### High Concurrency Systems {: .warning}
>
> SQLite3 may not be suitable for high-concurrency systems or for systems that need to handle
> large amounts of data. If you expect your background jobs to generate high loads, it would be
> better to use a more robust database solution that supports horizontal scalability, like
> Postgres.

## Configuring Queues

Queues are specified as a keyword list where the key is the name of the queue
Expand Down Expand Up @@ -598,318 +575,6 @@ config :my_app, Oban,
`discarded`. It'll never delete a new job, a scheduled job or a job that
failed and will be retried.

## Error Handling

When a job returns an error value, raises an error, or exits during execution the
details are recorded within the `errors` array on the job. When the number of
execution attempts is below the configured `max_attempts` limit, the job will
automatically be retried in the future.

The retry delay has an exponential backoff, meaning the job's second attempt
will be after 16s, third after 31s, fourth after 1m 36s, etc.

See the `Oban.Worker` documentation on "Customizing Backoff" for alternative
backoff strategies.

### Error Details

Execution errors are stored as a formatted exception along with metadata about
when the failure occurred and which attempt caused it. Each error is stored with
the following keys:

* `at` The UTC timestamp when the error occurred at
* `attempt` The attempt number when the error occurred
* `error` A formatted error message and stacktrace

See the [Instrumentation](#instrumentation-error-reporting-and-logging) docs for an example of
integrating with external error reporting systems.

### Limiting Retries

By default, jobs are retried up to 20 times. The number of retries is controlled
by the `max_attempts` value, which can be set at the Worker or Job level. For
example, to instruct a worker to discard jobs after three failures:

```elixir
use Oban.Worker, queue: :limited, max_attempts: 3
```

### Limiting Execution Time

By default, individual jobs may execute indefinitely. If this is undesirable you
may define a timeout in milliseconds with the `timeout/1` callback on your
worker module.

For example, to limit a worker's execution time to 30 seconds:

```elixir
def MyApp.Worker do
use Oban.Worker

@impl Oban.Worker
def perform(_job) do
something_that_may_take_a_long_time()

:ok
end

@impl Oban.Worker
def timeout(_job), do: :timer.seconds(30)
end
```

The `timeout/1` function accepts an `Oban.Job` struct, so you can customize the
timeout using any job attributes.

Define the `timeout` value through job args:

```elixir
def timeout(%_{args: %{"timeout" => timeout}}), do: timeout
```

Define the `timeout` based on the number of attempts:

```elixir
def timeout(%_{attempt: attempt}), do: attempt * :timer.seconds(5)
```

If the job fails to execute before the timeout period then it will error with a dedicated
`Oban.TimeoutError` exception. Timeouts are treated like any other failure and the job will be
retried as usual if more attempts are available.

## Instrumentation, Error Reporting, and Logging

Oban provides integration with [Telemetry][tele], a dispatching library for
metrics. It is easy to report Oban metrics to any backend by attaching to
`:oban` events.

Here is an example of a sample unstructured log handler:

```elixir
defmodule MyApp.ObanLogger do
require Logger

def handle_event([:oban, :job, :start], measure, meta, _) do
Logger.warning("[Oban] :started #{meta.worker} at #{measure.system_time}")
end

def handle_event([:oban, :job, event], measure, meta, _) do
Logger.warning("[Oban] #{event} #{meta.worker} ran in #{measure.duration}")
end
end
```

Attach the handler to success and failure events in `application.ex`:

```elixir
events = [[:oban, :job, :start], [:oban, :job, :stop], [:oban, :job, :exception]]

:telemetry.attach_many("oban-logger", events, &MyApp.ObanLogger.handle_event/4, [])
```

The `Oban.Telemetry` module provides a robust structured logger that handles all
of Oban's telemetry events. As in the example above, attach it within your
`application.ex` module:

```elixir
:ok = Oban.Telemetry.attach_default_logger()
```

For more details on the default structured logger and information on event
metadata see docs for the `Oban.Telemetry` module.

### Reporting Errors

Another great use of execution data is error reporting. Here is an example of
integrating with [Honeybadger][honeybadger] to report job failures:

```elixir
defmodule MyApp.ErrorReporter do
def attach do
:telemetry.attach(
"oban-errors",
[:oban, :job, :exception],
&__MODULE__.handle_event/4,
[]
)
end

def handle_event([:oban, :job, :exception], measure, meta, _) do
Honeybadger.notify(meta.reason, stacktrace: meta.stacktrace)
end
end

MyApp.ErrorReporter.attach()
```

You can use exception events to send error reports to Sentry, AppSignal, Honeybadger, Rollbar, or any other application monitoring platform.

Some of these services support reporting Oban errors out of the box:

- Sentry — [Oban integration documentation](https://docs.sentry.io/platforms/elixir/integrations/oban)
- AppSignal - [Oban integration documentation](https://docs.appsignal.com/elixir/integrations/oban.html)

[tele]: https://hexdocs.pm/telemetry
[honeybadger]: https://www.honeybadger.io

## Instance and Database Isolation

You can run multiple Oban instances with different prefixes on the same system
and have them entirely isolated, provided you give each supervisor a distinct
id.

Here we configure our application to start three Oban supervisors using the
"public", "special" and "private" prefixes, respectively:

```elixir
def start(_type, _args) do
children = [
Repo,
Endpoint,
{Oban, name: ObanA, repo: Repo},
{Oban, name: ObanB, repo: Repo, prefix: "special"},
{Oban, name: ObanC, repo: Repo, prefix: "private"}
]

Supervisor.start_link(children, strategy: :one_for_one, name: MyApp.Supervisor)
end
```

### Umbrella Apps

If you need to run Oban from an umbrella application where more than one of
the child apps need to interact with Oban, you may need to set the `:name` for
each child application that configures Oban.

For example, your umbrella contains two apps: `MyAppA` and `MyAppB`. `MyAppA` is
responsible for inserting jobs, while only `MyAppB` actually runs any queues.

Configure Oban with a custom name for `MyAppA`:

```elixir
config :my_app_a, Oban,
name: MyAppA.Oban,
repo: MyApp.Repo
```

Then configure Oban for `MyAppB` with a different name:

```elixir
config :my_app_b, Oban,
name: MyAppB.Oban,
repo: MyApp.Repo,
queues: [default: 10]
```

Now, use the configured name when calling functions like `Oban.insert/2`,
`Oban.insert_all/2`, `Oban.drain_queue/2`, etc., to reference the correct Oban
process for the current application.

```elixir
Oban.insert(MyAppA.Oban, MyWorker.new(%{}))
Oban.insert_all(MyAppB.Oban, multi, :multiname, [MyWorker.new(%{})])
Oban.drain_queue(MyAppB.Oban, queue: :default)
```

### Database Prefixes

Oban supports namespacing through PostgreSQL schemas, also called "prefixes" in
Ecto. With prefixes your jobs table can reside outside of your primary schema
(usually public) and you can have multiple separate job tables.

To use a prefix you first have to specify it within your migration:

```elixir
defmodule MyApp.Repo.Migrations.AddPrefixedObanJobsTable do
use Ecto.Migration

def up do
Oban.Migrations.up(prefix: "private")
end

def down do
Oban.Migrations.down(prefix: "private")
end
end
```

The migration will create the "private" schema and all tables, functions and
triggers within that schema. With the database migrated you'll then specify the
prefix in your configuration:

```elixir
config :my_app, Oban,
prefix: "private",
repo: MyApp.Repo,
queues: [default: 10]
```

Now all jobs are inserted and executed using the `private.oban_jobs` table. Note
that `Oban.insert/2,4` will write jobs in the `private.oban_jobs` table, you'll
need to specify a prefix manually if you insert jobs directly through a repo.

Not only is the `oban_jobs` table isolated within the schema, but all
notification events are also isolated. That means that insert/update events will
only dispatch new jobs for their prefix.

### Dynamic Repositories

Oban supports [Ecto dynamic repositories][dynamic] through the
`:get_dynamic_repo` option. To make this work, you need to run a separate Oban
instance per each dynamic repo instance. Most often it's worth bundling each
Oban and repo instance under the same supervisor:

```elixir
def start_repo_and_oban(instance_id) do
children = [
{MyDynamicRepo, name: nil, url: repo_url(instance_id)},
{Oban, name: instance_id, get_dynamic_repo: fn -> repo_pid(instance_id) end}
]

Supervisor.start_link(children, strategy: :one_for_one)
end
```

The function `repo_pid/1` must return the pid of the repo for the given
instance. You can use `Registry` to register the repo (for example in the repo's
`init/2` callback) and discover it.

If your application exclusively uses dynamic repositories and doesn't specify
all credentials upfront, you must implement an `init/1` callback in your Ecto
Repo. Doing so provides the Postgres notifier with the correct credentials on
init, allowing jobs to process as expected.

[dynamic]: https://hexdocs.pm/ecto/replicas-and-dynamic-repositories.html#dynamic-repositories

### Ecto Multi-tenancy

If you followed the Ecto guide on setting up multi-tenancy with foreign keys, you need to add an
exception for queries originating from Oban. All of Oban's queries have the custom option `oban:
true` to help you identify them in `prepare_query/3` or other instrumentation:

```elixir
# Sample code, only relevant if you followed the Ecto guide on multi tenancy with foreign keys.
defmodule MyApp.Repo do
use Ecto.Repo, otp_app: :my_app

require Ecto.Query

@impl true
def prepare_query(_operation, query, opts) do
cond do
opts[:skip_org_id] || opts[:schema_migration] || opts[:oban] ->
{query, opts}

org_id = opts[:org_id] ->
{Ecto.Query.where(query, org_id: ^org_id), opts}

true ->
raise "expected org_id or skip_org_id to be set"
end
end
end
```

<!-- MDOC -->
sorentwo marked this conversation as resolved.
Show resolved Hide resolved

## Community
Expand Down
Loading