Provides a simple to use framework to write checkers for a Rails app. The intention is to run regular checks in production to assure for example:
- data consistency (e.g. accounting)
- external APIs are healthy
- general assumptions about the app like user's are signing up every day, or perform certain actions in the app
This will help to catch issues in production early.
This is neither a replacement for tests nor for error handling, but rather extends an app with active monitoring of functionality and data consistency.
- Assume, your sign up/log in depends on an external OAuth provider. For what ever reason, your API key for this OAuth provider expired, or you forgot to configure it in production. It might take a while for you to figure this one out (since locally and on your staging this could work as it uses different API keys, and different credentials/ENV vars). One possibility for a checker is to verify, that you had at least one successful login (or sign up) on your app in the past 24 hours, and run this checker daily. If that checker fails, you know you must investivate the login and sign up functionality. Keeping a checker more general helps you catch more errors; No sign ups could for example also be rooted in a fault outside your app, like marketing could have sent out a wrong link in the last promotional email, or your loadbalancer is misconfigured, etc.
- Assume, your app depends on an external API for any e.g. data processing or part of a user flow (e.g. to fetch currency exchange rates, book a ticket, etc.). You likely test this feature against some sandbox environment (we assume, this test is passed), and you also test it after your first production release. Sadly, in our experience, external APIs sometimes introduce breaking changes, or they go offline for a longer period, or your IP has changed and you forgot to allowlist your new IP. When your app is large enough, you cannot possibly check all those things regularly manually. Write a checker that pings said external API once a day, or once a week, or once every hour, depending on your business needs. Know you can rest assured, that you'll be notified of any fault as soon as possible.
- Assume your application handles payment in any form, and stores these in a
transactions
table. Payments depends on many working pieces, like 3DS secure, external providers, correct CORS settings, etc. You could add a checker, that checks if you got at least one transaction record every hour/day/week to catch issues with your payment system/provider as early as possible.
Find an example checker as code below.
If you wish, you can write runbooks on how to handle each failed checker, and pass the URL to your runbook to the checker; this URL will then be included in the error messsge (i.e. for Slack, you can click on the failure notification and will be redirected to the correct runbook).
Tested against Rails 6
Tested against Ruby 2.6 - 3.0 (see .github/workflows/test.yml).
Contributions are welcome 🙂
Please check the open Issues (or open one if you find any!).
Please check open TODOs.
Just open a PR. If you are unsure about it, open an issue first (or comment on an existing one).
Install the gem
gem 'dbl-checker'
# with version, pointing to github (until the gem was released to Rubygems.org)
gem 'dbl-checker', '~> 0.5', git: 'git@github.com:dbl-works/checker'
This gem providers a generator
rails generate dbl_checker:install
which will create
- a model file at
app/models/dbl_check.rb
- a migration file
db/migrate/#{migration_version}_create_dbl_checks.rb
- a config file at
config/initializers/dbl_checker.rb
Checkers are expected to live under app/checkers/*_checker.rb
A simple checker may look like the following:
# app/checkers/transaction_checker.rb
class TransactionChecker
include DBLChecker::Job
check_options(
every: 4.hours,
description: 'Check transactions exist and are zero in sum',
name: 'transactions_checker',
sla: 1.day,
aggregate_failures: true,
runbook: 'notion.so/dbl/checkers/runbooks/project-name/transactions',
)
def perform
# Account balance must be zero at any time, because...
assert(transactions_sum.zero?, "Expected transactions sum to be 0, got #{transactions_sum} instead.")
assert(transactions_past_24h.count.positive?, 'Expected transactions to exist, but no records were persisted during the past 24 hours.')
end
private
def transactions_past_24h
Transaction.where('created_at >= ?', 24.hours.ago)
end
def transactions_sum
Transaction.sum(:amount_base_currency)
end
end
Options that can be configured per checker. You can set global defaults in the initializer as config.default_check_options
.
When using Rails, ensure to inflect on DBL
for this to work with Rails' autoloader/Zeitwerk.
# config/initializers/inflections.rb
ActiveSupport::Inflector.inflections(:en) do |inflect|
inflect.acronym 'DBL'
end
See the following config example; all options are optional.
# config/initializers/dbl_checker.rb
DBLChecker.configure do |config|
# @NOTE: for now, we cannot load Rails here due to the way we load the config in the dbl-checker process (outside Rails)
# config.slack_webhook_url = Rails.application.credentials.dig(:slack, :checkers_endpoint) # e.g. https://hooks.slack.com/services/XXX
config.slack_webhook_url = ENV['CHECKERS_SLACK_WEBHOOK_ENDPOINT'] # e.g. https://hooks.slack.com/services/XXX
config.app_version = ENV['COMMIT_HASH'] # let's you pin-point each checker-execution to a certain version of your app
config.dbl_checker_api_key = 'some-token' # API key for the DBLCheckerPlatform adapter
config.default_check_options = {
every: 24.hours, # how often a check is performed
sla: 3.days, # your commitment to resolve failed checks. Purely cosmetics
active: true, # e.g. set this to false outside production to not perform checks
slack_channel: 'checkers', # must set the persistence adapter to "Slack" (DBLCheckerPlatform can also publish to Slack)
timeout_in_seconds: 30, # If a checker hasn't finished after the given time, it is killed. This check counts as failed
aggregate_failures: false, # exit checker after the first assertion fails. Set to true to aggregate all failures
runbook: nil, # which runbook shall be displayed on failure that helps engineers resolve the issue
}
# an adapter class is expected to either be a singleton or a regular class
# internally, this gem will attempt to call ".instance" or ".new" on the class
# then the method `.call` is executed.
# The "Local" resolves require a `DBLCheck` model to exist, that inherites from ActiveRecord.
# Use the install script to scaffold a migration file.
config.adapters = {
# other adapters: `:slack`, `:mock`, `:local`
# either pass a symbol for build-in adapters, or a class/module; anything that has a #call/.call method, you can also use a Singelton
# :slack will only send messages if the check fails
persistance: %i[slack local],
# other adapters: `Mock`, `Local`
# the call method expects 0 arguments
job_executions: :local,
}
end
# spec/support/dbl_checker.rb
# you could also stub the adapters's "#call" method
# allow(DBLChecker::Adapters::Persistance::Local.instance).to receive(:call)
DBLChecker.configure do |config|
config.adapters = {
persistance: :mock,
job_executions: :mock,
}
end
To understand the structure of the used template, a simple example Slack template is provided. You may configure your own own one using the Slack Kite Builder (this link leads you to the Kite Builder with the default template of this gem prefilled in case you want some boilerplate to start).
If you want to use a custom Slack template, overwrite the existing Slack-Adapter accordingly.
You can pass any class/model that in one way or another implements a call
methods.
This gem will figure out, wether to instantiate your passed object (e.g. by calling .new
or .instance
on it), or if you passed a class/module that exposes a .call
class-method.
Persistance: The call
method expects one argument that is an instance of a Check
class. The aim is to persist this in some form (e.g. by sending it to Slack, another API, or an internal database). By default, this will send a notification to slack and persist a record in your local database.
JobExecutions: The call
method expects no arguments. It returns an array of checker-class names mapped to their last execution time. The checker service uses this to determin which checkers have to run (remember, each checker can define how often it has to be execute). By default, this queries your local database.
All errors happening within this gem are wrapped in one of these custom error classes:
DBLChecker::Errors::AssertionFailedError
-> an assertion in your check failed. This is swallowed and errors are written sent to the persistance layerDBLChecker::Errors::ConfigError
-> e.g. invalid or missing configurationDBLChecker::Errors::ServerError
-> cannot communicate with external server (e.g. Slack)
You can also handle all errors at once, because all errors inherit from DBLChecker::Errors::DBLCheckerError
(which inherits from StandardError
).
- must set ENV var
RAILS_ENV
- Run
bin/dbl-checker -c path/to/config/file
to launch the client process and the/healthz
TCP server - [optional]] ENV var
DBL_CHECKER_API_KEY
to persist jobs remotely - [optional] set
DBL_CHECKER_HEALTHZ_PORT
, defaults to3000
Run for example:
bundle exec dbl-checker -e production
# --locally--
bundle exec dbl-checker \
--environment development \
--config ".../config/initializers/dbl_checker" \
--inflections ".../config/initializers/inflections"
👉  replace ...
with whatever path you have locally
CLI args:
--environment
(short:-e
): set to same asRAILS_ENV
/RACK_ENV
, used for logging and passed to notifications--inflections
(short:-i
): path to the Rails inflections file, default:/app/config/initializers/inflections
--config
(short:-c
): path to the gem's initializer, default:/app/config/initializers/dbl_checker
--help
prints out all options; see this file for how they are implemented
You can check the current version:
bundle exec dbl-checker -v
# --OR--
bundle exec dbl-checker --version
build the latest version of the gem locally with
gem build dbl-checker.gemspec
take note of the most current version of the gem, which will be printed to console.
You can use that in your Gemfile passing a local path:
gem 'dbl-checker', path: '~/Sites/dbl-works/checker'