A PHP web app to monitor & alert cron jobs using a simple http api. It allows users to create job entries with given periods and grace windows. Then their scheduled tasks can do a simple HTTP GET to the unique URI generated each time they run. If a job doesn't call in within the given grace window then an alert is sent to the user.
Users can silence particular jobs (maybe a server is going offline for maintenence) or can silence their whole account (maybe they are going on holiday or know all their servers will be offline for a power-cut etc).
A sysadmin can also silence the whole site - which can be useful if there's going to be a widespread maintenence period, for instance. That way jobs are still recorded, just no alerts are sent. Sysadmins can also view/edit any job or user on the system which can be useful if a job starts going 'alert crazy'.
Users can also over-ride the email address for particular jobs. So maybe a specific backup job goes to person X, a particular facility/office jobs should go to a mailing list, etc. You can also have a 'fallback' email address which will start to get notified if a job has been alerting for a long while.
Users can create teams and allocate jobs/users to the teams. This allows other members of the team to view & edit any jobs which are in their team(s).
HTTP "Pings" can also pass a "data" field to record extra information against the record. This can be used to record, for instance, an exit code or amount of data transferred.
You can also create/update jobs via a simple HTTP API.
You can create 'templates' for jobs and then use those to create a new job via an HTTP call with all the options set to match the template.
See the Laravel requirements page. You also need a valid mail server to send the alerts. You will probably need an SQL server - although you can use SQLite if you have a small install.
If you have docker available you should be able to run :
export CRONMON_ADMIN_USERNAME=whatever
export CRONMON_ADMIN_EMAIL=whatever@example.com
export CRONMON_ADMIN_PASSWORD=mysecretpassword
docker-compose up
Then the test app will be available at http://localhost:3000/ and all email is redirected to a copy of Mailhog running at http://localhost:3025/.
Clone the repository into a new folder. Then in the new directory run :
cp .env.example .env
composer install
php artisan key:generate
If you don't have composer already then you can download it from https://getcomposer.org/ .
You will also need to point a webserver vhost with ${your_directory}/public as it's document root.
Then edit the .env file to suit your environment (email server etc). Then run the database migrations to create the tables :
php artisan migrate
Once you are ready you can create an initial admin account by running :
php artisan cronmon:createadmin username your-email@whatever.com
You should be sent a password reset link by email which will let you set a password and log in.
You should be able to run this inside swarm/kubernetes without too much problem. There is an example stack file for Docker swarm as prod-stack.yml. To go through the steps manually :
export PHP_VERSION=7.3
export IMAGE_NAME=cronmon
export DOTENV_NAME=cronmon-dotenv-1
cat .env | docker secret create ${DOTENV_NAME} -
docker build --target=prod -t $IMAGE_NAME .
docker stack deploy -c prod-stack.yml cronmon
If you want to automatically create the initial admin user you can either use environment variables or secrets. The enviroment variables it looks for are :
CRONMON_ADMIN_USERNAME=whatever
CRONMON_ADMIN_EMAIL=whatever@example.com
CRONMON_ADMIN_PASSWORD=mysecretpassword
or to use files (eg, Docker secrets) :
# docker secret create cronmon_username_1 "whatever"
# docker secret create cronmon_email_1 "whatever@example.com"
# docker secret create cronmon_password_1 "mysecretpassword"
CRONMON_ADMIN_USERNAME_FILE=/run/secrets/cronmon_username_1
CRONMON_ADMIN_EMAIL_FILE=/run/secrets/cronmon_email_1
CRONMON_ADMIN_PASSWORD_FILE=/run/secrets/cronmon_password_1
If you haved pulled down an updated version (by eg, git pull origin master) then you probably want to run :
composer update
php artisan config:clear
php artisan cache:clear
php artisan view:clear
php artisan migrate
That will clear out most of the cached/pre-compiled bits from the older copy and add any new fields to the database.
To manually run the checks for the jobs in the system you can call :
php artisan cronmon:checkjobs
(Tip: if you're just testing things out you can set your .env 'MAIL_DRIVER=log' - then the email is written to 'storage/logs/laravel.log' rather than via smtp)
In production you can set up a cron job to call laravel's scheduler which will then call the cronmon:checkjobs command
itself. See the laravel docs for instructions. By default the
checks are run every five minutes. If you want to change that then edit app/Console/Kernel.php in the schedule()
method. Again the laravel docs have some guidence for what you can do.
By default the alerts will only be sent to the user every 60 minutes. That's so they don't get bombarded with notifications every five minutes about a job that's failed to run. If you want to alter that you can change the 'alert_interval' value in config/cronmon.php. You can also change the prefix of the email alert there too - by default it's '[CRONMON]'.
In that config file you can also change the default number of hours a job can fail for before it is sent to any 'fallback' email address for the job. This is useful when, for instance, the primary user should get notified right away, but the 'fallback' will start to get notified if they haven't dealt with things after some time. The default is 24hrs.
If you want to change things about the actual email that is sent, have a look at app/Notifications/JobHasGoneAwol.php.
Also have a read of the the laravel notification docs.
You can manually truncate the logs for each job 'ping' (ie, each hit from the http call) by running :
php artisan cronmon:truncatepings
If you enable the scheduler as detailed above then this is automatically run once a week. By default it will keep 100 records for each job. You can alter that by changing 'keep_pings' in 'config/cronmon.php'.
To actually notify the app that a job has run, you sent a straight http get to it's URI (shown in the app). For
instance, you create a job that shows it's URI as 'http://cronmon.dev:8000/ping/c6ad9b87-f945-440e-bc90-c0bc8794f52a'.
At the end of a bash script you could use curl to 'ping' the app which will update it's status. A very basic
example might be :
#!/bin/bash
# basic exit on error
set -e
# do something
rsync -a /home/me /mnt/usb/backup
# now ping the app to let it know we ran ok
curl -s http://cronmon.dev:8000/ping/c6ad9b87-f945-440e-bc90-c0bc8794f52a
When you are viewing a particular job in the app it will show some copy'n'paste examples for pinging it from a few common scripting languages too.
If you want to send extra data along with your check, pass a "data" field with the information, eg :
curl -s http://cronmon.dev:8000/ping/c6ad9b87-f945-440e-bc90-c0bc8794f52a?data=exitcode=44
curl -X POST -F "data=exitcode=44" -s http://cronmon.dev:8000/ping/c6ad9b87-f945-440e-bc90-c0bc8794f52a
Users can create an API key in their 'My Account' page (pick Edit -> tick 'generate new api key'). They can then use that key to create or update a cron job in the system. Eg :
curl -X POST -H "Accept: application/json" -H "Content-Type: application/json" \
--data '{"name": "my new job", "schedule": "1 1 * * *", "api_key": "saldkj13240123nesdjna"}' \
http://cronmon.dev/api/cronjob
The parameters you can set are :
"name" - name of the job (existing or create a new one)
"schedule" - the cron-format you want to use
"team" - name of the team to associate with this job
"grace" - amount of time for the grace period (number)
"grace_units" - the units the grace period is set ('minute' ,'hour', 'day', 'week')
"period" - the frequency of the job (if you are not setting the 'schedule') - number
"period_units" - units for the frequency ('minute' ,'hour', 'day', 'week')
You must pass a name and your API key and either a schedule or the period & period_units.
In the main UI you can create a template and fill in the options you want. It will then show you a URL that you can make a POST request to - a new job will be created and it's details will be returned to you as a JSON object. Eg:
curl -X POST http://cronmon.dev:8000/api/templates/3-monthly-windows-backupand that would return :
{
"data": {
"id": 8,
"is_awol": false,
"show_url": "http://cronmon.dev:8000/job/8",
"user_url": "http://cronmon.dev:8000/admin/user/1",
"username": "admin",
"teamname": "None",
"uri": "http://cronmon.dev:8000/ping/a2a80167-b1a9-4913-a0f0-4cba1e837b1f",
"is_silenced": false,
"name": "Monthly Windows Backup Job 27/03/2020 12:45",
"schedule": "Every Week",
"last_run": "27/03/2020 12:45",
"last_run_diff": "1 second ago"
}
}You can get the details of a given job by just making a call to :
curl http://cronmon.dev:8000/api/cronjob/a2a80167-b1a9-4913-a0f0-4cba1e837b1fIt will return a JSON blob as above.
This can be useful if you know there will be a long maintenance window or a network outage for instance. To silence or un-silence the whole site you can run :
php artisan cronmon:silence
php artisan cronmon:unsilence
You can run the test suite by running vendor/bin/phpunit. It should cover most of the functionality.
There are quite a few SaaS which offer this kind of service. But we needed something on-site as a large percentage of our servers aren't internet-facing, so a 'cloud' SaaS was fairly useless to us.
Most of the on-site/opensource packages I could find were either in languages / frameworks that I'm not hugely familiar with (certainly not to the extent I could confidently make needed modifications) or used a specific DB backend which we don't currently run.
So I guess you could look on this as ideal if you're after an on-prem, relatively database-agnostic and can rustle up a server which can serve PHP.
- Do something meaningful with the 'ping' logs for jobs.
- DONE Possibly create an API for creating/editing jobs so it could be automated as part of a provisioning system, for instance.