Journald to AWS cloudwatch log shipper. Easy as pie 🥧
Deprecated. Just use https://vector.dev/ -- it's simpler and more flexible.
Cloudjournal will use the machine's AWS credentials. The AWS docs suggest managing your credentials using a file called 'credentials' -- like /root/.aws/credentials
(more information on the AWS documentation site). It should include the cloudjournal
profile and it should look something like this:
[cloudjournal]
aws_access_key_id = AWS_ACCESS_KEY_ID_HERE
aws_secret_access_key = aws_secret_access_key_here
The credentials for the cloudjournal
profile should have the following AWS permissions:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"logs:CreateLogGroup",
"logs:CreateLogStream",
"logs:DescribeLogGroups",
"logs:DescribeLogStreams",
"logs:PutLogEvents",
"logs:GetLogEvents",
"logs:FilterLogEvents"
],
"Resource": "*"
}
]
}
Get the latest .deb package for your architecture here: https://github.com/danesparza/cloudjournal/releases/latest
Install it using
sudo dpkg -i cloudjournal-1.0.45_armhf.deb
Configuration is done via /etc/cloudjournal/config.yaml. Here is an example configuration file:
server:
port: 3010
allowed-origins: "*"
datastore:
system: /var/lib/cloudjournal/db/system.db
log:
level: info
cloudwatch:
region: "us-east-1"
group: "/app/cloudjournal/{unit}"
stream: "{machineid}"
monitor:
units: daydash, avahi-daemon
interval: 10
server
indicates where a runtime diagnostic interface is hosted. It may be removed
datastore
is where state information is stored for cloudjournal. Defaults to ~/cloudjournal/db
log.level
can be debug, info, warn, error -- and it corresponds to your desired level of log verbosity. Defaults to info
cloudwatch.region
is the region you would like to log events to. Your credentials should be for this region. Defaults to us-east-1
cloudwatch.group
is the log group name to use. Both groups and streams can have tokens in their name. Defaults to /app/cloudjournal/{unit}
cloudwatch.stream
is the log stream name to use. Both groups and streams can have tokens in their name. Defaults to {hostname}
monitor.units
is a comma seperated list of units to monitor and sent to AWS Cloudwatch. required
monitor.interval
is the number of minutes to wait between log batches. Defaults to 1
There are several tokens you can use when naming cloudwatch.group
or cloudwatch.stream
:
{hostname}
- This will be replaced with the contents of /etc/hostname
{machineid}
- This will be replaced with the contents of /etc/machine-id
{unit}
- This will be replaced with the name of the current systemd unit being processed.
Getting your app log to cloudwatch is simple now: If your app is installed as a systemd unit, just output your logs to the console -- they'll automatically be added to journald under your systemd unit. Then cloudjournal can take the logs for your journald unit and ship them to cloudwatch every few minutes.
JSON logging is highly recommended because AWS Cloudwatch can automatically parse JSON logs and will provide structured log filters and searching.