Foucault03

Foucault03 is a anomaly log monitoring system.

Requirement

• Perl (> 5.20)
• Ansible

Description

Foucault03 system monitors logs treated by fluentd and tagged "multilinelog.**". The system detects anomaly logs defined by pre-generated patterns. Patterns are builded from sample logs and build rules. If you hope to monitor /var/log/messages, you may use /var/log/messages for a sample log as is.

Build rules may specify variable words in the logs by regexp, like following:

• \d+\.\d+\.\d+\.\d+ (IP address)
• (?:Jan|Feb|Mar|Apr|May|Jun|Jul|Aug|Sep|Oct|Nov|Dec)\x20[\x200-9][0-9]\x20[\x200-9][0-9]:[0-9][0-9]:[0-9][0-9] (timestamp)

Installation

1. git clone https://github.com/frisky-gh/foucault03.git
2. cd foucault03
3. sudo setup.sh
4. vi conf/fluentd.conf
5. ./bin/foucaultctl build_fluentd_conf
6. ./bin/foucaultctl build_patterns
7. /etc/init.d/td-agent restart

Synopsis

foucaultctl <SUBCOMMAND>

SUBCOMMAND is one of following:

build_fluentd_conf

Build a conf file for fluentd.

build_patterns

Build all pattern files which related to updated rules or sample file.

list_unmonitoredlog

List up all unmonitoredlogs.

capture_unmonitoredlog

Caputure unmonitoredlogs into capturedlogs.

capture_anomalylog

Caputure anomalylogs into capturedlogs.

show_capturedlog

Show all caputuredlogs.

strip_capturedlog

Strip redundant capturedlogs.

import_capturedlog

Append all capturedlogs into samples.

strip_samples

Strip redundant samples.

Files

conf/fluentd.conf

Configuration file for fluentd.

conf/fluentd.tt

Template file for a fluentd.conf.

conf/deliver.conf

Configuration file for report deliveries.

conf/deliver_flash.tt

Template file for a flash report of anomaly log by mail.

conf/deliver_daily.tt

conf/fluentd/fluentd_foucault03.conf

fluentd configuration file. It's included by /etc/td-agent/td-agent.conf.

conf/patterns/*.rules

Build rules file. You may customize it to adjust to your VMs.

conf/patterns/*.sample

Sample log file. You may put log file you want to target. Its size is hoped to be less than < 1MB.

conf/patterns/*.pattern

Pattern file. It's builded from a sample log and build rules, by `foucaultctl build_patterns`.

anomalylog/*

File of anomaly logs detected by foucault03.

unmonitoredlog/*

File of logs that is not monitored.

capturedlog/*

File of logs that is caputured from anomalylog or unmonitoredlog by 'capture_anomalylog' or 'capture_unmonitoredlog' subcommand.

deliveredevent/*

File of events that is delivered to recipients.

undeliveredevent/*

File of events file that is not delivered to any recipients.

Licence

MIT

Author

frisky-gh