This is a collection of logging configuration files made by Aaron for use with Solace PubSub+ event brokers and Solace Cloud services via Syslog forwarding. Currently includes:
- rsyslog rules (RainerScript)
- syslog-ng rules
- logstash grok rules (ELK / ElasticSearch)
- logrotate configuration
- cron example for log rotations
- (coming eventually) SEC (Simple Event Correlator) log correlation rules
- useful SEMP tool to grab logs remotely w/out Syslog config
Quickly deploy a logging server for Solace in AWS for free! Using one of the "free-tier" EC2 instances (e.g. t2.micro), and the pre-installed rsyslog.
- Boot / deploy an EC2 instance with Amazon Linux 2 AMI (HVM), t2.micro
- Edit the security group rules: add a TCP Custom rule for ports 51400-51422, from any Source: 0.0.0.0/0
- Once it's launched, take note of your EC2 instance Public IP address, we'll need this later
- Login to EC2 instance using your generated key:
ssh -i <keyfile> ec2-user@<pubic-ip-addr>
- Probably best practice to do a
sudo yum update
andsudo yum upgrade
- Probably best practice to do a
- Download this repo:
wget https://github.com/aaron-613/solace-logging-config/archive/master.zip -q
- Unzip it:
unzip master.zip; rm master.zip; cd solace-logging-config-master
- Copy (or symlink?) rules file to /etc/rsyslog.d/:
sudo cp solace_rsyslog.conf /etc/rsyslog.d/
- Restart rsyslog:
sudo systemctl restart rsyslog
Then, we need to configure the Solace broker.
- Login to Solace Cloud, Mission Control, Cluster Manager
- Select the broker / service you wish to add logging to, click on "Manage"
- Click on "Advanced Options" (top-right), "Syslog Forwarding": "Add"
- Give it a name (e.g. "external")
- Select all the Logs to forward
- Syslog Server hostname: enter the Public IP address from EC2 instance
- Port: 51400
- Protocol Type: TCP
- Login to CLI. Then:
enable
config
create syslog external
facility event
facility command
facility system
host <ec2-public-ip>:51400 transport tcp
exit
logging command all mode all-cmds
On your logging server, head over to /var/log/solace
, and you should start to see logs being populated there. Try connecting a client app or something to your broker, and you should see that echoed in realtime to the event.log
.
[ec2-user@ip-172-31-39-85 ~]$ cd /var/log/solace
[ec2-user@ip-172-31-39-85 solace]$ ls
sg-sol-3501-vmr ip-172-25-199-45
[ec2-user@ip-172-31-39-85 solace]$ cd sg-sol-3501-vmr
[ec2-user@ip-172-31-39-85 sg-sol-3501-vmr]$ ls -lh
total 15M
-rw-r--r--. 1 root root 77K Apr 19 01:50 auth.log
-rw-r--r--. 1 root root 9.7M Apr 19 00:55 command.log
-rw-r--r--. 1 root root 3.1M Apr 19 01:50 event.log
-rw-r--r--. 1 root root 79K Apr 19 01:51 show.log
-rw-r--r--. 1 root root 1.1M Apr 19 01:48 show.log.1
-rw-r--r--. 1 root root 2.2K Apr 19 01:06 show.log.2.gz
-rw-r--r--. 1 root root 2.2K Apr 19 01:04 show.log.3.gz
-rw-r--r--. 1 root root 22K Apr 19 00:55 system.log
drwxr-xr-x. 2 root root 23 Apr 18 19:47 bw
drwxr-xr-x. 2 root root 23 Apr 18 19:47 default
drwxr-xr-x. 2 root root 23 Apr 19 00:46 rest
[ec2-user@ip-172-31-39-85 sg-sol-3501-vmr]$
The 3 different rules configurations for the different Syslog engines do basically the same thing:
- Listen on 3 different inbound TCP ports, one each for
dev
,test
, andprod
, because that's awesome/advanced!- rsyslog: 51400 (dev), 51401 (test), 51402 (prod)
- syslog-ng: 51410 (dev), 51411 (test), 51412 (prod)
- logstash: 51420 (dev), 51421 (test), 51422 (prod)
- This way, I can add additional rules/processing/filtering later on depending on what port it is
- e.g. Don't log VPN Bridge UP/DOWN events to the alerts.log for dev dev environments
- e.g. Don't alert on SolCache DOWN events in dev if they come back up within 5 minutes
- This assumes a particular Solace router is designated as prod, test, or dev. Maybe change to prod/non-prod?
- Ability to process all 3 log facilities arriving from Solace, but does some filtering. For each router, it creates a directory
for it prefixed by
router.
. Then inside each:system.log
: this is essentially left alone, but I think this log is dumb as it's a subset ofevent.log
command.log
: all the commands, but filtering out all the 'show' commands (which can be exported to command via the CLI commanden --> con --> logging command all mode all
. Very useful for watching SEMP monitoring polling rate.show.log
: all of the show commands get put into a separate file to not junk up thecommand.log
event.log
: all of the router's event logs, minus a couple AUTH logs that are constantly generated by SEMP monitoring appsauth.log
: the events corresponding to CLI/SEMP logins/lougouts- per-VPN event logging (optional): for VPN and CLIENT events, parse out the VPN name... this allows me to create per-VPN directories, and keep an event log only for that VPN. This is very useful if other VPNs on the same appliance are very noisy and cause the main event log to roll quickly.
- Then I monitor the event logs for events of interest that are duplicated to a general
alerts.log
file. This file would ideally be watched by a monitoring program, rather than having the monitoring program trying to watch every router's complete event log.
Make sure your logs don't take up all your disk space! There are some included files to make sure you rotate your log files, and check them periodically.
- Copy the
solace_logrorate
file into your/etc/logrotate.d/
directory - Copy the
solace_cron
file into/etc/cron.d
directory- Or edit your crontab (
crontab -e
) and included the one-liner in there
- Or edit your crontab (
*Make sure you edit/verify the paths, as specified in the files
You might need EPEL installed first.
On AWS Linux, you'll need to install EPEL (extrams): sudo amazon-linux-extras install epel -y
Then you can copy the syslog_ng conf file into /etc/syslog-ng/conf.d
--or-- replace /etc/syslog-ng/syslog-ng.conf
.
https://www.elastic.co/guide/en/logstash/current/installing-logstash.html