This build will setup ELK stack, and bring you dashboards in order to visualize ELK logs as a first example. You can run on the vagrant provided, or on a test host, where logs from already running docker containers will automatically be forwarded to ELK.
One Docker Compose will run:
- ElasticSearch 1.7 (+data container)
- Logstash 1.5.3 (+conf for elk logs)
- Kibana 4 (+Dashboard for elk logs)
- cAdvisor (Collect & View containers performance)
- Ngnix Proxy 1.9.3 (for SSL + password access).
--> want to use the new version 5 of Elastic & kibana, have a look here: https://github.com/gregbkr/elk-dashboard-v5-docker
git clone https://github.com/gregbkr/docker-elk-cadvisor-dashboards
mv docker-elk-cadvisor-dashboards elk
cd elk
(you probably get the error with port 80, see error section below)
docker-compose up -d
Wait 5 min Deploy all init:
./deploy-init.sh
OR one by one:
Configure Kibana index pattern:
es-conf/indexpattern-init.sh
Configure storage for backup:
es-conf/backup-init.sh
Deploy fake logs (good to test Web dashboard for IP map):
kibana-conf/deploy-fake-logs.sh
(If script failed configure manually the index pattern: in Setting > Indice : tick both boxes and select @timestamp)
http://dev.local:5601 (direct without proxy)
https://dev.local:5600 (!HTTPS ONLY! enter the credentials admin/Kibana05)
Setting > Object > Import > /kibana-conf/export_vX.json
If some dashboards do not display well, need to wait 2 min for the data to come in, then refresh the fields in order to force ELK to initialize them now:
Setting > Indices > [logstash-]YYYY.MM.DD > Reload field list (the circle arrow in orange)
cAdvisor will catch containers metrics running on your docker host, and then send it to ELK. cAdvisor should already be running through docker-compose.
You can run other instance, on another host, via this standalone container: (if on another host, you will have to correct storage_driver_es_host)
docker run \
--volume=/:/rootfs:ro \
--volume=/var/run:/var/run:rw \
--volume=/sys:/sys:ro \
--volume=/var/lib/docker/:/var/lib/docker:ro \
--publish=8888:8080 \
--detach=true \
--name=cadvisor \
--link=elk_elasticsearch_1:elasticsearch \
google/cadvisor:latest \
-storage_driver="elasticsearch" -alsologtostderr=true -storage_driver_es_host="http://elk_elasticsearch_1:9200"
You should already have some data in the dashboard cAdvisor. If not, please check index-mgmt.md setting index-pattern, or template (raw field)
#---------------------- Config -------------------------------
You can run logstash to easily collect input string. Each input you paste in the invite will be process and output on screen by logstash. To setup, please run this container:
docker run -it --rm --name logstash -p 5001:5000 -p 5001:5000/udp -v $PWD/logstash-conf:/opt/logstash/conf.d logstash:1.5.3 -f /opt/logstash/conf.d/logstash-test.conf
And paste log messages like this and check if the output is correct. Make modification to logstash-test.conf and restart logstash container to refresh.
<14>2015-08-31T15:20:00 vagrant-ubuntu-trusty-64 docker/proxyelk[870]: 2015/07/20 17:19:13 routing all to syslog://dev.local:5000
If you got the field tags = ParseFailure, means your parsing is wrong somewhere... :-(
You can use https://grokdebug.herokuapp.com/ in order to check a log parsing.
#------ Backup and Restore and optimize ------
Configure backup storage : (done in initialization step - in our case, backups will go in the $PWD/backup local folder)
es-conf/backup-init.sh
Edit as you wish (correct with you --host IP) and set cron job for backup, restore, index rotation:
crontab -l | cat es-conf/backup.crontab | crontab -
You can now monitor the backup via CURATOR dashboard
More info in file: index-mgmt.md
#---------------------- Stop and refresh -------------------
docker-compose stop
docker restart elk_logstash_1
#--------------------- TODO NEXT --------------------------
- cAdvisor stats: correct some bugs and add more graphs.
- Backup and restore: tests full recovery
- Docker daemon logs to listen to container crashes
- Event alerting (via email)
#---------------------- Errors ----------------------------------
listen tcp 0.0.0.0:80: bind: address already in use: --> stop nginx service:
sudo service nginx stop
Checks logs on compose and container:
docker-compose logs
docker log -f elk_logstash_1
#--------------------Prerequisite -----------------------------
a. Use the vagrantfile provided in the root in order to have a configured environment with docker and compose already up to date.
b. In the whole page we use dev.local = your_host_ip = where the containers run. Please use your IP or map it in your local hosts file (laptop and vagrant VM).
ip a | grep 'scope global eth'
For Linux/MAC:
sudo nano /etc/hosts
For Windows (copy file to desktop, edit, and copy back ;-):
c:\windows\system32\etc\driver\hosts
wget -qO- https://get.docker.com/ | sed 's/lxc-docker/lxc-docker-1.8.1/' | sh
gpasswd -a vagrant docker
service docker restart
sudo sh -c "curl -L https://github.com/docker/compose/releases/download/1.4.0/docker-compose-`uname -s`-`uname -m` > /usr/local/bin/docker-compose"
sudo chmod +x /usr/local/bin/docker-compose
sudo sh -c "curl -L https://raw.githubusercontent.com/docker/compose/1.4.0/contrib/completion/bash/docker-compose > /etc/bash_completion.d/docker-compose"
Be carefull if you have firewall activated on the docker host. PLease follow that config:
start docker with --iptables=false : edit DOCKER_OPTS="--iptables=false"
sudo nano /etc/default/docker
sudo restart docker
FORWARD chain policy set to ACCEPT, edit DEFAULT_FORWARD_POLICY="ACCEPT"
sudo nano /etc/default/ufw
sudo ufw reload
Add the following NAT rule to allow outgoing connection from docker network:
iptables -t nat -A POSTROUTING ! -o docker0 -s 172.17.0.0/16 -j MASQUERADE
#-------------------- MISC -----------------------------
(height: 45px; width: 252px)
docker cp kibana-conf/no_border.png elk_kibana_1:/opt/kibana/s rc/public/images
Treesize: install and check your disk
apt-get install ncdu
ncdu /
Size of one path:
du -hs /path
Find all our data container:
docker ps -a | grep 'data'
Check your db size:
docker exec elk_elasticsearch_1 du -hs /var/lib/elasticsearch/data
BACKUP : Make a backup of your datacontainers: copy datacontainer data --> host:
cd to/your/backup/folder
docker run --rm --volumes-from elk_elasticdata_1 -v $PWD:/backup ubuntu tar cvf /backup/datacontainer-var-lib-elasticsearch-data_`date +%d-%m-%Y"_"%Hh%Mm%Ss`.tar /var/lib/elasticsearch/data
LIST dangling/orphan images:
docker images -f dangling=true | awk '{if(NR>1) print $0}'
DELETE dangling/orphan image: (not use or orphan images)
docker images -f dangling=true | awk '{if(NR>1) print $3}' | xargs docker rmi
LIST only created/exited weeks ago (and NOT minute/day), and NOT data container (container_name NOT contains "data") (please double check if no datacontainer inside that list)
docker ps -a -f status=exited | grep -v 'data'| grep -v 'minute' | grep -v 'day' | grep week
DELETE this list above:
docker ps -a -f status=exited | grep -v 'data'| grep -v 'minute' | grep -v 'day' | grep week | awk '{if(NR>1) print $1}' | xargs -r docker rm
LIST dead container:
docker ps -a -f status=dead | grep -v 'data'
DELETE dead container:
docker ps -a -f status=dead | grep -v 'data' | awk '{if(NR>1) print $1}' | xargs -r docker rm