Application Monitoring with Prometheus

Setting up Kubernetes Cluster

• swapoff -a
• setenforce 0
• Disable SE linux vi /etc/selinux/config cat < /etc/yum.repos.d/kubernetes.repo [kubernetes] name=Kubernetes baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64 enabled=1 gpgcheck=1 repo_gpgcheck=1 gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg EOF
• sudo yum install -y kubelet kubeadm kubectl
• systemctl start kubelet && systemctl enable kubelet
• cat < /etc/sysctl.d/k8s.conf

net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF

• sysctl --system
• Copy paste this token to worker nodes -> kubeadm join tokengenerated at master node
  
• systemctl restart kubelet

Setting up Prometheus

use script install_prometheus.sh

Install node exporter in worker nodes

use script install_node_exporter.sh

Install grafana

use script install_grafana.sh Create data source -> Prometheus

Recording rules

Recording rules allows us to precompute frequently needed or computationally expensive expressions and save their result as a new set of time series.
create a prometheus_rules.yml containing the necessary rule statements and then have Prometheus load the file via the rule_files field in the prometheus configuration i.e., prometheus.yml.

Alertmanager

The AlertManager uses a configuration file named alertmanager.yml
we create the AlertManager systemd service
Then reload the daemon and start the alertmanager service

This is how the Alerts will be visible on Prometheus Dashboard.

This is how alert will come on alertmanager dashboard

We can configure the notification part i.e. where we want to get alert notification, Slack, Email, Pagerduty etc. In this case, alert notification will come on Slack alerts channel.

After the problem causing alert is being resolved.