How to monitor log agent status via Grafana for EKS log ingestions?

[daixba]

After I created log ingestion for EKS logs via the solution, I would like to monitor the log agent, which is fluent-bit, to make sure the agent is running as normal, and I would like to see the status via Grafana Dashboard.

[daixba]

Basically, the Fluent Bit agent can expose metrics in Prometheus format, and if you created the log ingestion via the solution, we have enabled the auto-discovery to Prometheus by default. So if you open the fluent-bit-logging.yaml generated by the solution, you should be able to see below:

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: fluent-bit
  namespace: logging
  labels:
    app.kubernetes.io/name: fluent-bit-logging
spec:
  ...
  template:
    metadata:
      ...
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/port: "2022"
        prometheus.io/path: /api/v1/metrics/prometheus

The minimal requirement is that you must deploy Prometheus in the same EKS cluster. You may have different prometheus server in a different location, such as if you are using Amazon Managed Service for Prometheus, you will still need to run prometheus in the same EKS cluster, then you can use the remoteWrite to send the metrics to your own Prometheus server.

Note: We assume you already know how to use Prometheus and Grafana, so this doesn't contain details of how to deploy and set them up. If you haven't known them well, you can follow some steps in https://archive.eksworkshop.com/intermediate/240_monitoring/

Once you have the Prometheus in the same EKS cluster, you can open the Prometheus console and see if the auto-discovery is working or not (Fluent Bit is in the target list or not).

kubectl --namespace=prometheus port-forward deploy/prometheus-server 8080:9090

Then open http://localhost:8080/targets in your browser, you should see something as below and the status is UP

Next step, you can then use the Prometheus as data source and then view the data in Grafana Dashboard. There is an example fluent-bit dashboard you can use. Check Example Dashbaord for more details.

Just for your easy access, you can download it via

wget https://raw.githubusercontent.com/fluent/fluent-bit-docs/8172a24d278539a1420036a9434e9f56d987a040/monitoring/dashboard.json

Then login to your Grafana Dashboard, and click 'Dashboard' > 'Import'. And then import the json file you just downloaded.

That is all. You are now able to monitor the Fluent-Bit agent status running in EKS. You can see how to input and output goes and whether or not there are some errors. You can also set up some alerts based on your needs.