Fluent Bit v0.11 was just released at the end of March 2017 and it comes with support for Kubernetes logging needs: it can consume logs and enrich them with proper Kubernetes metadata, retrieved from the API server of course.
Our goal is to make logging cheaper in terms of memory consumption. In order to accomplish this we need to deal with different factors, not only processing incoming logs but also formatting the data for our backends like Elasticsearch, which can be a little expensive as it requires a JSON representation.
Note: Fluent Bit uses a binary representation for logs, converting this to JSON requires enough memory for the process.
Fluent Bit must be deployed as a DaemonSet, on that way it will be available on every node of your Kubernetes cluster.
This repository contains two Yaml DaemonSet files:
| Yaml file | Description |
|---|---|
| fluent-bit-daemonset-elasticsearch | deploys a stable release of Fluent Bit. |
The current DaemonSet points to this specific Docker Hub image:
0.11 fluent/fluent-bit-kubernetes-daemonset:0.11
-
Make sure your Elasticsearch backend is running and can be reach through the hostname elasticsearch-logging. This value can be changed in the Yaml file
-
Deploy the daemonset file from this repository:
$ kubectl apply -f fluent-bit-daemonset-elasticsearch.yamlThe default configuration of Fluent Bit makes sure of the following:
- Consume all containers logs from the running Node.
- The Tail input plugin will not append more than 5MB into the engine until they are flushed to the Elasticsearch backend. This limit aims to provide a workaround for backpressure scenarios.
- The Kubernetes filter will enrigh the logs with Kubernetes metadata, specifically labels and annotations. The filter only goes to the API Server when it cannot find the cached info, otherwise it uses the cache.
- The default backend in the configuration is Elasticsearch set by the Elasticsearch Ouput Plugin. It uses the Logstash format to ingest the logs. If you need a different Index and Type, please refer to the plugin option and do your own adjustments.
- There is an option called Retry_Limit set to False, that means if Fluent Bit cannot flush the records to Elasticsearch it will re-try indefinitely until it succeed.
Your contribution on testing is highly appreciated, we aim to make logging cheaper for everybody so your feedback is fundamental, try to get back to us on: