Fluent Bit Operator facilitates the deployment of Fluent Bit and provides great flexibility in building a logging layer based on Fluent Bit.
Once installed, the Fluent Bit Operator provides the following features:
- Fluent Bit Management: Deploy and destroy Fluent Bit DaemonSet automatically.
- Custom Configuration: Select input/filter/output plugins via labels.
- Dynamic Reloading: Update configuration without rebooting Fluent Bit pods.
- Overview
- Get Started
- Monitoring
- API Doc
- Best Practice
- Custom Parser
- Roadmap
- Development
- Contributing
Fluent Bit Operator defines five custom resources using CustomResourceDefinition (CRD):
FluentBit: Defines the Fluent Bit DaemonSet and its configs. A custom Fluent Bit imagekubesphere/fluent-bitis required to work with FluentBit Operator for dynamic configuration reloading.FluentBitConfig: Select input/filter/output plugins and generates the final config into a Secret.Input: Defines input config sections.Parser: Defines parser config sections.Filter: Defines filter config sections.Output: Defines output config sections.
Each Input, Parser, Filter, Output represents a Fluent Bit config section, which are selected by FluentBitConfig via label selectors. The operator watches those objects, constructs the final config, and finally creates a Secret to store the config. This secret will be mounted into the Fluent Bit DaemonSet. The entire workflow looks like below:
To enable fluent-bit to pick up and use the latest config whenever the fluent-bit config changes, a wrapper called fluent-bit watcher is added to restart the fluent-bit process as soon as fluent-bit config changes are detected. This way the fluent-bit pod needn't be restarted to reload the new config. The fluent-bit config is reloaded in this way because there is no reload interface in fluent-bit itself. Please refer to this known issue for more details.
Kubernetes v1.16.13+ is necessary for running Fluent Bit Operator.
Install the latest stable version
kubectl apply -f https://raw.githubusercontent.com/kubesphere/fluentbit-operator/release-0.12/manifests/setup/setup.yaml
# You can change the namespace in manifests/setup/kustomization.yaml in corresponding release branch
# and then use command below to install to another namespace
# kubectl kustomize manifests/setup/ | kubectl apply -f -Install the development version
kubectl apply -f https://raw.githubusercontent.com/kubesphere/fluentbit-operator/master/manifests/setup/setup.yaml
# You can change the namespace in manifests/setup/kustomization.yaml
# and then use command below to install to another namespace
# kubectl kustomize manifests/setup/ | kubectl apply -f -Note: For the Helm-based installation you need Helm v3.2.1 or later.
Fluent Bit Operator supports docker as well as containerd and CRI-O. containerd and CRI-O use the CRI Log format which is slightly different and requires additional parsing to parse JSON application logs. You should set different containerRuntime depending on your container runtime.
If your container runtime is docker
helm install fluentbit-operator --create-namespace -n kubesphere-logging-system charts/fluentbit-operator/ --set containerRuntime=dockerIf your container runtime is containerd
helm install fluentbit-operator --create-namespace -n kubesphere-logging-system charts/fluentbit-operator/ --set containerRuntime=containerdIf your container runtime is cri-o
helm install fluentbit-operator --create-namespace -n kubesphere-logging-system charts/fluentbit-operator/ --set containerRuntime=crioThe quick start instructs you to deploy fluent bit with dummy as input and stdout as output, which is equivalent to execute the binary with fluent-bit -i dummy -o stdout.
kubectl apply -f https://raw.githubusercontent.com/kubesphere/fluentbit-operator/master/manifests/quick-start/quick-start.yamlOnce everything is up, you'll observe messages in fluent bit pod logs like below:
[0] my_dummy: [1587991566.000091658, {"message"=>"dummy"}]
[1] my_dummy: [1587991567.000061572, {"message"=>"dummy"}]
[2] my_dummy: [1587991568.000056842, {"message"=>"dummy"}]
[3] my_dummy: [1587991569.000896217, {"message"=>"dummy"}]
[0] my_dummy: [1587991570.000172328, {"message"=>"dummy"}]It means the FluentBit Operator works properly if you see the above messages, you can delete the quick start test by
kubectl delete -f https://raw.githubusercontent.com/kubesphere/fluentbit-operator/master/manifests/quick-start/quick-start.yamlWhen starting the operator, you can pass an optional set of namespaces to watch for resources in with the --watch-namespaces flag. It takes a comma-separated list of namespaces to watch:
...
spec:
containers:
image: kubesphere/fluentbit-operator
- args:
- --watch-namespaces=namespace1,namespace2
This guide provisions a logging pipeline including the Fluent Bit DaemonSet and its log input/filter/output configurations to collect Kubernetes logs including container logs and kubelet logs.
Note that you need a running Elasticsearch v5+ cluster to receive log data before start. Remember to adjust output-elasticsearch.yaml to your own es setup. Kafka and Fluentd outputs are optional and are turned off by default.
kubectl apply -f manifests/logging-stack
# You can change the namespace in manifests/logging-stack/kustomization.yaml
# and then use command below to install to another namespace
# kubectl kustomize manifests/logging-stack/ | kubectl apply -f -If your container runtime is docker
helm upgrade fluentbit-operator --create-namespace -n kubesphere-logging-system charts/fluentbit-operator/ --set Kubernetes=true,containerRuntime=dockerIf your container runtime is containerd
helm upgrade fluentbit-operator --create-namespace -n kubesphere-logging-system charts/fluentbit-operator/ --set Kubernetes=true,containerRuntime=containerdIf your container runtime is cri-o
helm upgrade fluentbit-operator --create-namespace -n kubesphere-logging-system charts/fluentbit-operator/ --set Kubernetes=true,containerRuntime=crioWithin a couple of minutes, you should observe an index available:
$ curl localhost:9200/_cat/indices
green open ks-logstash-log-2020.04.26 uwQuoO90TwyigqYRW7MDYQ 1 1 99937 0 31.2mb 31.2mbSuccess!
The Linux audit framework provides a CAPP-compliant (Controlled Access Protection Profile) auditing system that reliably collects information about any security-relevant (or non-security-relevant) event on a system. Refer to manifests/logging-stack/auditd, it supports a method for collecting audit logs from the Linux audit framework.
kubectl apply -f manifests/logging-stack/auditd
# You can change the namespace in manifests/logging-stack/auditd/kustomization.yaml
# and then use command below to install to another namespace
# kubectl kustomize manifests/logging-stack/auditd/ | kubectl apply -f -Within a couple of minutes, you should observe an index available:
$ curl localhost:9200/_cat/indices
green open ks-logstash-log-2021.04.06 QeI-k_LoQZ2h1z23F3XiHg 5 1 404879 0 298.4mb 149.2mbFluent Bit comes with a built-in HTTP Server. According to the official documentation of fluentbit You can enable this by enabling the HTTP server from the fluent bit configuration file:
[SERVICE]
HTTP_Server On
HTTP_Listen 0.0.0.0
HTTP_PORT 2020
When you use the kubesphere/fluentbit-operator, You can enable this from FluentBitConfig manifest. Example is below:
apiVersion: logging.kubesphere.io/v1alpha2
kind: FluentBitConfig
metadata:
name: fluent-bit-config
namespace: logging-system
spec:
filterSelector:
matchLabels:
logging.kubesphere.io/enabled: 'true'
inputSelector:
matchLabels:
logging.kubesphere.io/enabled: 'true'
outputSelector:
matchLabels:
logging.kubesphere.io/enabled: 'true'
service:
httpListen: 0.0.0.0
httpPort: 2020
httpServer: true
parsersFile: parsers.conf
Once HTTP server is enabled, you should be able to get the information:
curl <podIP>:2020 | jq .
{
"fluent-bit": {
"version": "1.8.3",
"edition": "Community",
"flags": [
"FLB_HAVE_PARSER",
"FLB_HAVE_RECORD_ACCESSOR",
"FLB_HAVE_STREAM_PROCESSOR",
"FLB_HAVE_TLS",
"FLB_HAVE_OPENSSL",
"FLB_HAVE_AWS",
"FLB_HAVE_SIGNV4",
"FLB_HAVE_SQLDB",
"FLB_HAVE_METRICS",
"FLB_HAVE_HTTP_SERVER",
"FLB_HAVE_SYSTEMD",
"FLB_HAVE_FORK",
"FLB_HAVE_TIMESPEC_GET",
"FLB_HAVE_GMTOFF",
"FLB_HAVE_UNIX_SOCKET",
"FLB_HAVE_PROXY_GO",
"FLB_HAVE_JEMALLOC",
"FLB_HAVE_LIBBACKTRACE",
"FLB_HAVE_REGEX",
"FLB_HAVE_UTF8_ENCODER",
"FLB_HAVE_LUAJIT",
"FLB_HAVE_C_TLS",
"FLB_HAVE_ACCEPT4",
"FLB_HAVE_INOTIFY"
]
}
}The list below shows supported plugins which are based on Fluent Bit v1.7.x+. For more information, please refer to the API docs of each plugin.
Input, filter, and output plugins are connected by label selectors. For input and output plugins, always create Input or Output CRs for every plugin. Don't aggregate multiple inputs or outputs into one Input or Output object, except you have a good reason to do so. Take the demo logging stack for example, we have one yaml file for each output.
However, for filter plugins, if you want a filter chain, the order of filters matters. You need to organize multiple filters into an array as the demo logging stack suggests.
Path to file in Fluent Bit config should be well regulated. Fluent Bit Operator adopts the following convention internally.
| Dir Path | Description |
|---|---|
| /fluent-bit/tail | Stores tail related files, eg. file tracking db. Using fluentbit.spec.positionDB will mount a file pos.db under this dir by default. |
| /fluent-bit/secrets/{secret_name} | Stores secrets, eg. TLS files. Specify secrets to mount in fluentbit.spec.secrets, then you have access. |
| /fluent-bit/config | Stores the main config file and user-defined parser config file. |
Note that ServiceAccount files are mounted at
/var/run/secrets/kubernetes.io/serviceaccount.
To enable parsers, you must set the value of FluentBitConfig.Spec.Service.ParsersFile to parsers.conf. Your custom parsers will be included into the built-in parser config via @INCLUDE /fluent-bit/config/parsers.conf. Note that the parsers.conf contains a few built-in parsers, for example, docker. Read parsers.conf for more information.
Check out the demo in the folder /manifests/regex-parser for how to use a custom regex parser.
- Support containerd log format
- Add Fluentd CRDs as the log aggregation layer with group name
fluentd.fluent.io - Add FluentBit Cluster CRDs with new group name
fluentbit.fluent.io - Rename the entire project to Fluent Operator
- Support more Fluentd & FluentBit plugins
- golang v1.16+.requirement
- kubectl v1.16.13+.
- kubebuilder v2.3+ (the project is build with v2.3.2)
- Access to a Kubernetes cluster v1.16.13+
- Install CRDs:
make install - Run:
make run
API Doc is generated automatically. To modify it, edit the comment above struct fields, then run go run cmd/doc-gen/main.go.
Most files under the folder manifests/setup are automatically generated from config. Don't edit them directly, run make manifests instead, then replace these files accordingly.