Filebeat coredns module (#11200)

* Add Filebeat coredns module
elastic · Mar 26, 2019 · f98f2f4 · f98f2f4
1 parent 53500aa
commit f98f2f4
Show file tree

Hide file tree

Showing 24 changed files with 1,272 additions and 0 deletions.
diff --git a/CHANGELOG.next.asciidoc b/CHANGELOG.next.asciidoc
@@ -309,6 +309,7 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
 - Add support for loading custom NetFlow and IPFIX field definitions to netflow input. {pull}10945[10945]
 - Added categorization fields for SSH login events in the system/auth fileset. {pull}11334[11334]
 - Add support for MySQL 8.0 slow logs and tests also for Percona 8.0 and MariaDB 10.3. {pull}11417[11417]
+- New Filebeat coredns module to ingest coredns logs. It supports both native coredns deployment and coredns deployment in kubernetes. {pull}11200[11200]
 
 *Heartbeat*
 

diff --git a/filebeat/docs/fields.asciidoc b/filebeat/docs/fields.asciidoc
@@ -16,6 +16,7 @@ grouped in the following categories:
 * <<exported-fields-auditd>>
 * <<exported-fields-beat>>
 * <<exported-fields-cloud>>
+* <<exported-fields-coredns>>
 * <<exported-fields-docker-processor>>
 * <<exported-fields-ecs>>
 * <<exported-fields-elasticsearch>>
@@ -1125,6 +1126,114 @@ type: alias
 
 alias to: cloud.region
 
+--
+
+[[exported-fields-coredns]]
+== Coredns fields
+
+Module for handling logs produced by coredns
+
+
+
+[float]
+== coredns fields
+
+coredns fields after normalization
+
+
+
+*`coredns.id`*::
++
+--
+type: keyword
+
+id of the DNS transaction
+
+
+--
+
+*`coredns.query.size`*::
++
+--
+type: integer
+
+format: bytes
+
+size of the DNS query
+
+
+--
+
+*`coredns.query.class`*::
++
+--
+type: keyword
+
+DNS query class
+
+
+--
+
+*`coredns.query.name`*::
++
+--
+type: keyword
+
+DNS query name
+
+
+--
+
+*`coredns.query.type`*::
++
+--
+type: keyword
+
+DNS query type
+
+
+--
+
+*`coredns.response.code`*::
++
+--
+type: keyword
+
+DNS response code
+
+
+--
+
+*`coredns.response.flags`*::
++
+--
+type: keyword
+
+DNS response flags
+
+
+--
+
+*`coredns.response.size`*::
++
+--
+type: integer
+
+format: bytes
+
+size of the DNS response
+
+
+--
+
+*`coredns.dnssec_ok`*::
++
+--
+type: boolean
+
+dnssec flag
+
+
 --
 
 [[exported-fields-docker-processor]]

diff --git a/filebeat/docs/modules/coredns.asciidoc b/filebeat/docs/modules/coredns.asciidoc
@@ -0,0 +1,37 @@
+////
+This file is generated! See scripts/docs_collector.py
+////
+
+[[filebeat-module-coredns]]
+[role="xpack"]
+
+:modulename: coredns
+:has-dashboards: true
+
+== Coredns Module
+
+This is a filebeat module for coredns. It supports both standalone coredns deployment and 
+coredns deployment in Kubernetes. Standalone coredns deployment uses the log fileset, while
+Kubernetes coredns deployment uses the kubernetes fileset.
+
+[float]
+=== Compatibility
+
+Although this module has been developed against Kubernetes v1.13.x, it is expected to work
+with other versions of Kubernetes.
+
+[float]
+=== Example dashboard
+
+This module comes with a sample dashboard.
+
+[role="screenshot"]
+image::./images/Coredns_Overview_Dashboard.jpg[]
+
+
+[float]
+=== Fields
+
+For a description of each field in the module, see the
+<<exported-fields-coredns,exported fields>> section.
+
diff --git a/filebeat/docs/modules_list.asciidoc b/filebeat/docs/modules_list.asciidoc
@@ -5,6 +5,7 @@ This file is generated! See scripts/docs_collector.py
   * <<filebeat-modules-overview>>
   * <<filebeat-module-apache>>
   * <<filebeat-module-auditd>>
+  * <<filebeat-module-coredns>>
   * <<filebeat-module-elasticsearch>>
   * <<filebeat-module-haproxy>>
   * <<filebeat-module-icinga>>
@@ -33,6 +34,7 @@ This file is generated! See scripts/docs_collector.py
 include::modules-overview.asciidoc[]
 include::modules/apache.asciidoc[]
 include::modules/auditd.asciidoc[]
+include::modules/coredns.asciidoc[]
 include::modules/elasticsearch.asciidoc[]
 include::modules/haproxy.asciidoc[]
 include::modules/icinga.asciidoc[]

diff --git a/x-pack/filebeat/filebeat.reference.yml b/x-pack/filebeat/filebeat.reference.yml
@@ -82,6 +82,16 @@ filebeat.modules:
     # can be added under this section.
     #input:
 
+#------------------------------- Coredns Module -------------------------------
+- module: coredns
+  # Fileset for native deployment
+  log: 
+    enabled: true
+
+    # Set custom paths for the log files. If left empty,
+    # Filebeat will choose the paths depending on your OS.
+    #var.paths:
+
 #---------------------------- Elasticsearch Module ----------------------------
 - module: elasticsearch
   # Server log

diff --git a/x-pack/filebeat/include/list.go b/x-pack/filebeat/include/list.go
diff --git a/x-pack/filebeat/module/coredns/README.md b/x-pack/filebeat/module/coredns/README.md
@@ -0,0 +1,160 @@
+# Coredns Module
+
+This is a filebeat module for coredns. It supports both standalone coredns deployment and 
+coredns deployment in Kubernetes.
+
+## Caveats
+
+* Module is to be considered _beta_.
+
+## Download and install Filebeat
+
+Grab the filebeat binary from elastic.co, and install it by following the instructions.
+
+## Deployment Scenario #1: coredns native deployment
+
+Make sure to update coredns configuration to enable log plugin. This module assumes that coredns log
+entries will be written to /var/log/coredns.log. Should it be not the case, please point the module 
+log path to the path of the log file. 
+
+Update filebeat.yml to point to Elasticsearch and Kibana. 
+Setup Filebeat.
+```
+./filebeat setup --modules coredns -e
+```
+
+Enable the Filebeat coredns module
+```
+./filebeat modules enable coredns
+```
+
+Start Filebeat
+```
+./filebeat -e
+```
+
+Now, the Coredns logs and dashboard should appear in Kibana.
+
+
+## Deployment Scenario #2: coredns for kubernetes 
+
+For Kubernetes deployment, the filebeat daemon-set yaml file needs to be deployed to the 
+Kubernetes cluster. Sample configuration files is provided under the `beats/deploy/filebeat` 
+directory, and can be deployed by doing the following:
+```
+kubectl apply -f filebeat
+```
+
+#### Note the following section in the ConfigMap, make changes to the yaml file if necessary
+```
+  filebeat.autodiscover:
+    providers:
+      - type: kubernetes
+        hints.enabled: true
+        default.disable: true
+
+  processors:
+    - add_kubernetes_metadata:
+        in_cluster: true
+```
+
+This enables auto-discovery and hints for filebeat. When default.disable is set to true (default value is false), it will disable log harvesting for the pod/container, unless it has specific annotations enabled. This gives users more granular control on kubernetes log ingestion. The `add_kubernetes_metadata` processor will add enrichment data for Kubernetes to the ingest logs.
+
+#### Note the following section in the DaemonSet, make changes to the yaml file if necessary
+```
+apiVersion: extensions/v1beta1
+kind: DaemonSet
+metadata:
+  name: filebeat
+  namespace: kube-system
+  labels:
+    k8s-app: filebeat
+spec:
+  template:
+    metadata:
+      labels:
+        k8s-app: filebeat
+    spec:
+      serviceAccountName: filebeat
+      terminationGracePeriodSeconds: 30
+      containers:
+      - name: filebeat
+        image: docker.elastic.co/beats/filebeat:%VERSION%
+        args: [
+          "sh", "-c", "filebeat setup -e --modules coredns -c /etc/filebeat.yml && filebeat -e -c /etc/filebeat.yml"
+        ]
+        env:
+        # Edit the following values to reflect your setup accordingly
+        - name: ELASTICSEARCH_HOST
+          value: 192.168.99.1
+        - name: ELASTICSEARCH_USERNAME
+          value: elastic
+        - name: ELASTICSEARCH_PASSWORD
+          value: changeme
+        - name: KIBANA_HOST
+          value: 192.168.99.1
+```
+
+The module setup step can also be done separately without Kubernetes if applicable, and in that case, the args can be simplified to:
+```
+        args: [
+          "sh", "-c", "filebeat -e -c /etc/filebeat.yml"
+        ]
+```
+
+### Note that you probably need to update the coredns configmap to enable logging, and coredns deployment to add proper annotations. 
+
+##### Sample ConfigMap for coredns:
+
+```
+apiVersion: v1
+data:
+  Corefile: |
+    .:53 {
+        log
+        errors
+        health
+        kubernetes cluster.local in-addr.arpa ip6.arpa {
+           pods verified
+           endpoint_pod_names
+           upstream
+           fallthrough in-addr.arpa ip6.arpa
+        }
+        prometheus :9153
+        proxy . /etc/resolv.conf
+        cache 30
+        loop
+        reload
+        loadbalance
+    }
+kind: ConfigMap
+metadata:
+  creationTimestamp: "2019-01-31T21:02:57Z"
+  name: coredns
+  namespace: kube-system
+  resourceVersion: "185717"
+  selfLink: /api/v1/namespaces/kube-system/configmaps/coredns
+  uid: 95a5d5cb-259b-11e9-8e5d-080027971f3c
+```
+
+#### Sample Deployment for coredns. Note the annotations.
+
+```
+apiVersion: extensions/v1beta1
+kind: Deployment
+metadata:
+  name: coredns
+spec:
+  replicas: 2
+  template:
+    metadata:
+      annotations:
+        "co.elastic.logs/module": "coredns"
+        "co.elastic.logs/fileset": "log"
+        "co.elastic.logs/disable": "false"
+      labels:
+        k8s-app: coredns
+    spec:
+      <snipped>
+```
+
diff --git a/x-pack/filebeat/module/coredns/_meta/config.yml b/x-pack/filebeat/module/coredns/_meta/config.yml
@@ -0,0 +1,8 @@
+- module: coredns
+  # Fileset for native deployment
+  log: 
+    enabled: true
+
+    # Set custom paths for the log files. If left empty,
+    # Filebeat will choose the paths depending on your OS.
+    #var.paths: