Cherry-pick #15761 to 7.x: [Metricbeat][Istio] Add pilot metricset

CHANGELOG.next.asciidoc

@@ -195,6 +195,7 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
 - Enable script processor. {pull}14711[14711]
 - Add mixer metricset for Istio Metricbeat module {pull}15696[15696]
 - Add mesh metricset for Istio Metricbeat module{pull}15535[15535]
+- Add pilot metricset for Istio Metricbeat module {pull}15761[15761]
 - Add STAN dashboard {pull}15654[15654]
 - Add `key/value` mode for SQL module. {issue}15770[15770] {pull]15845[15845]
 - Add support for Unix socket in Memcached metricbeat module. {issue}13685[13685] {pull}15822[15822]

metricbeat/docs/fields.asciidoc

@@ -17138,6 +17138,203 @@ type: long
 
 --
 
+[float]
+=== pilot
+
+Contains statistics related to the Istio pilot service
+
+
+
+*`istio.pilot.xds.count`*::
++
+--
+Count of concurrent xDS client connections for Pilot.
+
+
+type: integer
+
+--
+
+*`istio.pilot.xds.pushes`*::
++
+--
+Count of xDS messages sent, as well as errors building or sending xDS messages for lds, rds, cds and eds.
+
+
+type: integer
+
+--
+
+*`istio.pilot.xds.push.time.sec`*::
++
+--
+Total time in seconds Pilot takes to push lds, rds, cds and eds.
+
+
+type: long
+
+--
+
+*`istio.pilot.xds.eds.instances`*::
++
+--
+Instances for each cluster, as of last push. Zero instances is an error.
+
+
+type: object
+
+--
+
+*`istio.pilot.xds.push.context.errors`*::
++
+--
+Number of errors (timeouts) initiating push context.
+
+
+type: integer
+
+--
+
+*`istio.pilot.xds.internal.errors`*::
++
+--
+Total number of internal XDS errors in pilot.
+
+
+type: integer
+
+--
+
+*`istio.pilot.conflict.listener.inbound`*::
++
+--
+Number of conflicting inbound listeners.
+
+
+type: integer
+
+--
+
+*`istio.pilot.conflict.listener.outbound.http.over.current.tcp`*::
++
+--
+Number of conflicting wildcard http listeners with current wildcard tcp listener.
+
+
+type: integer
+
+--
+
+*`istio.pilot.conflict.listener.outbound.http.over.https`*::
++
+--
+Number of conflicting HTTP listeners with well known HTTPS ports.
+
+
+type: integer
+
+--
+
+*`istio.pilot.conflict.listener.outbound.tcp.over.current.http`*::
++
+--
+Number of conflicting wildcard tcp listeners with current wildcard http listener.
+
+
+type: integer
+
+--
+
+*`istio.pilot.conflict.listener.outbound.tcp.over.current.tcp`*::
++
+--
+Number of conflicting tcp listeners with current tcp listener.
+
+
+type: integer
+
+--
+
+*`istio.pilot.proxy.conv.ms.bucket.*`*::
++
+--
+Time needed by Pilot to push Envoy configurations, histogram buckets in milliseconds.
+
+
+type: object
+
+--
+
+*`istio.pilot.proxy.conv.ms.sum`*::
++
+--
+Time needed by Pilot to push Envoy configurations, histogram sum of times in milliseconds.
+
+
+type: long
+
+--
+
+*`istio.pilot.services`*::
++
+--
+Total services known to pilot.
+
+
+type: integer
+
+--
+
+*`istio.pilot.virt.services`*::
++
+--
+Total virtual services known to pilot.
+
+
+type: integer
+
+--
+
+*`istio.pilot.proxy.conv.ms.count`*::
++
+--
+Time needed by Pilot to push Envoy configurations, histogram count of times.
+
+
+type: long
+
+--
+
+*`istio.pilot.no.ip`*::
++
+--
+Pods not found in the endpoint table, possibly invalid.
+
+
+type: object
+
+--
+
+*`istio.pilot.cluster`*::
++
+--
+The instance FQDN.
+
+
+type: text
+
+--
+
+*`istio.pilot.type`*::
++
+--
+The Envoy proxy configuration type.
+
+
+type: text
+
+--
+
 [[exported-fields-jolokia]]
 == Jolokia fields
 

metricbeat/docs/modules/istio.asciidoc

@@ -11,7 +11,7 @@ beta[]
 This is the Istio module. The Istio module collects metrics from the
 Istio https://istio.io/docs/tasks/observability/metrics/querying-metrics/#about-the-prometheus-add-on[prometheus exporters endpoints].
 
-The default metricsets are `mesh`, `mixer`.
+The default metricsets are `mesh`, `mixer`, `pilot`.
 
 [float]
 === Compatibility
@@ -41,6 +41,13 @@ metricbeat.modules:
   period: 10s
   # use istio-telemetry.istio-system:15014, when deploying Metricbeat in a kuberentes cluster as Pod or Daemonset
   hosts: ["localhost:15014"]
+
+# Istio pilot. To collect all Pilot-generated metrics.
+- module: istio
+  metricsets: ["pilot"]
+  period: 10s
+  # use istio-pilot.istio-system:15014, when deploying Metricbeat in a kuberentes cluster as Pod or Daemonset
+  hosts: ["localhost:15014"]
 ----
 
 [float]
@@ -52,7 +59,11 @@ The following metricsets are available:
 
 * <<metricbeat-metricset-istio-mixer,mixer>>
 
+* <<metricbeat-metricset-istio-pilot,pilot>>
+
 include::istio/mesh.asciidoc[]
 
 include::istio/mixer.asciidoc[]
 
+include::istio/pilot.asciidoc[]
+

metricbeat/docs/modules/istio/pilot.asciidoc

@@ -0,0 +1,23 @@
+////
+This file is generated! See scripts/mage/docs_collector.go
+////
+
+[[metricbeat-metricset-istio-pilot]]
+=== istio pilot metricset
+
+beta[]
+
+include::../../../../x-pack/metricbeat/module/istio/pilot/_meta/docs.asciidoc[]
+
+
+==== Fields
+
+For a description of each field in the metricset, see the
+<<exported-fields-istio,exported fields>> section.
+
+Here is an example document generated by this metricset:
+
+[source,json]
+----
+include::../../../../x-pack/metricbeat/module/istio/pilot/_meta/data.json[]
+----

metricbeat/docs/modules_list.asciidoc

@@ -104,8 +104,9 @@ This file is generated! See scripts/mage/docs_collector.go
 |<<metricbeat-module-ibmmq,IBM MQ>>  beta[]   |image:./images/icon-yes.png[Prebuilt dashboards are available]    |  
 .1+| .1+|  |<<metricbeat-metricset-ibmmq-qmgr,qmgr>> beta[]  
 |<<metricbeat-module-istio,istio>>  beta[]   |image:./images/icon-no.png[No prebuilt dashboards]    |  
-.2+| .2+|  |<<metricbeat-metricset-istio-mesh,mesh>> beta[]  
+.3+| .3+|  |<<metricbeat-metricset-istio-mesh,mesh>> beta[]  
 |<<metricbeat-metricset-istio-mixer,mixer>> beta[]  
+|<<metricbeat-metricset-istio-pilot,pilot>> beta[]  
 |<<metricbeat-module-jolokia,Jolokia>>     |image:./images/icon-no.png[No prebuilt dashboards]    |  
 .1+| .1+|  |<<metricbeat-metricset-jolokia-jmx,jmx>>   
 |<<metricbeat-module-kafka,Kafka>>     |image:./images/icon-yes.png[Prebuilt dashboards are available]    |  

x-pack/metricbeat/metricbeat.reference.yml

@@ -502,6 +502,13 @@ metricbeat.modules:
   # use istio-telemetry.istio-system:15014, when deploying Metricbeat in a kuberentes cluster as Pod or Daemonset
   hosts: ["localhost:15014"]
 
+# Istio pilot. To collect all Pilot-generated metrics.
+- module: istio
+  metricsets: ["pilot"]
+  period: 10s
+  # use istio-pilot.istio-system:15014, when deploying Metricbeat in a kuberentes cluster as Pod or Daemonset
+  hosts: ["localhost:15014"]
+
 #------------------------------- Jolokia Module -------------------------------
 - module: jolokia
   #metricsets: ["jmx"]

x-pack/metricbeat/module/istio/_meta/config.reference.yml

@@ -11,3 +11,10 @@
   period: 10s
   # use istio-telemetry.istio-system:15014, when deploying Metricbeat in a kuberentes cluster as Pod or Daemonset
   hosts: ["localhost:15014"]
+
+# Istio pilot. To collect all Pilot-generated metrics.
+- module: istio
+  metricsets: ["pilot"]
+  period: 10s
+  # use istio-pilot.istio-system:15014, when deploying Metricbeat in a kuberentes cluster as Pod or Daemonset
+  hosts: ["localhost:15014"]

x-pack/metricbeat/module/istio/_meta/config.yml

@@ -11,3 +11,10 @@
   period: 10s
   # use istio-telemetry.istio-system:15014, when deploying Metricbeat in a kuberentes cluster
   hosts: ["localhost:15014"]
+
+# Istio pilot. To collect all Pilot-generated metrics.
+- module: istio
+  metricsets: ["pilot"]
+  period: 10s
+  # use istio-pilot.istio-system:15014, when deploying Metricbeat in a kuberentes cluster
+  hosts: ["localhost:15014"]

x-pack/metricbeat/module/istio/_meta/docs.asciidoc

@@ -1,7 +1,7 @@
 This is the Istio module. The Istio module collects metrics from the
 Istio https://istio.io/docs/tasks/observability/metrics/querying-metrics/#about-the-prometheus-add-on[prometheus exporters endpoints].
 
-The default metricsets are `mesh`, `mixer`.
+The default metricsets are `mesh`, `mixer`, `pilot`.
 
 [float]
 === Compatibility

x-pack/metricbeat/module/istio/pilot/_meta/data.json

@@ -0,0 +1,26 @@
+{
+    "@timestamp": "2019-03-01T08:05:34.853Z",
+    "event": {
+        "dataset": "istio.pilot",
+        "duration": 115000,
+        "module": "istio"
+    },
+    "istio": {
+        "pilot": {
+            "cluster": "outbound_.15031_._.istio-ingressgateway.istio-system.svc.cluster.local",
+            "xds": {
+                "eds": {
+                    "instances": 1
+                }
+            }
+        }
+    },
+    "metricset": {
+        "name": "pilot",
+        "period": 10000
+    },
+    "service": {
+        "address": "127.0.0.1:55555",
+        "type": "istio"
+    }
+}

x-pack/metricbeat/module/istio/pilot/_meta/docs.asciidoc

@@ -0,0 +1 @@
+This is the pilot metricset of the module istio. This metricset collects all Pilot-generated metrics.