Cherry-pick #15301 to 7.x: [Metricbeat] Implement IBM MQ module (#15612)

* [Metricbeat] Implement IBM MQ module  (#15301)

* Modify cockroachdb source

* Define testdata

* Do not publish ports

* Update docs

* mage fmt update

* Describe containerized environment

* Update CHANGELOG.next.asciidoc

Co-Authored-By: Chris Mark <chrismarkou92@gmail.com>

* Update data.json

* Rename image

* Update source after review

* Filter ibmmq_ metrics

* mage check

* Fix: mage check

* Don't expose port

* Rename status to qmgr

* Add subscriptions overview dashboard for IBM MQ module

* Add calls, messages overview dashboard for IBM MQ module

* Add screenshots

* Fix: mage check

* Fix: CHANGELOG

* Add explanation

* Fix: mage check

Co-authored-by: Chris Mark <chrismarkou92@gmail.com>
(cherry picked from commit 2ed633b)

* Fix: link

* Bug

CHANGELOG.next.asciidoc

@@ -162,6 +162,7 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
 - Add cost warnings for the azure module. {pull}15356[15356]
 - Release elb module as GA. {pull}15485[15485]
 - Add a `system/network_summary` metricset {pull}15196[15196]
+- Add IBM MQ light-weight Metricbeat module {pull}15301[15301]
 
 *Packetbeat*
 

metricbeat/docs/fields.asciidoc

@@ -41,6 +41,7 @@ grouped in the following categories:
 * <<exported-fields-haproxy>>
 * <<exported-fields-host-processor>>
 * <<exported-fields-http>>
+* <<exported-fields-ibmmq>>
 * <<exported-fields-jolokia>>
 * <<exported-fields-jolokia-autodiscover>>
 * <<exported-fields-kafka>>
@@ -15840,6 +15841,14 @@ json metricset
 server
 
 
+[[exported-fields-ibmmq]]
+== IBM MQ fields
+
+IBM MQ module
+
+
+
+
 [[exported-fields-jolokia]]
 == Jolokia fields
 

metricbeat/docs/modules/ibmmq.asciidoc

@@ -0,0 +1,90 @@
+////
+This file is generated! See scripts/mage/docs_collector.go
+////
+
+[[metricbeat-module-ibmmq]]
+[role="xpack"]
+== IBM MQ module
+
+beta[]
+
+This module periodically fetches metrics from a containerized distribution of IBM MQ.
+
+[float]
+=== Compatibility
+
+The ibmmq `qmgr` metricset is compatible with a containerized distribution of IBM MQ (since version 9.1.0).
+The Docker image starts the `runmqserver` process, which spawns the HTTP server exposing metrics in Prometheus
+format ([source code](https://github.com/ibm-messaging/mq-container/blob/9.1.0/internal/metrics/metrics.go)).
+
+The Docker container lifecycle, including metrics collection, has been described in the [Internals](https://github.com/ibm-messaging/mq-container/blob/9.1.0/docs/internals.md)
+document.
+
+The image provides an option to easily enable metrics exporter using an environment
+variable:
+
+`MQ_ENABLE_METRICS` - Set this to `true` to generate Prometheus metrics for the Queue Manager.
+
+[float]
+=== Dashboard
+
+The ibmmq module includes predefined dashboards with overview information
+of the monitored Queue Manager, including subscriptions, calls and messages.
+
+image::./images/metricbeat-ibmmq-calls.png[]
+
+image::./images/metricbeat-ibmmq-messages.png[]
+
+image::./images/metricbeat-ibmmq-subscriptions.png[]
+
+
+[float]
+=== Example configuration
+
+The IBM MQ module supports the standard configuration options that are described
+in <<configuration-metricbeat>>. Here is an example configuration:
+
+[source,yaml]
+----
+metricbeat.modules:
+- module: ibmmq
+  metricsets: ['qmgr']
+  period: 10s
+  hosts: ['localhost:9157']
+
+  # This module uses the Prometheus collector metricset, all
+  # the options for this metricset are also available here.
+  metrics_path: /metrics
+
+  # The custom processor is responsible for filtering Prometheus metrics
+  # not stricly related to the IBM MQ domain, e.g. system load, process,
+  # metrics HTTP server.
+  processors:
+    - script:
+        lang: javascript
+        source: >
+          function process(event) {
+            var metrics = event.Get("prometheus.metrics");
+            Object.keys(metrics).forEach(function(key) {
+              if (!(key.match(/^ibmmq_.*$/))) {
+                event.Delete("prometheus.metrics." + key);
+              }
+            });
+            metrics = event.Get("prometheus.metrics");
+            if (Object.keys(metrics).length == 0) {
+              event.Cancel();
+            }
+          }
+----
+
+It also supports the options described in <<module-http-config-options>>.
+
+[float]
+=== Metricsets
+
+The following metricsets are available:
+
+* <<metricbeat-metricset-ibmmq-qmgr,qmgr>>
+
+include::ibmmq/qmgr.asciidoc[]
+

metricbeat/docs/modules/ibmmq/qmgr.asciidoc

@@ -0,0 +1,24 @@
+////
+This file is generated! See scripts/mage/docs_collector.go
+////
+
+[[metricbeat-metricset-ibmmq-qmgr]]
+=== IBM MQ qmgr metricset
+
+beta[]
+
+include::../../../../x-pack/metricbeat/module/ibmmq/qmgr/_meta/docs.asciidoc[]
+
+This is a default metricset. If the host module is unconfigured, this metricset is enabled by default.
+
+==== Fields
+
+For a description of each field in the metricset, see the
+<<exported-fields-ibmmq,exported fields>> section.
+
+Here is an example document generated by this metricset:
+
+[source,json]
+----
+include::../../../../x-pack/metricbeat/module/ibmmq/qmgr/_meta/data.json[]
+----

metricbeat/docs/modules_list.asciidoc

@@ -99,6 +99,8 @@ This file is generated! See scripts/mage/docs_collector.go
 |<<metricbeat-module-http,HTTP>>     |image:./images/icon-no.png[No prebuilt dashboards]    |  
 .2+| .2+|  |<<metricbeat-metricset-http-json,json>>   
 |<<metricbeat-metricset-http-server,server>>   
+|<<metricbeat-module-ibmmq,IBM MQ>>  beta[]   |image:./images/icon-yes.png[Prebuilt dashboards are available]    |  
+.1+| .1+|  |<<metricbeat-metricset-ibmmq-qmgr,qmgr>> beta[]  
 |<<metricbeat-module-jolokia,Jolokia>>     |image:./images/icon-no.png[No prebuilt dashboards]    |  
 .1+| .1+|  |<<metricbeat-metricset-jolokia-jmx,jmx>>   
 |<<metricbeat-module-kafka,Kafka>>     |image:./images/icon-yes.png[Prebuilt dashboards are available]    |  
@@ -254,6 +256,7 @@ include::modules/googlecloud.asciidoc[]
 include::modules/graphite.asciidoc[]
 include::modules/haproxy.asciidoc[]
 include::modules/http.asciidoc[]
+include::modules/ibmmq.asciidoc[]
 include::modules/jolokia.asciidoc[]
 include::modules/kafka.asciidoc[]
 include::modules/kibana.asciidoc[]

x-pack/metricbeat/metricbeat.reference.yml

@@ -503,6 +503,36 @@ metricbeat.modules:
   #    fields: # added to the the response in root. overwrites existing fields
   #      key: "value"
 
+#-------------------------------- IBM MQ Module --------------------------------
+- module: ibmmq
+  metricsets: ['qmgr']
+  period: 10s
+  hosts: ['localhost:9157']
+
+  # This module uses the Prometheus collector metricset, all
+  # the options for this metricset are also available here.
+  metrics_path: /metrics
+
+  # The custom processor is responsible for filtering Prometheus metrics
+  # not stricly related to the IBM MQ domain, e.g. system load, process,
+  # metrics HTTP server.
+  processors:
+    - script:
+        lang: javascript
+        source: >
+          function process(event) {
+            var metrics = event.Get("prometheus.metrics");
+            Object.keys(metrics).forEach(function(key) {
+              if (!(key.match(/^ibmmq_.*$/))) {
+                event.Delete("prometheus.metrics." + key);
+              }
+            });
+            metrics = event.Get("prometheus.metrics");
+            if (Object.keys(metrics).length == 0) {
+              event.Cancel();
+            }
+          }
+
 #------------------------------- Jolokia Module -------------------------------
 - module: jolokia
   #metricsets: ["jmx"]

x-pack/metricbeat/module/ibmmq/_meta/Dockerfile

@@ -0,0 +1,11 @@
+ARG IBMMQ_VERSION
+
+FROM ibmcom/mq:${IBMMQ_VERSION}
+
+ENV IBMMQ_METRICS_REST_PORT=9157
+
+ENV LICENSE=accept
+ENV MQ_QMGR_NAME=QM1
+ENV MQ_ENABLE_METRICS=true
+
+HEALTHCHECK --interval=1s --retries=90 CMD curl -s --fail http://127.0.0.1:${IBMMQ_METRICS_REST_PORT}/metrics | grep -q "ibmmq_qmgr_commit_total"

x-pack/metricbeat/module/ibmmq/_meta/config.yml

@@ -0,0 +1,28 @@
+- module: ibmmq
+  metricsets: ['qmgr']
+  period: 10s
+  hosts: ['localhost:9157']
+
+  # This module uses the Prometheus collector metricset, all
+  # the options for this metricset are also available here.
+  metrics_path: /metrics
+
+  # The custom processor is responsible for filtering Prometheus metrics
+  # not stricly related to the IBM MQ domain, e.g. system load, process,
+  # metrics HTTP server.
+  processors:
+    - script:
+        lang: javascript
+        source: >
+          function process(event) {
+            var metrics = event.Get("prometheus.metrics");
+            Object.keys(metrics).forEach(function(key) {
+              if (!(key.match(/^ibmmq_.*$/))) {
+                event.Delete("prometheus.metrics." + key);
+              }
+            });
+            metrics = event.Get("prometheus.metrics");
+            if (Object.keys(metrics).length == 0) {
+              event.Cancel();
+            }
+          }

x-pack/metricbeat/module/ibmmq/_meta/docs.asciidoc

@@ -0,0 +1,28 @@
+This module periodically fetches metrics from a containerized distribution of IBM MQ.
+
+[float]
+=== Compatibility
+
+The ibmmq `qmgr` metricset is compatible with a containerized distribution of IBM MQ (since version 9.1.0).
+The Docker image starts the `runmqserver` process, which spawns the HTTP server exposing metrics in Prometheus
+format ([source code](https://github.com/ibm-messaging/mq-container/blob/9.1.0/internal/metrics/metrics.go)).
+
+The Docker container lifecycle, including metrics collection, has been described in the [Internals](https://github.com/ibm-messaging/mq-container/blob/9.1.0/docs/internals.md)
+document.
+
+The image provides an option to easily enable metrics exporter using an environment
+variable:
+
+`MQ_ENABLE_METRICS` - Set this to `true` to generate Prometheus metrics for the Queue Manager.
+
+[float]
+=== Dashboard
+
+The ibmmq module includes predefined dashboards with overview information
+of the monitored Queue Manager, including subscriptions, calls and messages.
+
+image::./images/metricbeat-ibmmq-calls.png[]
+
+image::./images/metricbeat-ibmmq-messages.png[]
+
+image::./images/metricbeat-ibmmq-subscriptions.png[]

x-pack/metricbeat/module/ibmmq/_meta/fields.yml

@@ -0,0 +1,10 @@
+- key: ibmmq
+  title: 'IBM MQ'
+  release: beta
+  description: >
+    IBM MQ module
+  settings: ["http"]
+  fields:
+    - name: ibmmq
+      type: group
+      fields: