[8.15](backport #40678) [Metricbeat][vSphere] Support for configurabl…

…e IntervalId for performance API (#40833) * [Metricbeat][vSphere] Support for configurable IntervalId for performance API (#40678) * initial commit for intervalId supports for performance metrics * update docs and fix CI * Add changelog entry * fix CI * resolve review comments * fix loggers * resolved review comments * update versions * update UTs * update integration tests * 10s -> 20s * Update CHANGELOG.next.asciidoc Co-authored-by: Aman <38116245+devamanv@users.noreply.github.com> * Update metricbeat/docs/modules/vsphere.asciidoc Co-authored-by: Aman <38116245+devamanv@users.noreply.github.com> * make update * add recover for ToMetricSeries panic * return error instead just logging it. * remove restriction of interval IDs * remove unnecessary validations * remove recover and add empty condition * update changelog entry * Fix wrapping of errors in loggers * update data.json * update data.json * fix CI and loggers * update changelog entries * make update * fix changelog entries * update changelog entry --------- Co-authored-by: Aman <38116245+devamanv@users.noreply.github.com> (cherry picked from commit c75a7a4) * Update CHANGELOG.next.asciidoc --------- Co-authored-by: Kush Rana <89848966+kush-elastic@users.noreply.github.com> Co-authored-by: Ishleen Kaur <102962586+ishleenk17@users.noreply.github.com>
elastic · Sep 15, 2024 · 398fce0 · 398fce0
1 parent 1ce389c
commit 398fce0
Show file tree

Hide file tree

Showing 20 changed files with 434 additions and 255 deletions.
diff --git a/CHANGELOG.next.asciidoc b/CHANGELOG.next.asciidoc
@@ -33,8 +33,6 @@ https://github.com/elastic/beats/compare/v8.8.1\...main[Check the HEAD diff]
 *Metricbeat*
 
 - Setting period for counter cache for Prometheus remote_write at least to 60sec {pull}38553[38553]
-- Add new metrics for the vSphere Datastore metricset. {pull}40441[40441]
-- Update metrics for the vSphere Host metricset. {pull}40429[40429]
 
 *Osquerybeat*
 
@@ -206,16 +204,20 @@ https://github.com/elastic/beats/compare/v8.8.1\...main[Check the HEAD diff]
 - Add new memory/cgroup metrics to Kibana module {pull}37232[37232]
 - Add SSL support to mysql module {pull}37997[37997]
 - Add SSL support for aerospike module {pull}38126[38126]
+- Add new metrics for the vSphere Host metricset. {pull}40429[40429]
+- Add new metrics for the vSphere Datastore metricset. {pull}40441[40441]
+- Add metrics for the vSphere Virtualmachine metricset. {pull}40485[40485]
 - Add new metricset network for the vSphere module. {pull}40559[40559]
-- Add new metricset resourcepool for the vSphere module. {pull}40456[40456]
 - Add new metricset cluster for the vSphere module. {pull}40536[40536]
-- Add metrics for the vSphere Virtualmachine metricset. {pull}40485[40485]
+- Add new metricset resourcepool for the vSphere module. {pull}40456[40456]
+- Add new metricset datastorecluster for vSphere module. {pull}40634[40634]
 - Add support for snapshot in vSphere virtualmachine metricset {pull}40683[40683]
+- Add new metricset datastorecluster for vSphere module. {pull}40694[40694]
 - Update fields to use mapstr in vSphere virtualmachine metricset  {pull}40707[40707]
+- Add support for period based intervalID in vSphere host and datastore metricsets {pull}40678[40678]
 
 *Metricbeat*
 
-- Add support for new metrics for vSphere module datastorecluster metricset. {pull}40694[40694]
 
 *Osquerybeat*
 

diff --git a/metricbeat/docs/modules/vsphere.asciidoc b/metricbeat/docs/modules/vsphere.asciidoc
@@ -9,14 +9,79 @@ This file is generated! See scripts/mage/docs_collector.go
 [[metricbeat-module-vsphere]]
 == vSphere module
 
-The vSphere module uses the https://github.com/vmware/govmomi[Govmomi] library to collect metrics from any Vmware SDK URL (ESXi/VCenter). This library is built for and tested against ESXi and vCenter 5.5, 6.0 and 6.5.
+The vSphere module uses the https://github.com/vmware/govmomi[Govmomi] library to collect metrics from any VMware SDK URL (ESXi/VCenter).
 
-By default it enables the metricsets `cluster`, `datastore`, `datastorecluster`, `host`, `network`, `resourcepool` and `virtualmachine`.
+This module has been tested against ESXi and vCenter versions 5.5, 6.0, 6.5, and 7.0.3.
+
+By default, the vSphere module enables the following metricsets:
+
+1. cluster
+
+2. datastore
+
+3. datastorecluster
+
+4. host
+
+5. network
+
+6. resourcepool
+
+7. virtualmachine
+
+[float]
+=== Supported Periods:
+The Datastore and Host metricsets support performance data collection using the vSphere performance API. Given that the performance API imposes usage restrictions based on data collection intervals, users should configure the period optimally to ensure the receipt of real-time data. This configuration can be determined based on the https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.monitoring.doc/GUID-247646EA-A04B-411A-8DD4-62A3DCFCF49B.html[Data Collection Intervals] and https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.monitoring.doc/GUID-25800DE4-68E5-41CC-82D9-8811E27924BC.html[Data Collection Levels].
+
+[IMPORTANT]
+
+Only host and datastore metricsets have limitation of system configured period from vSphere instance. Users can still collect summary metrics if performance metrics are not supported for the configured instance.
+
+[float]
+==== Real-time data collection default interval:
+- 20s
+
+[float]
+==== Historical data collection default intervals:
+- 300s
+- 1800s
+- 7200s
+- 86400s
+
+[float]
+=== Example:
+If you need to configure multiple metricsets with different periods, you can achieve this by setting up multiple vSphere modules with different metricsets as demonstrated below:
+
+[source,yaml]
+----
+- module: vsphere
+  metricsets:
+   - cluster
+   - datastorecluster
+   - network
+   - resourcepool
+   - virtualmachine
+  period: 10s
+  hosts: ["https://localhost/sdk"]
+  username: "user"
+  password: "password"
+  insecure: false
+
+- module: vsphere
+  metricsets:
+   - datastore
+   - host
+  period: 300s
+  hosts: ["https://localhost/sdk"]
+  username: "user"
+  password: "password"
+  insecure: false
+----
 
 [float]
 === Dashboard
 
-The vsphere module comes with a predefined dashboard. For example:
+The vSphere module includes a predefined dashboard. For example:
 
 image::./images/metricbeat_vsphere_dashboard.png[]
 image::./images/metricbeat_vsphere_vm_dashboard.png[]
@@ -36,15 +101,25 @@ metricbeat.modules:
 - module: vsphere
   enabled: true
   metricsets: ["cluster", "datastore", "datastorecluster", "host", "network", "resourcepool", "virtualmachine"]
-  # Real-time data collection – An ESXi Server collects data for each performance counter every 20 seconds.
+  
+  # Real-time data collection – An ESXi Server collects data for each performance counter every 20 seconds by default.
+  # Supported Periods:
+  # The Datastore and Host metricsets support performance data collection using the vSphere performance API.
+  # Since the performance API has usage restrictions based on data collection intervals,
+  # users should ensure that the period is configured optimally to receive real-time data.
+  # users can still collect summary metrics if performance metrics are not supported for the configured instance.
+  # This configuration can be determined based on the Data Collection Intervals and Data Collection Levels.
+  # Reference Links:
+  # Data Collection Intervals: https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.monitoring.doc/GUID-247646EA-A04B-411A-8DD4-62A3DCFCF49B.html
+  # Data Collection Levels: https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.monitoring.doc/GUID-25800DE4-68E5-41CC-82D9-8811E27924BC.html
   period: 20s
   hosts: ["https://localhost/sdk"]
 
   username: "user"
   password: "password"
   # If insecure is true, don't verify the server's certificate chain
   insecure: false
-  # Get custom fields when using virtualmachine metric set. Default false.
+  # Get custom fields when using virtualmachine metricset. Default false.
   # get_custom_fields: false
 ----
 

diff --git a/metricbeat/metricbeat.reference.yml b/metricbeat/metricbeat.reference.yml
@@ -1008,15 +1008,25 @@ metricbeat.modules:
 - module: vsphere
   enabled: true
   metricsets: ["cluster", "datastore", "datastorecluster", "host", "network", "resourcepool", "virtualmachine"]
-  # Real-time data collection – An ESXi Server collects data for each performance counter every 20 seconds.
+
+  # Real-time data collection – An ESXi Server collects data for each performance counter every 20 seconds by default.
+  # Supported Periods:
+  # The Datastore and Host metricsets support performance data collection using the vSphere performance API.
+  # Since the performance API has usage restrictions based on data collection intervals,
+  # users should ensure that the period is configured optimally to receive real-time data.
+  # users can still collect summary metrics if performance metrics are not supported for the configured instance.
+  # This configuration can be determined based on the Data Collection Intervals and Data Collection Levels.
+  # Reference Links:
+  # Data Collection Intervals: https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.monitoring.doc/GUID-247646EA-A04B-411A-8DD4-62A3DCFCF49B.html
+  # Data Collection Levels: https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.monitoring.doc/GUID-25800DE4-68E5-41CC-82D9-8811E27924BC.html
   period: 20s
   hosts: ["https://localhost/sdk"]
 
   username: "user"
   password: "password"
   # If insecure is true, don't verify the server's certificate chain
   insecure: false
-  # Get custom fields when using virtualmachine metric set. Default false.
+  # Get custom fields when using virtualmachine metricset. Default false.
   # get_custom_fields: false
 
 #------------------------------- Windows Module -------------------------------

diff --git a/metricbeat/module/vsphere/_meta/config.reference.yml b/metricbeat/module/vsphere/_meta/config.reference.yml
@@ -1,13 +1,23 @@
 - module: vsphere
   enabled: true
   metricsets: ["cluster", "datastore", "datastorecluster", "host", "network", "resourcepool", "virtualmachine"]
-  # Real-time data collection – An ESXi Server collects data for each performance counter every 20 seconds.
+
+  # Real-time data collection – An ESXi Server collects data for each performance counter every 20 seconds by default.
+  # Supported Periods:
+  # The Datastore and Host metricsets support performance data collection using the vSphere performance API.
+  # Since the performance API has usage restrictions based on data collection intervals,
+  # users should ensure that the period is configured optimally to receive real-time data.
+  # users can still collect summary metrics if performance metrics are not supported for the configured instance.
+  # This configuration can be determined based on the Data Collection Intervals and Data Collection Levels.
+  # Reference Links:
+  # Data Collection Intervals: https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.monitoring.doc/GUID-247646EA-A04B-411A-8DD4-62A3DCFCF49B.html
+  # Data Collection Levels: https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.monitoring.doc/GUID-25800DE4-68E5-41CC-82D9-8811E27924BC.html
   period: 20s
   hosts: ["https://localhost/sdk"]
 
   username: "user"
   password: "password"
   # If insecure is true, don't verify the server's certificate chain
   insecure: false
-  # Get custom fields when using virtualmachine metric set. Default false.
+  # Get custom fields when using virtualmachine metricset. Default false.
   # get_custom_fields: false
diff --git a/metricbeat/module/vsphere/_meta/config.yml b/metricbeat/module/vsphere/_meta/config.yml
@@ -7,13 +7,23 @@
   #  - network
   #  - resourcepool
   #  - virtualmachine
-  # Real-time data collection – An ESXi Server collects data for each performance counter every 20 seconds.
+
+  # Real-time data collection – An ESXi Server collects data for each performance counter every 20 seconds by default.
+  # Supported Periods:
+  # The Datastore and Host metricsets support performance data collection using the vSphere performance API.
+  # Since the performance API has usage restrictions based on data collection intervals,
+  # users should ensure that the period is configured optimally to receive real-time data.
+  # users can still collect summary metrics if performance metrics are not supported for the configured instance.
+  # This configuration can be determined based on the Data Collection Intervals and Data Collection Levels.
+  # Reference Links:
+  # Data Collection Intervals: https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.monitoring.doc/GUID-247646EA-A04B-411A-8DD4-62A3DCFCF49B.html
+  # Data Collection Levels: https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.monitoring.doc/GUID-25800DE4-68E5-41CC-82D9-8811E27924BC.html
   period: 20s
   hosts: ["https://localhost/sdk"]
 
   username: "user"
   password: "password"
   # If insecure is true, don't verify the server's certificate chain
   insecure: false
-  # Get custom fields when using virtualmachine metric set. Default false.
+  # Get custom fields when using virtualmachine metricset. Default false.
   # get_custom_fields: false
diff --git a/metricbeat/module/vsphere/_meta/docs.asciidoc b/metricbeat/module/vsphere/_meta/docs.asciidoc
@@ -1,11 +1,76 @@
-The vSphere module uses the https://github.com/vmware/govmomi[Govmomi] library to collect metrics from any Vmware SDK URL (ESXi/VCenter). This library is built for and tested against ESXi and vCenter 5.5, 6.0 and 6.5.
+The vSphere module uses the https://github.com/vmware/govmomi[Govmomi] library to collect metrics from any VMware SDK URL (ESXi/VCenter).
 
-By default it enables the metricsets `cluster`, `datastore`, `datastorecluster`, `host`, `network`, `resourcepool` and `virtualmachine`.
+This module has been tested against ESXi and vCenter versions 5.5, 6.0, 6.5, and 7.0.3.
+
+By default, the vSphere module enables the following metricsets:
+
+1. cluster
+
+2. datastore
+
+3. datastorecluster
+
+4. host
+
+5. network
+
+6. resourcepool
+
+7. virtualmachine
+
+[float]
+=== Supported Periods:
+The Datastore and Host metricsets support performance data collection using the vSphere performance API. Given that the performance API imposes usage restrictions based on data collection intervals, users should configure the period optimally to ensure the receipt of real-time data. This configuration can be determined based on the https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.monitoring.doc/GUID-247646EA-A04B-411A-8DD4-62A3DCFCF49B.html[Data Collection Intervals] and https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.monitoring.doc/GUID-25800DE4-68E5-41CC-82D9-8811E27924BC.html[Data Collection Levels].
+
+[IMPORTANT]
+
+Only host and datastore metricsets have limitation of system configured period from vSphere instance. Users can still collect summary metrics if performance metrics are not supported for the configured instance.
+
+[float]
+==== Real-time data collection default interval:
+- 20s
+
+[float]
+==== Historical data collection default intervals:
+- 300s
+- 1800s
+- 7200s
+- 86400s
+
+[float]
+=== Example:
+If you need to configure multiple metricsets with different periods, you can achieve this by setting up multiple vSphere modules with different metricsets as demonstrated below:
+
+[source,yaml]
+----
+- module: vsphere
+  metricsets:
+   - cluster
+   - datastorecluster
+   - network
+   - resourcepool
+   - virtualmachine
+  period: 10s
+  hosts: ["https://localhost/sdk"]
+  username: "user"
+  password: "password"
+  insecure: false
+
+- module: vsphere
+  metricsets:
+   - datastore
+   - host
+  period: 300s
+  hosts: ["https://localhost/sdk"]
+  username: "user"
+  password: "password"
+  insecure: false
+----
 
 [float]
 === Dashboard
 
-The vsphere module comes with a predefined dashboard. For example:
+The vSphere module includes a predefined dashboard. For example:
 
 image::./images/metricbeat_vsphere_dashboard.png[]
 image::./images/metricbeat_vsphere_vm_dashboard.png[]
diff --git a/metricbeat/module/vsphere/cluster/cluster.go b/metricbeat/module/vsphere/cluster/cluster.go
@@ -75,7 +75,7 @@ func (m *ClusterMetricSet) Fetch(ctx context.Context, reporter mb.ReporterV2) er
 	}
 	defer func() {
 		if err := client.Logout(ctx); err != nil {
-			m.Logger().Debug(fmt.Errorf("error trying to logout from vSphere: %w", err))
+			m.Logger().Errorf("error trying to logout from vSphere: %v", err)
 		}
 	}()
 
@@ -91,7 +91,7 @@ func (m *ClusterMetricSet) Fetch(ctx context.Context, reporter mb.ReporterV2) er
 
 	defer func() {
 		if err := v.Destroy(ctx); err != nil {
-			m.Logger().Errorf("error trying to destroy view from vSphere: %w", err)
+			m.Logger().Errorf("error trying to destroy view from vSphere: %v", err)
 		}
 	}()