[Metricbeat][vSphere] Support for configurable IntervalId for performance API

CHANGELOG.next.asciidoc

@@ -299,6 +299,7 @@ https://github.com/elastic/beats/compare/v8.8.1\...main[Check the HEAD diff]
 - Disable event normalization for netflow input {pull}40635[40635]
 - Allow attribute selection in the Active Directory entity analytics provider. {issue}40482[40482] {pull}40662[40662]
 - Improve error quality when CEL program does not correctly return an events array. {pull}40580[40580]
+- Add support for period based intervalID in vSphere host and datastore metricsets {pull}40678[40678]
 
 *Auditbeat*
 

metricbeat/docs/modules/vsphere.asciidoc

@@ -9,14 +9,79 @@ This file is generated! See scripts/mage/docs_collector.go
 [[metricbeat-module-vsphere]]
 == vSphere module
 
-The vSphere module uses the https://github.com/vmware/govmomi[Govmomi] library to collect metrics from any Vmware SDK URL (ESXi/VCenter). This library is built for and tested against ESXi and vCenter 5.5, 6.0 and 6.5.
+The vSphere module uses the https://github.com/vmware/govmomi[Govmomi] library to collect metrics from any VMware SDK URL (ESXi/VCenter).
 
-By default it enables the metricsets `cluster`, `datastore`, `datastorecluster`, `host`, `network`, `resourcepool` and `virtualmachine`.
+This module has been tested against ESXi and vCenter versions 5.5, 6.0, 6.5, and 7.0.3.
+
+By default, the vSphere module enables the following metricsets:
+
+1. cluster
+
+2. datastore
+
+3. datastorecluster
+
+4. host
+
+5. network
+
+6. resourcepool
+
+7. virtualmachine
+
+[float]
+=== Supported Periods:
+The Datastore and Host metricsets support performance data collection using the vSphere performance API. Given that the performance API imposes usage restrictions based on data collection intervals, users should configure the period optimally to ensure the receipt of real-time data. This configuration can be determined based on the https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.monitoring.doc/GUID-247646EA-A04B-411A-8DD4-62A3DCFCF49B.html[Data Collection Intervals] and https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.monitoring.doc/GUID-25800DE4-68E5-41CC-82D9-8811E27924BC.html[Data Collection Levels].
+
+[IMPORTANT]
+
+Only host and datastore metricsets have limitation of period restrictions.
+
+[float]
+==== Real-time data collection interval:
+- 20s
+
+[float]
+==== Historical data collection interval:
+- 300s
+- 1800s
+- 7200s
+- 86400s
+
+[float]
+=== Example:
+If you need to configure multiple metricsets with different periods, you can achieve this by setting up multiple vSphere modules with different metricsets as demonstrated below:
+
+[source,yaml]
+----
+- module: vsphere
+  metricsets:
+   - cluster
+   - datastorecluster
+   - network
+   - resourcepool
+   - virtualmachine
+  period: 10s
+  hosts: ["https://localhost/sdk"]
+  username: "user"
+  password: "password"
+  insecure: false
+
+- module: vsphere
+  metricsets:
+   - datastore
+   - host
+  period: 300s
+  hosts: ["https://localhost/sdk"]
+  username: "user"
+  password: "password"
+  insecure: false
+----
 
 [float]
 === Dashboard
 
-The vsphere module comes with a predefined dashboard. For example:
+The vSphere module includes a predefined dashboard. For example:
 
 image::./images/metricbeat_vsphere_dashboard.png[]
 image::./images/metricbeat_vsphere_vm_dashboard.png[]
@@ -36,15 +101,24 @@ metricbeat.modules:
 - module: vsphere
   enabled: true
   metricsets: ["cluster", "datastore", "datastorecluster", "host", "network", "resourcepool", "virtualmachine"]
+
   # Real-time data collection – An ESXi Server collects data for each performance counter every 20 seconds.
+  # Supported Periods:
+  # The Datastore and Host metricsets support performance data collection using the vSphere performance API.
+  # Since the performance API has usage restrictions based on data collection intervals,
+  # users should ensure that the period is configured optimally to receive real-time data.
+  # This configuration can be determined based on the Data Collection Intervals and Data Collection Levels.
+  # Reference Links:
+  # Data Collection Intervals: https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.monitoring.doc/GUID-247646EA-A04B-411A-8DD4-62A3DCFCF49B.html
+  # Data Collection Levels: https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.monitoring.doc/GUID-25800DE4-68E5-41CC-82D9-8811E27924BC.html
   period: 20s
   hosts: ["https://localhost/sdk"]
 
   username: "user"
   password: "password"
   # If insecure is true, don't verify the server's certificate chain
   insecure: false
-  # Get custom fields when using virtualmachine metric set. Default false.
+  # Get custom fields when using virtualmachine metricset. Default false.
   # get_custom_fields: false
 ----
 

metricbeat/metricbeat.reference.yml

@@ -1008,15 +1008,24 @@ metricbeat.modules:
 - module: vsphere
   enabled: true
   metricsets: ["cluster", "datastore", "datastorecluster", "host", "network", "resourcepool", "virtualmachine"]
+
   # Real-time data collection – An ESXi Server collects data for each performance counter every 20 seconds.
+  # Supported Periods:
+  # The Datastore and Host metricsets support performance data collection using the vSphere performance API.
+  # Since the performance API has usage restrictions based on data collection intervals,
+  # users should ensure that the period is configured optimally to receive real-time data.
+  # This configuration can be determined based on the Data Collection Intervals and Data Collection Levels.
+  # Reference Links:
+  # Data Collection Intervals: https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.monitoring.doc/GUID-247646EA-A04B-411A-8DD4-62A3DCFCF49B.html
+  # Data Collection Levels: https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.monitoring.doc/GUID-25800DE4-68E5-41CC-82D9-8811E27924BC.html
   period: 20s
   hosts: ["https://localhost/sdk"]
 
   username: "user"
   password: "password"
   # If insecure is true, don't verify the server's certificate chain
   insecure: false
-  # Get custom fields when using virtualmachine metric set. Default false.
+  # Get custom fields when using virtualmachine metricset. Default false.
   # get_custom_fields: false
 
 #------------------------------- Windows Module -------------------------------

metricbeat/module/vsphere/_meta/config.reference.yml

@@ -1,13 +1,22 @@
 - module: vsphere
   enabled: true
   metricsets: ["cluster", "datastore", "datastorecluster", "host", "network", "resourcepool", "virtualmachine"]
+
   # Real-time data collection – An ESXi Server collects data for each performance counter every 20 seconds.
+  # Supported Periods:
+  # The Datastore and Host metricsets support performance data collection using the vSphere performance API.
+  # Since the performance API has usage restrictions based on data collection intervals,
+  # users should ensure that the period is configured optimally to receive real-time data.
+  # This configuration can be determined based on the Data Collection Intervals and Data Collection Levels.
+  # Reference Links:
+  # Data Collection Intervals: https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.monitoring.doc/GUID-247646EA-A04B-411A-8DD4-62A3DCFCF49B.html
+  # Data Collection Levels: https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.monitoring.doc/GUID-25800DE4-68E5-41CC-82D9-8811E27924BC.html
   period: 20s
   hosts: ["https://localhost/sdk"]
 
   username: "user"
   password: "password"
   # If insecure is true, don't verify the server's certificate chain
   insecure: false
-  # Get custom fields when using virtualmachine metric set. Default false.
+  # Get custom fields when using virtualmachine metricset. Default false.
   # get_custom_fields: false

metricbeat/module/vsphere/_meta/config.yml

@@ -7,13 +7,22 @@
   #  - network
   #  - resourcepool
   #  - virtualmachine
+
   # Real-time data collection – An ESXi Server collects data for each performance counter every 20 seconds.
+  # Supported Periods:
+  # The Datastore and Host metricsets support performance data collection using the vSphere performance API.
+  # Since the performance API has usage restrictions based on data collection intervals,
+  # users should ensure that the period is configured optimally to receive real-time data.
+  # This configuration can be determined based on the Data Collection Intervals and Data Collection Levels.
+  # Reference Links:
+  # Data Collection Intervals: https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.monitoring.doc/GUID-247646EA-A04B-411A-8DD4-62A3DCFCF49B.html
+  # Data Collection Levels: https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.monitoring.doc/GUID-25800DE4-68E5-41CC-82D9-8811E27924BC.html
   period: 20s
   hosts: ["https://localhost/sdk"]
 
   username: "user"
   password: "password"
   # If insecure is true, don't verify the server's certificate chain
   insecure: false
-  # Get custom fields when using virtualmachine metric set. Default false.
+  # Get custom fields when using virtualmachine metricset. Default false.
   # get_custom_fields: false

metricbeat/module/vsphere/_meta/docs.asciidoc

@@ -1,11 +1,76 @@
-The vSphere module uses the https://github.com/vmware/govmomi[Govmomi] library to collect metrics from any Vmware SDK URL (ESXi/VCenter). This library is built for and tested against ESXi and vCenter 5.5, 6.0 and 6.5.
+The vSphere module uses the https://github.com/vmware/govmomi[Govmomi] library to collect metrics from any VMware SDK URL (ESXi/VCenter).
 
-By default it enables the metricsets `cluster`, `datastore`, `datastorecluster`, `host`, `network`, `resourcepool` and `virtualmachine`.
+This module has been tested against ESXi and vCenter versions 5.5, 6.0, 6.5, and 7.0.3.
+
+By default, the vSphere module enables the following metricsets:
+
+1. cluster
+
+2. datastore
+
+3. datastorecluster
+
+4. host
+
+5. network
+
+6. resourcepool
+
+7. virtualmachine
+
+[float]
+=== Supported Periods:
+The Datastore and Host metricsets support performance data collection using the vSphere performance API. Given that the performance API imposes usage restrictions based on data collection intervals, users should configure the period optimally to ensure the receipt of real-time data. This configuration can be determined based on the https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.monitoring.doc/GUID-247646EA-A04B-411A-8DD4-62A3DCFCF49B.html[Data Collection Intervals] and https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.monitoring.doc/GUID-25800DE4-68E5-41CC-82D9-8811E27924BC.html[Data Collection Levels].
+
+[IMPORTANT]
+
+Only host and datastore metricsets have limitation of period restrictions.
+
+[float]
+==== Real-time data collection interval:
+- 20s
+
+[float]
+==== Historical data collection interval:
+- 300s
+- 1800s
+- 7200s
+- 86400s
+
+[float]
+=== Example:
+If you need to configure multiple metricsets with different periods, you can achieve this by setting up multiple vSphere modules with different metricsets as demonstrated below:
+
+[source,yaml]
+----
+- module: vsphere
+  metricsets:
+   - cluster
+   - datastorecluster
+   - network
+   - resourcepool
+   - virtualmachine
+  period: 10s
+  hosts: ["https://localhost/sdk"]
+  username: "user"
+  password: "password"
+  insecure: false
+
+- module: vsphere
+  metricsets:
+   - datastore
+   - host
+  period: 300s
+  hosts: ["https://localhost/sdk"]
+  username: "user"
+  password: "password"
+  insecure: false
+----
 
 [float]
 === Dashboard
 
-The vsphere module comes with a predefined dashboard. For example:
+The vSphere module includes a predefined dashboard. For example:
 
 image::./images/metricbeat_vsphere_dashboard.png[]
 image::./images/metricbeat_vsphere_vm_dashboard.png[]