Add s3_request metricset for AWS module (#10949)

* Add s3_request metricset for AWS module * Add changelog * Change metric names and add a check for timestamp * Add ms unit to latency metrics
elastic · Feb 28, 2019 · e03f966 · e03f966
1 parent 2e551f4
commit e03f966
Show file tree

Hide file tree

Showing 21 changed files with 790 additions and 36 deletions.
diff --git a/CHANGELOG.next.asciidoc b/CHANGELOG.next.asciidoc
@@ -312,6 +312,7 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
 - Add Consul Metricbeat module with Agent Metricset {pull}8631[8631]
 - Add filters and pie chart for AWS EC2 dashboard. {pull}10596[10596]
 - Add AWS SQS metricset. {pull}10684[10684] {issue}10053[10053]
+- Add AWS s3_request metricset. {pull}10949[10949] {issue}10055[10055]
 
 *Packetbeat*
 

diff --git a/metricbeat/docs/fields.asciidoc b/metricbeat/docs/fields.asciidoc
@@ -1083,6 +1083,183 @@ type: integer
 The state of the instance (pending | running | shutting-down | terminated | stopping | stopped).
 
 
+--
+
+[float]
+== s3_request fields
+
+`s3_request` contains request metrics that were scraped from AWS CloudWatch which contains monitoring metrics sent by AWS S3.
+
+
+
+*`aws.s3_request.bucket.name`*::
++
+--
+type: keyword
+
+Name of a S3 bucket.
+
+
+--
+
+*`aws.s3_request.requests.total`*::
++
+--
+type: long
+
+The total number of HTTP requests made to an Amazon S3 bucket, regardless of type.
+
+
+--
+
+*`aws.s3_request.requests.get`*::
++
+--
+type: long
+
+The number of HTTP GET requests made for objects in an Amazon S3 bucket.
+
+
+--
+
+*`aws.s3_request.requests.put`*::
++
+--
+type: long
+
+The number of HTTP PUT requests made for objects in an Amazon S3 bucket.
+
+
+--
+
+*`aws.s3_request.requests.delete`*::
++
+--
+type: long
+
+The number of HTTP DELETE requests made for objects in an Amazon S3 bucket.
+
+
+--
+
+*`aws.s3_request.requests.head`*::
++
+--
+type: long
+
+The number of HTTP HEAD requests made to an Amazon S3 bucket.
+
+
+--
+
+*`aws.s3_request.requests.post`*::
++
+--
+type: long
+
+The number of HTTP POST requests made to an Amazon S3 bucket.
+
+
+--
+
+*`aws.s3_request.requests.select`*::
++
+--
+type: long
+
+The number of Amazon S3 SELECT Object Content requests made for objects in an Amazon S3 bucket.
+
+
+--
+
+*`aws.s3_request.requests.select.scanned.bytes`*::
++
+--
+type: scaled_float
+
+The number of bytes of data scanned with Amazon S3 SELECT Object Content requests in an Amazon S3 bucket.
+
+
+--
+
+*`aws.s3_request.requests.select.returned.bytes`*::
++
+--
+type: scaled_float
+
+The number of bytes of data returned with Amazon S3 SELECT Object Content requests in an Amazon S3 bucket.
+
+
+--
+
+*`aws.s3_request.requests.list`*::
++
+--
+type: long
+
+The number of HTTP requests that list the contents of a bucket.
+
+
+--
+
+*`aws.s3_request.downloaded.bytes`*::
++
+--
+type: scaled_float
+
+The number bytes downloaded for requests made to an Amazon S3 bucket, where the response includes a body.
+
+
+--
+
+*`aws.s3_request.uploaded.bytes`*::
++
+--
+type: scaled_float
+
+The number bytes uploaded that contain a request body, made to an Amazon S3 bucket.
+
+
+--
+
+*`aws.s3_request.errors.4xx`*::
++
+--
+type: long
+
+The number of HTTP 4xx client error status code requests made to an Amazon S3 bucket with a value of either 0 or 1.
+
+
+--
+
+*`aws.s3_request.errors.5xx`*::
++
+--
+type: long
+
+The number of HTTP 5xx server error status code requests made to an Amazon S3 bucket with a value of either 0 or 1.
+
+
+--
+
+*`aws.s3_request.latency.first_byte.ms`*::
++
+--
+type: long
+
+The per-request time from the complete request being received by an Amazon S3 bucket to when the response starts to be returned.
+
+
+--
+
+*`aws.s3_request.latency.total_request.ms`*::
++
+--
+type: long
+
+The elapsed per-request time from the first byte received to the last byte sent to an Amazon S3 bucket.
+
+
 --
 
 [float]

diff --git a/metricbeat/docs/modules/aws.asciidoc b/metricbeat/docs/modules/aws.asciidoc
@@ -9,7 +9,7 @@ This module periodically fetches monitoring metrics from AWS Cloudwatch using
 https://docs.aws.amazon.com/AmazonCloudWatch/latest/APIReference/API_GetMetricData.html[GetMetricData API] for running
 EC2 instances. Note: extra AWS charges on GetMetricData API requests will be generated by this module.
 
-The default metricsets are `ec2` and `sqs`.
+The default metricsets are `ec2`, `sqs` and `s3_request`.
 
 [float]
 === Module-specific configuration notes
@@ -51,6 +51,7 @@ metricbeat.modules:
   default_region: '${AWS_REGION:us-west-1}'
 ----
 
+=== ec2 metricset
 By default, Amazon EC2 sends metric data to CloudWatch every 5 minutes. With this basic monitoring, `period` in aws module
 configuration should be larger or equal than `300s`. If `period` is set to be less than `300s`, the same cloudwatch metrics
 will be collected more than once which will cause extra fees without getting more granular metrics. For example, in `US East (N. Virginia)` region, it costs
@@ -63,13 +64,23 @@ larger than `60s`. Since AWS sends metric data to CloudWatch in 1-minute periods
 than `60s` will cause extra API requests which means extra charges on AWS. To avoid unnecessary charges, `period` is
 preferred to be set to `60s` or multiples of `60s`, such as `120s` and `180s`.
 
-Since cloudWatch metrics for Amazon SQS queues are automatically collected and pushed to CloudWatch every 5 minutes,
-`period` for `sqs` is recommended to set to `300s` or multiples of `300s`.
-
-The AWS module comes with a predefined dashboard. For example:
+The ec2 metricset comes with a predefined dashboard. For example:
 
 image::./images/metricbeat-aws-ec2-overview.png[]
 
+=== sqs metricset
+Cloudwatch metrics for Amazon SQS queues are automatically collected and pushed to CloudWatch every 5 minutes,
+the `period` for `sqs` metricset is recommended to be `300s` or multiples of `300s`.
+
+=== s3_request metricset
+Request metrics are available at 1-minute intervals with additional charges. The s3_request metricset will give more
+granular data to track S3 bucket usage. The `period` for `s3_request` metricset can be set to `60s` or multiples of `60s`.
+But because of the extra charges for querying these metrics, the `period` is recommended to set to `86400s`. The user can
+always adjust this to the granularity they want. Request metrics are not enabled by default for S3 buckets. Please see
+https://docs.aws.amazon.com/AmazonS3/latest/user-guide/configure-metrics.html[How to
+Configure Request Metrics for S3] for instructions on how to enable request metrics for
+each S3 bucket.
+
 
 [float]
 === Example configuration
@@ -89,6 +100,14 @@ metricbeat.modules:
   secret_access_key: '${AWS_SECRET_ACCESS_KEY:""}'
   session_token: '${AWS_SESSION_TOKEN:""}'
   default_region: '${AWS_REGION:us-west-1}'
+- module: aws
+  period: 86400s
+  metricsets:
+    - "s3_request"
+  access_key_id: '${AWS_ACCESS_KEY_ID:""}'
+  secret_access_key: '${AWS_SECRET_ACCESS_KEY:""}'
+  session_token: '${AWS_SESSION_TOKEN:""}'
+  default_region: '${AWS_REGION:us-west-1}'
 ----
 
 [float]
@@ -98,9 +117,13 @@ The following metricsets are available:
 
 * <<metricbeat-metricset-aws-ec2,ec2>>
 
+* <<metricbeat-metricset-aws-s3_request,s3_request>>
+
 * <<metricbeat-metricset-aws-sqs,sqs>>
 
 include::aws/ec2.asciidoc[]
 
+include::aws/s3_request.asciidoc[]
+
 include::aws/sqs.asciidoc[]
 
diff --git a/metricbeat/docs/modules/aws/s3_request.asciidoc b/metricbeat/docs/modules/aws/s3_request.asciidoc
@@ -0,0 +1,23 @@
+////
+This file is generated! See scripts/docs_collector.py
+////
+
+[[metricbeat-metricset-aws-s3_request]]
+=== aws s3_request metricset
+
+beta[]
+
+include::../../../../x-pack/metricbeat/module/aws/s3_request/_meta/docs.asciidoc[]
+
+
+==== Fields
+
+For a description of each field in the metricset, see the
+<<exported-fields-aws,exported fields>> section.
+
+Here is an example document generated by this metricset:
+
+[source,json]
+----
+include::../../../../x-pack/metricbeat/module/aws/s3_request/_meta/data.json[]
+----
diff --git a/metricbeat/docs/modules_list.asciidoc b/metricbeat/docs/modules_list.asciidoc
@@ -10,7 +10,8 @@ This file is generated! See scripts/docs_collector.py
 |<<metricbeat-module-apache,Apache>>     |image:./images/icon-yes.png[Prebuilt dashboards are available]    |  
 .1+| .1+|  |<<metricbeat-metricset-apache-status,status>>   
 |<<metricbeat-module-aws,aws>>     |image:./images/icon-yes.png[Prebuilt dashboards are available]    |  
-.2+| .2+|  |<<metricbeat-metricset-aws-ec2,ec2>>   
+.3+| .3+|  |<<metricbeat-metricset-aws-ec2,ec2>>   
+|<<metricbeat-metricset-aws-s3_request,s3_request>> beta[]  
 |<<metricbeat-metricset-aws-sqs,sqs>> beta[]  
 |<<metricbeat-module-ceph,Ceph>>     |image:./images/icon-no.png[No prebuilt dashboards]    |  
 .7+| .7+|  |<<metricbeat-metricset-ceph-cluster_disk,cluster_disk>>   

diff --git a/x-pack/metricbeat/include/list.go b/x-pack/metricbeat/include/list.go
diff --git a/x-pack/metricbeat/metricbeat.reference.yml b/x-pack/metricbeat/metricbeat.reference.yml
@@ -157,6 +157,14 @@ metricbeat.modules:
   secret_access_key: '${AWS_SECRET_ACCESS_KEY:""}'
   session_token: '${AWS_SESSION_TOKEN:""}'
   default_region: '${AWS_REGION:us-west-1}'
+- module: aws
+  period: 86400s
+  metricsets:
+    - "s3_request"
+  access_key_id: '${AWS_ACCESS_KEY_ID:""}'
+  secret_access_key: '${AWS_SECRET_ACCESS_KEY:""}'
+  session_token: '${AWS_SESSION_TOKEN:""}'
+  default_region: '${AWS_REGION:us-west-1}'
 
 #--------------------------------- Ceph Module ---------------------------------
 - module: ceph

diff --git a/x-pack/metricbeat/module/aws/_meta/config.yml b/x-pack/metricbeat/module/aws/_meta/config.yml
@@ -7,3 +7,11 @@
   secret_access_key: '${AWS_SECRET_ACCESS_KEY:""}'
   session_token: '${AWS_SESSION_TOKEN:""}'
   default_region: '${AWS_REGION:us-west-1}'
+- module: aws
+  period: 86400s
+  metricsets:
+    - "s3_request"
+  access_key_id: '${AWS_ACCESS_KEY_ID:""}'
+  secret_access_key: '${AWS_SECRET_ACCESS_KEY:""}'
+  session_token: '${AWS_SESSION_TOKEN:""}'
+  default_region: '${AWS_REGION:us-west-1}'
diff --git a/x-pack/metricbeat/module/aws/_meta/docs.asciidoc b/x-pack/metricbeat/module/aws/_meta/docs.asciidoc
@@ -2,7 +2,7 @@ This module periodically fetches monitoring metrics from AWS Cloudwatch using
 https://docs.aws.amazon.com/AmazonCloudWatch/latest/APIReference/API_GetMetricData.html[GetMetricData API] for running
 EC2 instances. Note: extra AWS charges on GetMetricData API requests will be generated by this module.
 
-The default metricsets are `ec2` and `sqs`.
+The default metricsets are `ec2`, `sqs` and `s3_request`.
 
 [float]
 === Module-specific configuration notes
@@ -44,6 +44,7 @@ metricbeat.modules:
   default_region: '${AWS_REGION:us-west-1}'
 ----
 
+=== ec2 metricset
 By default, Amazon EC2 sends metric data to CloudWatch every 5 minutes. With this basic monitoring, `period` in aws module
 configuration should be larger or equal than `300s`. If `period` is set to be less than `300s`, the same cloudwatch metrics
 will be collected more than once which will cause extra fees without getting more granular metrics. For example, in `US East (N. Virginia)` region, it costs
@@ -56,9 +57,19 @@ larger than `60s`. Since AWS sends metric data to CloudWatch in 1-minute periods
 than `60s` will cause extra API requests which means extra charges on AWS. To avoid unnecessary charges, `period` is
 preferred to be set to `60s` or multiples of `60s`, such as `120s` and `180s`.
 
-Since cloudWatch metrics for Amazon SQS queues are automatically collected and pushed to CloudWatch every 5 minutes,
-`period` for `sqs` is recommended to set to `300s` or multiples of `300s`.
-
-The AWS module comes with a predefined dashboard. For example:
+The ec2 metricset comes with a predefined dashboard. For example:
 
 image::./images/metricbeat-aws-ec2-overview.png[]
+
+=== sqs metricset
+Cloudwatch metrics for Amazon SQS queues are automatically collected and pushed to CloudWatch every 5 minutes,
+the `period` for `sqs` metricset is recommended to be `300s` or multiples of `300s`.
+
+=== s3_request metricset
+Request metrics are available at 1-minute intervals with additional charges. The s3_request metricset will give more
+granular data to track S3 bucket usage. The `period` for `s3_request` metricset can be set to `60s` or multiples of `60s`.
+But because of the extra charges for querying these metrics, the `period` is recommended to set to `86400s`. The user can
+always adjust this to the granularity they want. Request metrics are not enabled by default for S3 buckets. Please see
+https://docs.aws.amazon.com/AmazonS3/latest/user-guide/configure-metrics.html[How to
+Configure Request Metrics for S3] for instructions on how to enable request metrics for
+each S3 bucket.
diff --git a/x-pack/metricbeat/module/aws/aws.go b/x-pack/metricbeat/module/aws/aws.go
@@ -139,3 +139,13 @@ func convertPeriodToDuration(period string) (string, int, error) {
 		return duration, numberPeriod, err
 	}
 }
+
+// StringInSlice checks if a string is already exists in list
+func StringInSlice(str string, list []string) bool {
+	for _, v := range list {
+		if v == str {
+			return true
+		}
+	}
+	return false
+}