Add AWS s3_daily_storage metricset

CHANGELOG.next.asciidoc

@@ -317,6 +317,7 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
 - Add filters and pie chart for AWS EC2 dashboard. {pull}10596[10596]
 - Add AWS SQS metricset. {pull}10684[10684] {issue}10053[10053]
 - Add AWS s3_request metricset. {pull}10949[10949] {issue}10055[10055]
+- Add s3_daily_storage metricset. {pull}10940[10940] {issue}10055[10055]
 
 *Packetbeat*
 

metricbeat/docs/fields.asciidoc

@@ -1083,6 +1083,43 @@ type: integer
 The state of the instance (pending | running | shutting-down | terminated | stopping | stopped).
 
 
+--
+
+[float]
+== s3_daily_storage fields
+
+`s3_daily_storage` contains the daily storage metrics that were scraped from AWS CloudWatch which contains monitoring metrics sent by AWS S3.
+
+
+
+*`aws.s3_daily_storage.bucket.name`*::
++
+--
+type: keyword
+
+Name of a S3 bucket.
+
+
+--
+
+*`aws.s3_daily_storage.bucket.size.bytes`*::
++
+--
+type: scaled_float
+
+The amount of data in bytes stored in a bucket.
+
+
+--
+
+*`aws.s3_daily_storage.number_of_objects`*::
++
+--
+type: long
+
+The total number of objects stored in a bucket for all storage classes.
+
+
 --
 
 [float]
@@ -1172,7 +1209,7 @@ The number of Amazon S3 SELECT Object Content requests made for objects in an Am
 
 --
 
-*`aws.s3_request.requests.select.scanned.bytes`*::
+*`aws.s3_request.requests.select_scanned.bytes`*::
 +
 --
 type: scaled_float
@@ -1182,7 +1219,7 @@ The number of bytes of data scanned with Amazon S3 SELECT Object Content request
 
 --
 
-*`aws.s3_request.requests.select.returned.bytes`*::
+*`aws.s3_request.requests.select_returned.bytes`*::
 +
 --
 type: scaled_float

metricbeat/docs/modules/aws.asciidoc

@@ -9,7 +9,7 @@ This module periodically fetches monitoring metrics from AWS Cloudwatch using
 https://docs.aws.amazon.com/AmazonCloudWatch/latest/APIReference/API_GetMetricData.html[GetMetricData API] for running
 EC2 instances. Note: extra AWS charges on GetMetricData API requests will be generated by this module.
 
-The default metricsets are `ec2`, `sqs` and `s3_request`.
+The default metricsets are `ec2`, `sqs`, `s3_request` and `s3_daily_storage`.
 
 [float]
 === Module-specific configuration notes
@@ -72,8 +72,11 @@ image::./images/metricbeat-aws-ec2-overview.png[]
 Cloudwatch metrics for Amazon SQS queues are automatically collected and pushed to CloudWatch every 5 minutes,
 the `period` for `sqs` metricset is recommended to be `300s` or multiples of `300s`.
 
-=== s3_request metricset
-Request metrics are available at 1-minute intervals with additional charges. The s3_request metricset will give more
+=== s3_request and s3_daily_storage metricset
+Daily storage metrics for S3 buckets are reported once per day with no additional cost. Since they are daily metrics,
+`period` for `s3_daily_storage` metricset is recommended to be `86400s` or multiples of `86400s`.
+Request metrics are available
+at 1-minute intervals with additional charges. The s3_request metricset will give more
 granular data to track S3 bucket usage. The `period` for `s3_request` metricset can be set to `60s` or multiples of `60s`.
 But because of the extra charges for querying these metrics, the `period` is recommended to set to `86400s`. The user can
 always adjust this to the granularity they want. Request metrics are not enabled by default for S3 buckets. Please see
@@ -104,6 +107,7 @@ metricbeat.modules:
   period: 86400s
   metricsets:
     - "s3_request"
+    - "s3_daily_storage"
   access_key_id: '${AWS_ACCESS_KEY_ID:""}'
   secret_access_key: '${AWS_SECRET_ACCESS_KEY:""}'
   session_token: '${AWS_SESSION_TOKEN:""}'
@@ -117,12 +121,16 @@ The following metricsets are available:
 
 * <<metricbeat-metricset-aws-ec2,ec2>>
 
+* <<metricbeat-metricset-aws-s3_daily_storage,s3_daily_storage>>
+
 * <<metricbeat-metricset-aws-s3_request,s3_request>>
 
 * <<metricbeat-metricset-aws-sqs,sqs>>
 
 include::aws/ec2.asciidoc[]
 
+include::aws/s3_daily_storage.asciidoc[]
+
 include::aws/s3_request.asciidoc[]
 
 include::aws/sqs.asciidoc[]

metricbeat/docs/modules/aws/s3_daily_storage.asciidoc

@@ -0,0 +1,23 @@
+////
+This file is generated! See scripts/docs_collector.py
+////
+
+[[metricbeat-metricset-aws-s3_daily_storage]]
+=== aws s3_daily_storage metricset
+
+beta[]
+
+include::../../../../x-pack/metricbeat/module/aws/s3_daily_storage/_meta/docs.asciidoc[]
+
+
+==== Fields
+
+For a description of each field in the metricset, see the
+<<exported-fields-aws,exported fields>> section.
+
+Here is an example document generated by this metricset:
+
+[source,json]
+----
+include::../../../../x-pack/metricbeat/module/aws/s3_daily_storage/_meta/data.json[]
+----

metricbeat/docs/modules_list.asciidoc

@@ -10,7 +10,8 @@ This file is generated! See scripts/docs_collector.py
 |<<metricbeat-module-apache,Apache>>     |image:./images/icon-yes.png[Prebuilt dashboards are available]    |  
 .1+| .1+|  |<<metricbeat-metricset-apache-status,status>>   
 |<<metricbeat-module-aws,aws>>     |image:./images/icon-yes.png[Prebuilt dashboards are available]    |  
-.3+| .3+|  |<<metricbeat-metricset-aws-ec2,ec2>>   
+.4+| .4+|  |<<metricbeat-metricset-aws-ec2,ec2>>   
+|<<metricbeat-metricset-aws-s3_daily_storage,s3_daily_storage>> beta[]  
 |<<metricbeat-metricset-aws-s3_request,s3_request>> beta[]  
 |<<metricbeat-metricset-aws-sqs,sqs>> beta[]  
 |<<metricbeat-module-ceph,Ceph>>     |image:./images/icon-no.png[No prebuilt dashboards]    |  

x-pack/metricbeat/metricbeat.reference.yml

@@ -161,6 +161,7 @@ metricbeat.modules:
   period: 86400s
   metricsets:
     - "s3_request"
+    - "s3_daily_storage"
   access_key_id: '${AWS_ACCESS_KEY_ID:""}'
   secret_access_key: '${AWS_SECRET_ACCESS_KEY:""}'
   session_token: '${AWS_SESSION_TOKEN:""}'

x-pack/metricbeat/module/aws/_meta/config.yml

@@ -11,6 +11,7 @@
   period: 86400s
   metricsets:
     - "s3_request"
+    - "s3_daily_storage"
   access_key_id: '${AWS_ACCESS_KEY_ID:""}'
   secret_access_key: '${AWS_SECRET_ACCESS_KEY:""}'
   session_token: '${AWS_SESSION_TOKEN:""}'

x-pack/metricbeat/module/aws/_meta/docs.asciidoc

@@ -2,7 +2,7 @@ This module periodically fetches monitoring metrics from AWS Cloudwatch using
 https://docs.aws.amazon.com/AmazonCloudWatch/latest/APIReference/API_GetMetricData.html[GetMetricData API] for running
 EC2 instances. Note: extra AWS charges on GetMetricData API requests will be generated by this module.
 
-The default metricsets are `ec2`, `sqs` and `s3_request`.
+The default metricsets are `ec2`, `sqs`, `s3_request` and `s3_daily_storage`.
 
 [float]
 === Module-specific configuration notes
@@ -65,8 +65,11 @@ image::./images/metricbeat-aws-ec2-overview.png[]
 Cloudwatch metrics for Amazon SQS queues are automatically collected and pushed to CloudWatch every 5 minutes,
 the `period` for `sqs` metricset is recommended to be `300s` or multiples of `300s`.
 
-=== s3_request metricset
-Request metrics are available at 1-minute intervals with additional charges. The s3_request metricset will give more
+=== s3_request and s3_daily_storage metricset
+Daily storage metrics for S3 buckets are reported once per day with no additional cost. Since they are daily metrics,
+`period` for `s3_daily_storage` metricset is recommended to be `86400s` or multiples of `86400s`.
+Request metrics are available
+at 1-minute intervals with additional charges. The s3_request metricset will give more
 granular data to track S3 bucket usage. The `period` for `s3_request` metricset can be set to `60s` or multiples of `60s`.
 But because of the extra charges for querying these metrics, the `period` is recommended to set to `86400s`. The user can
 always adjust this to the granularity they want. Request metrics are not enabled by default for S3 buckets. Please see

x-pack/metricbeat/module/aws/ec2/ec2_test.go

@@ -11,7 +11,6 @@ import (
 
 	awssdk "github.com/aws/aws-sdk-go-v2/aws"
 	"github.com/aws/aws-sdk-go-v2/service/cloudwatch"
-	"github.com/aws/aws-sdk-go-v2/service/cloudwatch/cloudwatchiface"
 	"github.com/aws/aws-sdk-go-v2/service/ec2"
 	"github.com/aws/aws-sdk-go-v2/service/ec2/ec2iface"
 	"github.com/stretchr/testify/assert"
@@ -27,11 +26,6 @@ type MockEC2Client struct {
 	ec2iface.EC2API
 }
 
-// MockCloudWatchClient struct is used for unit tests.
-type MockCloudWatchClient struct {
-	cloudwatchiface.CloudWatchAPI
-}
-
 var (
 	regionName = "us-west-1"
 	instanceID = "i-123"

x-pack/metricbeat/module/aws/s3_daily_storage/_meta/data.json

@@ -0,0 +1,34 @@
+{
+    "@timestamp": "2017-10-12T08:05:34.853Z",
+    "agent": {
+        "hostname": "host.example.com",
+        "name": "host.example.com"
+    },
+    "aws": {
+        "s3_daily_storage": {
+            "bucket": {
+                "name": "test-s3-ks",
+                "size": {
+                    "bytes": 1679631
+                }
+            },
+            "number_of_objects": 3
+        }
+    },
+    "cloud": {
+        "provider": "aws",
+        "region": "ap-southeast-1"
+    },
+    "event": {
+        "dataset": "aws.s3_daily_storage",
+        "duration": 115000,
+        "module": "aws"
+    },
+    "metricset": {
+        "name": "s3_daily_storage"
+    },
+    "service": {
+        "name": "s3_daily_storage",
+        "type": "s3_daily_storage"
+    }
+}

x-pack/metricbeat/module/aws/s3_daily_storage/_meta/docs.asciidoc

@@ -0,0 +1,10 @@
+The s3_daily_storage metricset of aws module allows you to monitor your AWS S3 buckets. `s3_daily_storage` metricset
+fetches Cloudwatch daily storage metrics for each S3 bucket from
+https://docs.aws.amazon.com/AmazonS3/latest/dev/cloudwatch-monitoring.html[S3 CloudWatch Daily Storage Metrics for Buckets].
+
+ === AWS Permissions
+Some specific AWS permissions are required for IAM user to collect AWS s3_daily_storage metrics.
+----
+ec2:DescribeRegions
+cloudwatch:GetMetricData
+----

x-pack/metricbeat/module/aws/s3_daily_storage/_meta/fields.yml

@@ -0,0 +1,18 @@
+- name: s3_daily_storage
+  type: group
+  description: >
+    `s3_daily_storage` contains the daily storage metrics that were scraped from AWS CloudWatch which contains monitoring metrics sent by AWS S3.
+  release: beta
+  fields:
+    - name: bucket.name
+      type: keyword
+      description: >
+        Name of a S3 bucket.
+    - name: bucket.size.bytes
+      type: scaled_float
+      description: >
+        The amount of data in bytes stored in a bucket.
+    - name: number_of_objects
+      type: long
+      description: >
+        The total number of objects stored in a bucket for all storage classes.

x-pack/metricbeat/module/aws/s3_daily_storage/data.go

@@ -0,0 +1,21 @@
+// Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
+// or more contributor license agreements. Licensed under the Elastic License;
+// you may not use this file except in compliance with the Elastic License.
+
+package s3_daily_storage
+
+import (
+	s "github.com/elastic/beats/libbeat/common/schema"
+	c "github.com/elastic/beats/libbeat/common/schema/mapstrstr"
+)
+
+var (
+	schemaMetricSetFields = s.Schema{
+		"bucket": s.Object{
+			"size": s.Object{
+				"bytes": c.Float("BucketSizeBytes"),
+			},
+		},
+		"number_of_objects": c.Int("NumberOfObjects"),
+	}
+)