-
Notifications
You must be signed in to change notification settings - Fork 4.9k
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Add AWS s3_daily_storage metricset (#10940)
* First patch on s3 daily storage metricset * Add s3 daily storage metricset and TestData * Change number_of_object to number_of_objects * Add timestamp check for s3_daily_storage * Change select.scanned to select_scanned
- Loading branch information
1 parent
90f1f1a
commit 0acee7c
Showing
24 changed files
with
468 additions
and
24 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,23 @@ | ||
| //// | ||
| This file is generated! See scripts/docs_collector.py | ||
| //// | ||
|
|
||
| [[metricbeat-metricset-aws-s3_daily_storage]] | ||
| === aws s3_daily_storage metricset | ||
|
|
||
| beta[] | ||
|
|
||
| include::../../../../x-pack/metricbeat/module/aws/s3_daily_storage/_meta/docs.asciidoc[] | ||
|
|
||
|
|
||
| ==== Fields | ||
|
|
||
| For a description of each field in the metricset, see the | ||
| <<exported-fields-aws,exported fields>> section. | ||
|
|
||
| Here is an example document generated by this metricset: | ||
|
|
||
| [source,json] | ||
| ---- | ||
| include::../../../../x-pack/metricbeat/module/aws/s3_daily_storage/_meta/data.json[] | ||
| ---- |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
Oops, something went wrong.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
Oops, something went wrong.
34 changes: 34 additions & 0 deletions
34
x-pack/metricbeat/module/aws/s3_daily_storage/_meta/data.json
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,34 @@ | ||
| { | ||
| "@timestamp": "2017-10-12T08:05:34.853Z", | ||
| "agent": { | ||
| "hostname": "host.example.com", | ||
| "name": "host.example.com" | ||
| }, | ||
| "aws": { | ||
| "s3_daily_storage": { | ||
| "bucket": { | ||
| "name": "test-s3-ks", | ||
| "size": { | ||
| "bytes": 1679631 | ||
| } | ||
| }, | ||
| "number_of_objects": 3 | ||
| } | ||
| }, | ||
| "cloud": { | ||
| "provider": "aws", | ||
| "region": "ap-southeast-1" | ||
| }, | ||
| "event": { | ||
| "dataset": "aws.s3_daily_storage", | ||
| "duration": 115000, | ||
| "module": "aws" | ||
| }, | ||
| "metricset": { | ||
| "name": "s3_daily_storage" | ||
| }, | ||
| "service": { | ||
| "name": "s3_daily_storage", | ||
| "type": "s3_daily_storage" | ||
| } | ||
| } |
10 changes: 10 additions & 0 deletions
10
x-pack/metricbeat/module/aws/s3_daily_storage/_meta/docs.asciidoc
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,10 @@ | ||
| The s3_daily_storage metricset of aws module allows you to monitor your AWS S3 buckets. `s3_daily_storage` metricset | ||
| fetches Cloudwatch daily storage metrics for each S3 bucket from | ||
| https://docs.aws.amazon.com/AmazonS3/latest/dev/cloudwatch-monitoring.html[S3 CloudWatch Daily Storage Metrics for Buckets]. | ||
|
|
||
| === AWS Permissions | ||
| Some specific AWS permissions are required for IAM user to collect AWS s3_daily_storage metrics. | ||
| ---- | ||
| ec2:DescribeRegions | ||
| cloudwatch:GetMetricData | ||
| ---- |
18 changes: 18 additions & 0 deletions
18
x-pack/metricbeat/module/aws/s3_daily_storage/_meta/fields.yml
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,18 @@ | ||
| - name: s3_daily_storage | ||
| type: group | ||
| description: > | ||
| `s3_daily_storage` contains the daily storage metrics that were scraped from AWS CloudWatch which contains monitoring metrics sent by AWS S3. | ||
| release: beta | ||
| fields: | ||
| - name: bucket.name | ||
| type: keyword | ||
| description: > | ||
| Name of a S3 bucket. | ||
| - name: bucket.size.bytes | ||
| type: scaled_float | ||
| description: > | ||
| The amount of data in bytes stored in a bucket. | ||
| - name: number_of_objects | ||
| type: long | ||
| description: > | ||
| The total number of objects stored in a bucket for all storage classes. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,21 @@ | ||
| // Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one | ||
| // or more contributor license agreements. Licensed under the Elastic License; | ||
| // you may not use this file except in compliance with the Elastic License. | ||
|
|
||
| package s3_daily_storage | ||
|
|
||
| import ( | ||
| s "github.com/elastic/beats/libbeat/common/schema" | ||
| c "github.com/elastic/beats/libbeat/common/schema/mapstrstr" | ||
| ) | ||
|
|
||
| var ( | ||
| schemaMetricSetFields = s.Schema{ | ||
| "bucket": s.Object{ | ||
| "size": s.Object{ | ||
| "bytes": c.Float("BucketSizeBytes"), | ||
| }, | ||
| }, | ||
| "number_of_objects": c.Int("NumberOfObjects"), | ||
| } | ||
| ) |
Oops, something went wrong.