Add SQS metricset into AWS metricbeat module (#10684)

* Initial PR for adding SQS metricset * Address comments in sqs metricset * Remove regionName as input param * Update docs * Remove count from metric names * Remove duplicate functions from rebase
elastic · Feb 25, 2019 · 5d4dc24 · 5d4dc24
1 parent 2cdb5b1
commit 5d4dc24
Show file tree

Hide file tree

Showing 17 changed files with 536 additions and 38 deletions.
diff --git a/CHANGELOG.next.asciidoc b/CHANGELOG.next.asciidoc
@@ -305,6 +305,7 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
 - Add overview dashboard to Zookeeper Metricbeat module {pull}10379[10379]
 - Add Consul Metricbeat module with Agent Metricset {pull}8631[8631]
 - Add filters and pie chart for AWS EC2 dashboard. {pull}10596[10596]
+- Add AWS SQS metricset. {pull}10684[10684] {issue}10053[10053]
 
 *Packetbeat*
 

diff --git a/metricbeat/docs/fields.asciidoc b/metricbeat/docs/fields.asciidoc
@@ -1083,6 +1083,103 @@ type: integer
 The state of the instance (pending | running | shutting-down | terminated | stopping | stopped).
 
 
+--
+
+[float]
+== sqs fields
+
+`sqs` contains the metrics that were scraped from AWS CloudWatch which contains monitoring metrics sent by AWS SQS.
+
+
+
+*`aws.sqs.oldest_message_age.sec`*::
++
+--
+type: long
+
+The approximate age of the oldest non-deleted message in the queue.
+
+
+--
+
+*`aws.sqs.message.delayed`*::
++
+--
+type: long
+
+TThe number of messages in the queue that are delayed and not available for reading immediately.
+
+
+--
+
+*`aws.sqs.message.not_visible`*::
++
+--
+type: long
+
+The number of messages that are in flight.
+
+
+--
+
+*`aws.sqs.message.visible`*::
++
+--
+type: long
+
+The number of messages available for retrieval from the queue.
+
+
+--
+
+*`aws.sqs.message.deleted`*::
++
+--
+type: long
+
+The number of messages deleted from the queue.
+
+
+--
+
+*`aws.sqs.message.received`*::
++
+--
+type: long
+
+The number of messages returned by calls to the ReceiveMessage action.
+
+
+--
+
+*`aws.sqs.message.sent`*::
++
+--
+type: long
+
+The number of messages added to a queue.
+
+
+--
+
+*`aws.sqs.empty_receives`*::
++
+--
+type: long
+
+The number of ReceiveMessage API calls that did not return a message.
+
+
+--
+
+*`aws.sqs.sent_message_size.bytes`*::
++
+--
+type: scaled_float
+
+The size of messages added to a queue.
+
+
 --
 
 [[exported-fields-beat]]

diff --git a/metricbeat/docs/modules/aws.asciidoc b/metricbeat/docs/modules/aws.asciidoc
@@ -9,7 +9,7 @@ This module periodically fetches monitoring metrics from AWS Cloudwatch using
 https://docs.aws.amazon.com/AmazonCloudWatch/latest/APIReference/API_GetMetricData.html[GetMetricData API] for running
 EC2 instances. Note: extra AWS charges on GetMetricData API requests will be generated by this module.
 
-The default metricset is `ec2`.
+The default metricsets are `ec2` and `sqs`.
 
 [float]
 === Module-specific configuration notes
@@ -35,36 +35,36 @@ aws> sts get-session-token --serial-number arn:aws:iam::1234:mfa/your-email@exam
 Specific permissions needs to be added into the IAM user's policy to authorize Metricbeat to collect AWS monitoring metrics. Please
 see documentation under each metricset for required permissions.
 
-By default, Amazon EC2 sends metric data to CloudWatch every 5 minutes. With this basic monitoring, `period` in aws module
-configuration should be larger or equal than `300s`. If `period` is set to be less than `300s`, the same cloudwatch metrics
-will be collected more than once which will cause extra fees without getting more granular metrics. For example, in `US East (N. Virginia)` region, it costs
-$0.01/1000 metrics requested using GetMetricData. Please see https://aws.amazon.com/cloudwatch/pricing/[AWS Cloudwatch Pricing]
-for more details. To avoid unnecessary charges, `period` is preferred to be set to `300s` or multiples of `300s`, such as
-`600s` and `900s`.
-
-For more granular monitoring data you can enable detailed monitoring on the instance to get metrics every 1 minute. Please see
-https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-cloudwatch-new.html[Enabling Detailed Monitoring] for instructions
-on how to enable detailed monitoring. With detailed monitoring enabled, `period` in aws module configuration can be any number
-larger than `60s`. Since AWS sends metric data to CloudWatch in 1-minute periods, setting metricbeat module `period` less
-than `60s` will cause extra API requests which means extra charges on AWS. To avoid unnecessary charges, `period` is
-preferred to be set to `60s` or multiples of `60s`, such as `120s` and `180s`.
-
 Here is an example of aws metricbeat module configuration:
 
 [source,yaml]
 ----
 metricbeat.modules:
 - module: aws
   period: 300s
-  metricsets: ["ec2"]
+  metricsets:
+    - "ec2"
+    - "sqs"
   access_key_id: '${AWS_ACCESS_KEY_ID}'
   secret_access_key: '${AWS_SECRET_ACCESS_KEY}'
   session_token: '${AWS_SESSION_TOKEN}'
   default_region: '${AWS_REGION:us-west-1}'
 ----
 
-This module only collects metrics for EC2 instances that are in `running` state and exist more than 10 minutes to make sure
-there are monitoring metrics exist in Cloudwatch already.
+By default, Amazon EC2 sends metric data to CloudWatch every 5 minutes. With this basic monitoring, `period` in aws module
+configuration should be larger or equal than `300s`. If `period` is set to be less than `300s`, the same cloudwatch metrics
+will be collected more than once which will cause extra fees without getting more granular metrics. For example, in `US East (N. Virginia)` region, it costs
+$0.01/1000 metrics requested using GetMetricData. Please see https://aws.amazon.com/cloudwatch/pricing/[AWS Cloudwatch Pricing]
+for more details. To avoid unnecessary charges, `period` is preferred to be set to `300s` or multiples of `300s`, such as
+`600s` and `900s`. For more granular monitoring data you can enable detailed monitoring on the instance to get metrics every 1 minute. Please see
+https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-cloudwatch-new.html[Enabling Detailed Monitoring] for instructions
+on how to enable detailed monitoring. With detailed monitoring enabled, `period` in aws module configuration can be any number
+larger than `60s`. Since AWS sends metric data to CloudWatch in 1-minute periods, setting metricbeat module `period` less
+than `60s` will cause extra API requests which means extra charges on AWS. To avoid unnecessary charges, `period` is
+preferred to be set to `60s` or multiples of `60s`, such as `120s` and `180s`.
+
+Since cloudWatch metrics for Amazon SQS queues are automatically collected and pushed to CloudWatch every 5 minutes,
+`period` for `sqs` is recommended to set to `300s` or multiples of `300s`.
 
 The AWS module comes with a predefined dashboard. For example:
 
@@ -84,6 +84,7 @@ metricbeat.modules:
   period: 300s
   metricsets:
     - "ec2"
+    - "sqs"
   access_key_id: '${AWS_ACCESS_KEY_ID:""}'
   secret_access_key: '${AWS_SECRET_ACCESS_KEY:""}'
   session_token: '${AWS_SESSION_TOKEN:""}'
@@ -97,5 +98,9 @@ The following metricsets are available:
 
 * <<metricbeat-metricset-aws-ec2,ec2>>
 
+* <<metricbeat-metricset-aws-sqs,sqs>>
+
 include::aws/ec2.asciidoc[]
 
+include::aws/sqs.asciidoc[]
+
diff --git a/metricbeat/docs/modules/aws/sqs.asciidoc b/metricbeat/docs/modules/aws/sqs.asciidoc
@@ -0,0 +1,23 @@
+////
+This file is generated! See scripts/docs_collector.py
+////
+
+[[metricbeat-metricset-aws-sqs]]
+=== aws sqs metricset
+
+beta[]
+
+include::../../../../x-pack/metricbeat/module/aws/sqs/_meta/docs.asciidoc[]
+
+
+==== Fields
+
+For a description of each field in the metricset, see the
+<<exported-fields-aws,exported fields>> section.
+
+Here is an example document generated by this metricset:
+
+[source,json]
+----
+include::../../../../x-pack/metricbeat/module/aws/sqs/_meta/data.json[]
+----
diff --git a/metricbeat/docs/modules_list.asciidoc b/metricbeat/docs/modules_list.asciidoc
@@ -10,7 +10,8 @@ This file is generated! See scripts/docs_collector.py
 |<<metricbeat-module-apache,Apache>>     |image:./images/icon-yes.png[Prebuilt dashboards are available]    |  
 .1+| .1+|  |<<metricbeat-metricset-apache-status,status>>   
 |<<metricbeat-module-aws,aws>>     |image:./images/icon-yes.png[Prebuilt dashboards are available]    |  
-.1+| .1+|  |<<metricbeat-metricset-aws-ec2,ec2>>   
+.2+| .2+|  |<<metricbeat-metricset-aws-ec2,ec2>>   
+|<<metricbeat-metricset-aws-sqs,sqs>> beta[]  
 |<<metricbeat-module-ceph,Ceph>>     |image:./images/icon-no.png[No prebuilt dashboards]    |  
 .7+| .7+|  |<<metricbeat-metricset-ceph-cluster_disk,cluster_disk>>   
 |<<metricbeat-metricset-ceph-cluster_health,cluster_health>>   

diff --git a/x-pack/metricbeat/include/list.go b/x-pack/metricbeat/include/list.go
diff --git a/x-pack/metricbeat/metricbeat.reference.yml b/x-pack/metricbeat/metricbeat.reference.yml
@@ -152,6 +152,7 @@ metricbeat.modules:
   period: 300s
   metricsets:
     - "ec2"
+    - "sqs"
   access_key_id: '${AWS_ACCESS_KEY_ID:""}'
   secret_access_key: '${AWS_SECRET_ACCESS_KEY:""}'
   session_token: '${AWS_SESSION_TOKEN:""}'

diff --git a/x-pack/metricbeat/module/aws/_meta/config.yml b/x-pack/metricbeat/module/aws/_meta/config.yml
@@ -2,6 +2,7 @@
   period: 300s
   metricsets:
     - "ec2"
+    - "sqs"
   access_key_id: '${AWS_ACCESS_KEY_ID:""}'
   secret_access_key: '${AWS_SECRET_ACCESS_KEY:""}'
   session_token: '${AWS_SESSION_TOKEN:""}'

diff --git a/x-pack/metricbeat/module/aws/_meta/docs.asciidoc b/x-pack/metricbeat/module/aws/_meta/docs.asciidoc
@@ -2,7 +2,7 @@ This module periodically fetches monitoring metrics from AWS Cloudwatch using
 https://docs.aws.amazon.com/AmazonCloudWatch/latest/APIReference/API_GetMetricData.html[GetMetricData API] for running
 EC2 instances. Note: extra AWS charges on GetMetricData API requests will be generated by this module.
 
-The default metricset is `ec2`.
+The default metricsets are `ec2` and `sqs`.
 
 [float]
 === Module-specific configuration notes
@@ -28,36 +28,36 @@ aws> sts get-session-token --serial-number arn:aws:iam::1234:mfa/your-email@exam
 Specific permissions needs to be added into the IAM user's policy to authorize Metricbeat to collect AWS monitoring metrics. Please
 see documentation under each metricset for required permissions.
 
-By default, Amazon EC2 sends metric data to CloudWatch every 5 minutes. With this basic monitoring, `period` in aws module
-configuration should be larger or equal than `300s`. If `period` is set to be less than `300s`, the same cloudwatch metrics
-will be collected more than once which will cause extra fees without getting more granular metrics. For example, in `US East (N. Virginia)` region, it costs
-$0.01/1000 metrics requested using GetMetricData. Please see https://aws.amazon.com/cloudwatch/pricing/[AWS Cloudwatch Pricing]
-for more details. To avoid unnecessary charges, `period` is preferred to be set to `300s` or multiples of `300s`, such as
-`600s` and `900s`.
-
-For more granular monitoring data you can enable detailed monitoring on the instance to get metrics every 1 minute. Please see
-https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-cloudwatch-new.html[Enabling Detailed Monitoring] for instructions
-on how to enable detailed monitoring. With detailed monitoring enabled, `period` in aws module configuration can be any number
-larger than `60s`. Since AWS sends metric data to CloudWatch in 1-minute periods, setting metricbeat module `period` less
-than `60s` will cause extra API requests which means extra charges on AWS. To avoid unnecessary charges, `period` is
-preferred to be set to `60s` or multiples of `60s`, such as `120s` and `180s`.
-
 Here is an example of aws metricbeat module configuration:
 
 [source,yaml]
 ----
 metricbeat.modules:
 - module: aws
   period: 300s
-  metricsets: ["ec2"]
+  metricsets:
+    - "ec2"
+    - "sqs"
   access_key_id: '${AWS_ACCESS_KEY_ID}'
   secret_access_key: '${AWS_SECRET_ACCESS_KEY}'
   session_token: '${AWS_SESSION_TOKEN}'
   default_region: '${AWS_REGION:us-west-1}'
 ----
 
-This module only collects metrics for EC2 instances that are in `running` state and exist more than 10 minutes to make sure
-there are monitoring metrics exist in Cloudwatch already.
+By default, Amazon EC2 sends metric data to CloudWatch every 5 minutes. With this basic monitoring, `period` in aws module
+configuration should be larger or equal than `300s`. If `period` is set to be less than `300s`, the same cloudwatch metrics
+will be collected more than once which will cause extra fees without getting more granular metrics. For example, in `US East (N. Virginia)` region, it costs
+$0.01/1000 metrics requested using GetMetricData. Please see https://aws.amazon.com/cloudwatch/pricing/[AWS Cloudwatch Pricing]
+for more details. To avoid unnecessary charges, `period` is preferred to be set to `300s` or multiples of `300s`, such as
+`600s` and `900s`. For more granular monitoring data you can enable detailed monitoring on the instance to get metrics every 1 minute. Please see
+https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-cloudwatch-new.html[Enabling Detailed Monitoring] for instructions
+on how to enable detailed monitoring. With detailed monitoring enabled, `period` in aws module configuration can be any number
+larger than `60s`. Since AWS sends metric data to CloudWatch in 1-minute periods, setting metricbeat module `period` less
+than `60s` will cause extra API requests which means extra charges on AWS. To avoid unnecessary charges, `period` is
+preferred to be set to `60s` or multiples of `60s`, such as `120s` and `180s`.
+
+Since cloudWatch metrics for Amazon SQS queues are automatically collected and pushed to CloudWatch every 5 minutes,
+`period` for `sqs` is recommended to set to `300s` or multiples of `300s`.
 
 The AWS module comes with a predefined dashboard. For example:
 

diff --git a/x-pack/metricbeat/module/aws/fields.go b/x-pack/metricbeat/module/aws/fields.go
diff --git a/x-pack/metricbeat/module/aws/sqs/_meta/data.json b/x-pack/metricbeat/module/aws/sqs/_meta/data.json
@@ -0,0 +1,39 @@
+{
+    "@timestamp": "2017-10-12T08:05:34.853Z",
+    "agent": {
+        "hostname": "host.example.com",
+        "name": "host.example.com"
+    },
+    "aws": {
+        "sqs": {
+            "empty_receives": 0,
+            "messages": {
+                "delayed": 0,
+                "deleted": 0,
+                "not_visible": 0,
+                "received": 0,
+                "sent": 0,
+                "visible": 91
+            },
+            "oldest_message_age": {
+                "sec": 86404
+            },
+            "sent_message_size": {}
+        }
+    },
+    "cloud": {
+        "region": "us-east-1"
+    },
+    "event": {
+        "dataset": "aws.sqs",
+        "duration": 115000,
+        "module": "aws"
+    },
+    "metricset": {
+        "name": "sqs"
+    },
+    "service": {
+        "name": "sqs",
+        "type": "sqs"
+    }
+}
diff --git a/x-pack/metricbeat/module/aws/sqs/_meta/docs.asciidoc b/x-pack/metricbeat/module/aws/sqs/_meta/docs.asciidoc
@@ -0,0 +1,12 @@
+The sqs metricset of aws module allows you to monitor your AWS SQS queues. `sqs` metricset fetches a set of values from
+https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-available-cloudwatch-metrics.html[Amazon SQS Metrics].
+CloudWatch metrics for Amazon SQS queues are automatically collected and pushed to CloudWatch every five minutes.
+
+=== AWS Permissions
+Some specific AWS permissions are required for IAM user to collect AWS SQS metrics.
+----
+cloudwatch:GetMetricData
+ec2:DescribeRegions
+----
+
+=== Dashboard