This solution shows how to automate the ingestion of your AWS Config snapshots into the ElasticSearch/Logstash/Kibana (ELK) stack for searching and mapping your AWS environments. We'll be using a Lambda function which will automatically be fired when a new file is uploaded to the S3 bucket used by AWS Config. This readme updates an article "How to Analyze AWS Config Snapshots with ElasticSearch and Kibana by Vladimir Budilov" referenced below and provides a more basic step by step process.
Use the AWS Console to configure the AWS Config Service. This is a step by step process.
Get started
Resource types to record
All resources: checked
Amazon S3 bucket
Create a bucket: selected
AWS Config role
Create a role: selected
Next
Next
Confirm
S3 Bucket
config-bucket-<AWS Account Id>
IAM Roles
config-role-us-east-1
IAM Policies
Customer managed: config-role-us-east-1_AWSConfigDeliveryPermissions_us-east-1
Use the AWS Console to configure the AWS Elasticsearch Service. This is a step by step process.
Create a new domain
Elasticsearch domain name:
aws-config-es
Elasticsearch version:
5.5
Next
Instance count:
1
Instance type:
t2.small.elasticsearch
Next
Network configuration
Public access
Access policy
Select a template pull-down
Allow open access to the domain
Note: acknowledge risk pop-up
Next
Confirm
From the AWS Elasticsearch Console you will see aws-config-es domain configuration status. "Configuration state” shows the status of the just created domain and initially will display “Loading”. Once it shows “Active” the domain will be ready to access. Node configuration took 12 minutes to become active for this example.
Drill down to the aws-config-es domain and select the "Overview" tab.
Of specific interest for this task are:
"Endpoint: <Endpoint URL>"
"Kibana: <Kibana URL>"
Use the AWS Console to configure the AWS Lambda function. The Lambda function will be used to import AWS Config Snapshots into Elasticsearch automatically. This is a step by step process.
Create a function
Author from scratch
Name: aws-config-lambda
Runtime: Python 2.7
Role: Create new roll from template(s)
Role name: aws-config-lambda-role
Policy templates: S3 object read-only permissions
Create function
Designer section "Add triggers from the list on the left"
Add triggers
Select "S3"
Configure triggers Section
Event type: PUT
Enable trigger: checked
Add
Save
Designer section Select "aws-config-lambda"
Function code Section
Code entry type pull-down
Upload a .ZIP file
Upload
See eclipse-pydev-aws-config-lambda github project for instructions on how to build the deployment archive
deployment/aws-config-lambda.zip
Save
Environment variables Section
Use <Endpoint URL> from "AWS Elasticsearch Resources Created"
Example: https://search-aws-config-es-rylg5tz5l7j77uocyuzy5gjcu4.us-east-1.es.amazonaws.com
aws_config_es_endpoint <Endpoint URL>
Save
Basic settings Section
Timeout: 3 min 0 sec
Save
S3 Bucket config-bucket-<AWS::AccountId>
Properties/Events
<id>
Events: Put
Send to: Lambda Function
Lambda: aws-config-lambda
IAM Roles
aws-config-lambda-role
IAM Policies
Customer managed:
AWSLambdaS3ExecutionRole-<id>
AWSLambdaBasicExecutionRole-<id>
AWS Lambda provides a Test harness which can be found in the upper right hand corner of the "Configuration tab" of the function. There's a little easier way to test the function by using the AWS S3 Console. Simply locate an AWS Config Snapshot file on S3, download it using the AWS S3 Console and simply upload it again to the same location. This will change the timestamp and fire the lambda rule.
Example:
cd config-bucket-754487812300/AWSLogs/754487812300/Config/us-east-1/2018/3/9/ConfigSnapshot
download
754487812300_Config_us-east-1_ConfigSnapshot_20180309T130612Z_6cff9e2b-c6a1-44c2-9437-0cb038e5e09d.json.gz
upload to same location:
754487812300_Config_us-east-1_ConfigSnapshot_20180309T130612Z_6cff9e2b-c6a1-44c2-9437-0cb038e5e09d.json.gz
You can also use the AWS CLI to query AWS Config and push config snapshots:
Example:
aws configservice describe-configuration-recorder-status --configuration-recorder-names
aws configservice describe-delivery-channel-status
aws configservice describe-config-rules
Run the following command to push config snapshot to the delivery channel:
aws configservice deliver-config-snapshot --delivery-channel-name <delivery-channel>
Additionally you can use AWS Cloudwatch Console to monitor and debug activity from the lambda function
Use the Kabana Console to configure the index pattern and time filter. This is a step by step process.
Using <Kibana URL> from "AWS Elasticsearch Resources Created"
Index name or pattern
*
Time Filter field name
snapshotTimeIso
Create
Go to the Discover View to browse data
How to Analyze AWS Config Snapshots with Elasticsearch and Kibana
https://aws.amazon.com/blogs/developer/how-to-analyze-aws-config-snapshots-with-elasticsearch-and-kibana/
Import your AWS Config Snapshots into Elasticsearch
https://github.com/awslabs/aws-config-to-elasticsearch
Understanding the basic components of AWS Config will help you get the most out of this service
https://docs.aws.amazon.com/config/latest/developerguide/config-concepts.html
Creating and Configuring Amazon Elasticsearch Service Domains
https://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/es-createupdatedomains.html
Starting from Elasticsearch 6.0, all REST requests that include a body must also provide the correct content-type for that body
https://www.elastic.co/blog/strict-content-type-checking-for-elasticsearch-rest-requests