From a154b48fcdcfded75c36cad06e9f5d7ed2c4c969 Mon Sep 17 00:00:00 2001 From: Larry Hitchon Date: Wed, 18 Apr 2018 10:21:13 -0700 Subject: [PATCH] update documentation --- README.md | 128 +++++++++--------------------------------- TODO.md | 15 +++++ docs/example-rules.md | 71 +++++++++++++++++++++++ docs/lambda.md | 46 --------------- docs/operations.md | 6 +- docs/rules.md | 2 +- 6 files changed, 115 insertions(+), 153 deletions(-) create mode 100644 TODO.md create mode 100644 docs/example-rules.md delete mode 100644 docs/lambda.md diff --git a/README.md b/README.md index 674f318..c0fc38b 100644 --- a/README.md +++ b/README.md @@ -19,12 +19,16 @@ Alternatively, you can install manually from the [releases](https://github.com/s # Build Command Line tool ``` -make config-lint +make all ``` # Run -The program currently supports scanning of the following types of files: +The program has a set of built-in rules for scanning the following types of files: + +* Terraform + +The program can also read files from a separate YAML file, and can scan these types of files: * Terraform * Kubernetes @@ -36,18 +40,25 @@ And also the scanning of information from AWS Descibe API calls for: * Security Groups * IAM Users + ## Example invocations -### Validate Terraform files +### Validate Terraform files with built-in rules ``` -./config-lint --rules example-files/rules/terraform.yml example-files/config/* +config-lint -terraform example-files/config/* +``` + +### Validate Terraform files with custom rules + +``` +config-lint -rules examples-files/rules/terraform.yml example-files/config/* ``` ### Validate Kubernetes files ``` -./config-lint --rules example-files/rules/kubernetes.yml example-files/config/* +config-lint -rules example-files/rules/kubernetes.yml example-files/config/* ``` ### Validate LintRules files @@ -55,95 +66,32 @@ And also the scanning of information from AWS Descibe API calls for: This type of linting allows the tool to lint its own rules. ``` -./config-lint --rules example-files/rules/lint-rules.yml example-files/rules/* +config-lint -rules example-files/rules/lint-rules.yml example-files/rules/* ``` ### Validate Existing Security Groups ``` -./config-lint --rules example-files/rules/security-groups.yml +config-lint -rules example-files/rules/security-groups.yml ``` ### Validate Existing IAM Users ``` -./config-lint --rules example-files/rules/iam-users.yml +config-lint -rules example-files/rules/iam-users.yml ``` -# Rules File +# Rules + +A YAML file that specifies what kinds of files to process, and what validations to perform, [documented here](docs/rules.md). -A YAML file that specifies what kinds of files to process, and what validations to perform. +# Operations -[Documented Here](docs/rules.md) +The rules contain expressions that can use expresssions [documented here](docs/operations.md). ## Examples -To test that an AWS instance type has one of two values: -``` -Version: 1 -Description: Example rules -Type: Terraform -Files: - - "*.tf" -Rules: - - id: EC2_INSTANCE_TYPE - message: Instance type should be t2.micro or m3.medium - resource: aws_instance - assertions: - - key: instance_type - op: in - value: t2.micro,m3.medium - severity: WARNING -``` - -This could also be done by using the or operation with two different assertions: - -``` -Version: 1 -Description: Example rules -Type: Terraform -Files: - - "*.tf" -Rules: - - id: EC2_INSTANCE_TYPE - message: Instance type should be t2.micro or m3.medium - resource: aws_instance - assertions: - or: - - key: instance_type - op: eq - value: t2.micro - - key: instance_type - op: eq - value: m3.medium - severity: WARNING -``` - -And this could also be done by looking up the valid values in an S3 object (HTTP endpoints are also supported) - -``` -Version: 1 -Description: Example rules -Type: Terraform -Files: - - "*.tf" -Rules: - - id: EC2_INSTANCE_TYPE - message: Instance type should be t2.micro or m3.medium - resource: aws_instance - assertions: - - key: instance_type - op: eq - value_from: s3://your-bucket/instance-types.txt - severity: FAILURE -``` - -The assertions and operations were inspired by those in Cloud Custodian: http://capitalone.github.io/cloud-custodian/docs/ - - -## Valid Operations - -[Documented Here](docs/operations.md) +See [here](docs/example-rules.md) for examples of custom rules. # Output @@ -179,32 +127,6 @@ This example will scan the example terraform file and print the "ami" attribute If you specify --search, the rules files is only used to determine the type of configuration files. The files will *not* be scanned for violations. - -# Support for AWS Config Custom Rules - -It is also possible to use a rules files in a Lambda that handles events from AWS Config. - -[Documented Here](docs/lambda.md) - # Releasing To release a new version, run `make bumpversion` to increment the patch version and push a tag to GitHub to start the release process. -# TODO - -* Add an optional YAML file for project settings, such as ignoring certain rules for certain resources -* Figure out how dependency management works in go -* The lambda function does not handle OverSizedChangeNotification -* The lambda function name is hard-coded in the Makefile -* Region is hard-coded to us-east-1 for GetValueFromS3 -* Use type switch as more idiomatic way to handle multiple types in match.go -* Start using go testing coverage tools -* Use log package for error reporting -* Deal with a few FIXME comments in code, mostly error handling -* Should there be some pre-defined RuleSets? -* Would it be useful to have helper utilities to send output to CloudWatch/SNS/Kinesis? -* Add variable interpolation for Terraform files -* Update value_from to handle JSON return values -* Create a Provider interface for AWS calls, create a mock for testing SecurityGroupLinter -* Starting to have inconsistent naming in ops: is-true, is-false, has-properties vs. present, absent, empty, null -* Add options to Assertion type, for things like 'ignore-case' for string compares? Or just use a regex? -* Provide a default -query of 'Violations[]', and add an option for a full report diff --git a/TODO.md b/TODO.md new file mode 100644 index 0000000..46dfd33 --- /dev/null +++ b/TODO.md @@ -0,0 +1,15 @@ +# TODO + +* Add an optional YAML file for project settings, such as ignoring certain rules for certain resources +* Region is hard-coded to us-east-1 for GetValueFromS3 +* Use type switch as more idiomatic way to handle multiple types in match.go +* Use log package for error reporting +* Deal with a few FIXME comments in code, mostly error handling +* Would it be useful to have helper utilities to send output to CloudWatch/SNS/Kinesis? +* Update value_from to handle JSON return values +* Create a Provider interface for AWS calls, create a mock for testing SecurityGroupLinter +* Starting to have inconsistent naming in ops: is-true, is-false, has-properties vs. present, absent, empty, null +* Add options to Assertion type, for things like 'ignore-case' for string compares? Or just use a regex? +* Provide a default -query of 'Violations[]', and add an option for a full report +* Document conditions + diff --git a/docs/example-rules.md b/docs/example-rules.md new file mode 100644 index 0000000..a3d44be --- /dev/null +++ b/docs/example-rules.md @@ -0,0 +1,71 @@ +# Example Rules + +Add these rules to a YAML file, and pass the filename to config-lint using the -rules option. +Each rule contains a list of assertions, and these assertions use operations that are [documented here](operations.md) + +To test that an AWS instance type has one of two values: + +``` +Version: 1 +Description: Example rules +Type: Terraform +Files: + - "*.tf" +Rules: + - id: EC2_INSTANCE_TYPE + message: Instance type should be t2.micro or m3.medium + resource: aws_instance + assertions: + - key: instance_type + op: in + value: t2.micro,m3.medium + severity: WARNING +``` + +This could also be done by using the or operation with two different assertions: + +``` +Version: 1 +Description: Example rules +Type: Terraform +Files: + - "*.tf" +Rules: + - id: EC2_INSTANCE_TYPE + message: Instance type should be t2.micro or m3.medium + resource: aws_instance + assertions: + or: + - key: instance_type + op: eq + value: t2.micro + - key: instance_type + op: eq + value: m3.medium + severity: WARNING +``` + +And this could also be done by looking up the valid values in an S3 object (HTTP endpoints are also supported) + +``` +Version: 1 +Description: Example rules +Type: Terraform +Files: + - "*.tf" +Rules: + - id: EC2_INSTANCE_TYPE + message: Instance type should be t2.micro or m3.medium + resource: aws_instance + assertions: + - key: instance_type + op: eq + value_from: s3://your-bucket/instance-types.txt + severity: FAILURE +``` + +The assertions and operations were inspired by those in Cloud Custodian: http://capitalone.github.io/cloud-custodian/docs/ + + + + diff --git a/docs/lambda.md b/docs/lambda.md deleted file mode 100644 index 0b41cd7..0000000 --- a/docs/lambda.md +++ /dev/null @@ -1,46 +0,0 @@ -# Custom Rule for AWS Config - -Build and deploy the lambda function: - -``` -make lambda -``` - -This builds and deploys an AWS Lambda function. The ARN for the Lambda is used to set up a custom AWS Config rule. -The same YAML format is used to specify the rules to test for compliance. The severity of the rules for -this use case should be set to NON_COMPLIANT - -There are two parameters that need to also be configured for the AWS Config rule: - -|Name |Description | -|-----------|------------------------------------------------------------------------------------| -|bucket | S3 bucket that contains the S3 object with the YAML rules | -|key | Key of the S3 object | - - -## AWS Config example - -Here's an example of an AWS Config rule that checks for port 22 being open to all IP addresses. -It also includes the 'except:' option which allows the check to be ignored for some resources. - -``` -Version: 1 -Description: Rules for AWS Config -Type: AWSConfig -Rules: - - id: SG1 - message: Security group should not allow ingress from 0.0.0.0/0 - resource: AWS::EC2::SecurityGroup - except: - - sg-88206cff - severity: NON_COMPLIANT - assertions: - - not: - - and: - - key: ipPermissions[].fromPort[] - op: contains - value: "22" - - key: ipPermissions[].ipRanges[] - op: contains - value: 0.0.0.0/0 -``` diff --git a/docs/operations.md b/docs/operations.md index 1410457..311dafa 100644 --- a/docs/operations.md +++ b/docs/operations.md @@ -251,7 +251,7 @@ Example: assertions: - every: key: Location - assertions: + expresssions: - key: latitude op: present - key: longitude @@ -277,7 +277,7 @@ Example: assertions: - some: key: Location - assertions: + expresssions: - key: latitude op: present - key: longitude @@ -303,7 +303,7 @@ Example: assertions: - none: key: "ipPermissions[]" - assertions: + expresssions: - key: "fromPort" op: eq value: 22 diff --git a/docs/rules.md b/docs/rules.md index 3445fde..e051444 100644 --- a/docs/rules.md +++ b/docs/rules.md @@ -19,6 +19,7 @@ Each rule contains the following attributes: |id | A unique identifier for the rule | |message | A string to be printed when a validation error is detected | |resource | The resource type to which the rule will be applied | +|conditions | Expressions (in addition to resource) that determine if a rule should apply | |except | An optional list of resource ids that should not be validated | |severity | FAILURE, WARNING, NON_COMPLIANT | |assertions | A list of expressions used to detect validation errors, see next section | @@ -43,4 +44,3 @@ Each expression contains the following attributes: |Url | HTTP endpoint to invoke | |Payload | Optional JMESPATH to use for payload, default is '@' | -