StackMaster is a CLI tool to manage CloudFormation stacks, with the following features:
- Synchronous visibility into stack updates. See exactly what is changing and what will happen before agreeing to apply a change.
- Dynamic parameter resolvers.
- Template compiler support for YAML and SparkleFormation.
Stack updates can cause a lot of damage if applied blindly. StackMaster helps with this by providing the operator with as much information about the proposed change as possible before asking for confirmation to continue. That information includes:
- Template body and parameter diffs.
- Change sets are displayed for review.
- Once the diffs & change set have been reviewed, the change can be applied and stack events monitored.
- Stack events will be displayed until an end state is reached.
Stack parameters can be dynamically resolved at runtime using one of the built in parameter resolvers. Parameters can be sourced from other stacks outputs, or by querying various AWS APIs to get resource ARNs, etc.
gem install stack_master
# if you want linting capabilities:
pip install cfn-lintpip install cfn-lintif you need lint functionality- Add
gem 'stack_master'to your Gemfile. - Run
bundle install - Run
bundle exec stack_master initto generate a directory structure and stack_master.yml file
Stacks are defined inside a stack_master.yml YAML file. When running
stack_master, it is assumed that this file will exist in the current working
directory, or that the file is passed in with --config /path/to/stack_master.yml. Here's an example configuration file:
region_aliases:
production: us-east-1
staging: ap-southeast-2
stack_defaults:
tags:
application: my-awesome-app
role_arn: service_role_arn
region_defaults:
us-east-1:
tags:
environment: production
notification_arns:
- test_arn
ap-southeast-2:
tags:
environment: staging
stacks:
production:
myapp-vpc:
template: myapp_vpc.rb
tags:
purpose: front-end
myapp-db:
template: myapp_db.rb
stack_policy_file: db_stack_policy.json
tags:
purpose: back-end
myapp-web:
template: myapp_web.rb
tags:
purpose: front-end
staging:
myapp-vpc:
template: myapp_vpc.rb
allowed_accounts: '123456789'
tags:
purpose: front-end
myapp-db:
template: myapp_db.rb
allowed_accounts:
- '1234567890'
- '9876543210'
tags:
purpose: back-end
myapp-web:
template: myapp_web.rb
tags:
purpose: front-end
eu-central-1:
myapp-vpc:
template: myapp_vpc.rb
tags:
purpose: vpcStackMaster can optionally use S3 to store the templates before creating a stack. This requires you to configure an S3 bucket in stack_master.yml:
stack_defaults:
s3:
bucket: my_bucket_name
prefix: cfn_templates/my-awesome-app
region: us-west-2Additional files can be configured to be uploaded to S3 alongside the templates:
stacks:
production:
myapp-vpc:
template: myapp_vpc.rb
files:
- userdata.shtemplates- CloudFormation, SparkleFormation or CfnDsl templates.parameters- Parameters as YAML files.secrets- encrypted secret files.policies- Stack policy JSON files.
StackMaster supports CloudFormation templates in plain JSON or YAML. Any .yml or .yaml file will be processed as
YAML, while any .json file will be processed as JSON. Additionally, YAML files can be pre-processed using ERB and
compile-time parameters.
By default, any template ending with .rb will be processed as a SparkleFormation
template. However, if you want to use CfnDsl templates you can add
the following lines to your stack_master.yml.
template_compilers:
rb: cfndsl
By default, parameters are loaded from multiple YAML files, merged from the following lookup paths from bottom to top:
- parameters/[stack_name].yaml
- parameters/[stack_name].yml
- parameters/[region]/[stack_name].yaml
- parameters/[region]/[stack_name].yml
- parameters/[region_alias]/[stack_name].yaml
- parameters/[region_alias]/[stack_name].yml
A simple parameter file could look like this:
key_name: myapp-us-east-1Alternatively, a parameter_files array can be defined to explicitly list
parameter files that will be loaded. If parameter_files are defined, the
automatic search locations will not be used.
parameters_dir: parameters # the default
stacks:
us-east-1:
my-app:
parameter_files:
- my-app.yml # parameters/my-app.ymlParameters can also be defined inline with stack definitions:
stacks:
us-east-1:
my-app:
parameters:
VpcId:
stack_output: my-vpc/VpcIdCompile time parameters can be defined in a stack's parameters file, using the key compile_time_parameters. Keys in
parameter files are automatically converted to camel case.
As an example:
# parameters/some_stack.yml
vpc_cidr: 10.0.0.0/16
compile_time_parameters:
subnet_cidrs:
- 10.0.0.0/28
- 10.0.2.0/28Compile time parameters can be used for SparkleFormation templates. It conforms and allows you to use the Compile Time Parameters feature.
Compile time parameters can be used to pre-process YAML CloudFormation templates. An example template:
# templates/some_stack_template.yml.erb
Parameters:
VpcCidr:
Type: String
Resources:
Vpc:
Type: AWS::EC2::VPC
Properties:
CidrBlock: !Ref VpcCidr
# Given the two subnet_cidrs parameters, this creates two resources:
# SubnetPrivate0 with a CidrBlock of 10.0.0.0/28, and
# SubnetPrivate1 with a CidrBlock of 10.0.2.0/28
<% params["SubnetCidrs"].each_with_index do |cidr, index| %>
SubnetPrivate<%= index %>:
Type: AWS::EC2::Subnet
Properties:
VpcId: !Ref Vpc
AvailabilityZone: ap-southeast-2
CidrBlock: <%= cidr %>
<% end %>Parameter values can be sourced dynamically using parameter resolvers.
One benefit of using parameter resolvers instead of hard coding values like VPC IDs and resource ARNs is that the same configuration works cross region/account, even though the resolved values will be different.
One way to resolve parameter values from different accounts to the one StackMaster runs in, is to assume a role in another account with the relevant IAM permissions to execute successfully.
This is supported in StackMaster by specifying the role and account properties for the
parameter resolver in the stack's parameters file.
All parameter resolvers are supported.
vpc_peering_id:
role: cross-account-parameter-resolver
account: 1234567890
stack_output: vpc-peering-stack-in-other-account/peering_name
an_array_param:
role: cross-account-parameter-resolver
account: 1234567890
stack_outputs:
- stack-in-account1/output
- stack-in-account1/another_output
another_array_param:
- role: cross-account-parameter-resolver
account: 1234567890
stack_output: stack-in-account1/output
- role: cross-account-parameter-resolver
account: 0987654321
stack_output: stack-in-account2/output
my_secret:
role: cross-account-parameter-resolver
account: 1234567890
parameter_store: ssm_parameter_nameAn example of use case where cross-account parameter resolving is particularly useful is when setting up VPC peering where you need the VPC ID of the peer. Without the ability to assume a role in another account, the only option was to hard code the peer's VPC ID.
The stack output parameter resolver looks up outputs from other stacks in the
same or different region. The expected format is [(region|region-alias):]stack-name/(OutputName|output_name).
vpc_id:
# Output from a stack in the same region
stack_output: my-vpc-stack/VpcId
bucket_name:
# Output from a stack in a different region
stack_output: us-east-1:init-bucket/bucket_name
zone_name:
# Output from a stack in a different region using its alias
stack_output: global:hosted-zone/ZoneNameThis is the most used parameter resolver because it enables stacks to be split up into their separated concerns (VPC, web, database etc) with outputs feeding into parameters of dependent stacks.
Note: The GPG parameter resolver has been extracted into a dedicated gem. Please install and follow the instructions for the stack_master-gpg_parameter_resolver gem.
An alternative to the secrets store, uses the AWS SSM Parameter store to protect
secrets. Expects a parameter of either String or SecureString type to be present in the
same region as the stack. You can store the parameter using a command like this
aws ssm put-parameter --region <region> --name <parameter name> --value <secret> --type (String|SecureString)
When doing so make sure you don't accidentally store the secret in your .bash_history and
you will likely want to set the parameter to NoEcho in your template.
db_password:
parameter_store: ssm_parameter_nameAn alternative to the secrets store is accessing 1password secrets using the 1password cli (op).
You declare a 1password lookup with the following parameters in your parameters file:
# parameters/database.yml
database_password:
one_password:
title: production database
vault: Shared
type: password1password stores the name of the secret in the title. You can pass the vault you expect the secret to be in.
Currently we support two types of secrets, passwords and secureNotes. All values must be declared, there are no defaults.
For more information on 1password cli please see here
ejson is a tool to manage asymmetrically encrypted values in JSON format.
This allows you to keep secrets securely in git/Github and gives anyone the ability the capability to add new
secrets without requiring access to the private key. ejson_wrapper
encrypts the underlying EJSON private key with KMS and stores it in the ejson file as _private_key_enc. Each
time an ejson secret is required, the underlying EJSON private key is first decrypted before passing it onto
ejson to decrypt the file.
First, generate an ejson file with ejson_wrapper, specifying the KMS key ID to be used:
gem install ejson_wrapper
ejson_wrapper generate --region us-east-1 --kms-key-id [key_id] --file secrets/production.ejsonThen, add the ejson_file argument to your stack in stack_master.yml:
stacks:
us-east-1:
my_app:
template: my_app.json
ejson_file: production.ejsonfinally refer to the secret key in the parameter file, i.e. parameters/my_app.yml:
my_param:
ejson: "my_secret"Additional configuration options:
ejson_file_regionThe AWS region to attempt to decrypt private key withejson_file_kmsDefault: true. Set to false to use ejson without KMS.
Looks up a security group by name and returns the ARN.
ssh_sg:
security_group: SSHSecurityGroupAn array of security group names can also be provided.
ssh_sg:
security_groups:
- SSHSecurityGroup
- WebAccessSecurityGroupLooks up an SNS topic by name and returns the ARN.
notification_topic:
sns_topic_name: PagerDutyLooks up the latest AMI ID by a given set of tags.
web_ami:
latest_ami_by_tags: role=web,application=myappNote that the corresponding array resolver is named latest_amis_by_tags.
Looks up the latest AMI ID by a given set of attributes. By default it will only return AMIs from the account the stack is created in, but you can specify the account ID or certain keywords mentioned in the aws documentation
This selects the latest wily hvm AMI from Ubuntu (using the account id):
bastion_ami:
latest_ami:
owners: 099720109477
filters:
name: ubuntu/images/hvm/ubuntu-wily-15.10-amd64-server-*A set of possible attributes is available in the AWS documentation.
Any value can be an array of possible matches.
Looks up the a Container Image from an ECR repository. By default this will return the latest container in a repository.
If tag is specified we return the sha digest of the image with this tag.
This avoids the issue where CloudFormation won't update a Task Definition if we use a tag such as latest, because it only updates resources if a parameter has changed.
This allows us to tag an image and deploy the latest version of that image via CloudFormation and avoids versioning our tags and having to store the metadata about the latest tag version somewhere.
Returns the docker repository URI, i.e. aws_account_id.dkr.ecr.region.amazonaws.com/container@sha256:digest
container_image_id:
latest_container:
repository_name: nginx # Required. The name of the repository
registry_id: "012345678910" # The AWS Account ID the repository is located in. Defaults to the current account's default registry. Must be in quotes.
region: us-east-1 # Defaults to the region the stack is located in
tag: production # By default we'll find the latest image pushed to the repository. If tag is specified we return the sha digest of the image with this tagLookup an environment variable:
db_username:
env: DB_USERNAMEFind an ACM certificate by domain name:
cert:
acm_certificate: www.example.comNew parameter resolvers can be created in a separate gem.
To create a resolver named my_resolver:
- Create a new gem using your favorite tool
- The gem structure must contain the following path:
lib/stack_master/parameter_resolvers/my_resolver.rb
- That file needs to contain a class named
StackMaster::ParameterResolvers::MyResolverthat implements aresolvemethod and an initializer taking 2 parameters :
module StackMaster
module ParameterResolvers
class MyResolver < Resolver
array_resolver # Also create a MyResolvers resolver to handle arrays
def initialize(config, stack_definition)
@config = config
@stack_definition = stack_definition
end
def resolve(value)
value
end
end
end
end- Note that the filename and classname are both derived from the resolver name passed in the parameter file. In our case, the parameters YAML would look like:
vpc_id:
my_resolver: dummy_valueMost resolvers support taking an array of values that will each be resolved. Unless stated otherwise in the documentation, the array version of the resolver will be named with the pluralized name of the original resolver.
When creating a new resolver, one can automatically create the array resolver by adding a array_resolver statement
in the class definition, with an optional class name if different from the default one.
module StackMaster
module ParameterResolvers
class MyResolver < Resolver
array_resolver class_name: 'MyCustomArrayResolver'
...
end
end
endIn that example, using the array resolver would look like:
my_parameter:
my_custom_array_resolver:
- value1
- value2Array parameter values can include nested parameter resolvers.
For example, given the following parameter definition:
my_parameter:
- stack_output: my-stack/output # value resolves to 'value1'
- value2The parameter value will resolve to:
my_parameter: 'value1,value2'An extension to SparkleFormation is the user_data_file! method, which evaluates templates in templates/user_data/[file_name]. Most of the usual SparkleFormation methods are available in user data templates. Example:
# templates/user_data/app.erb
REGION=<%= region! %>
ROLE=<%= role %>
And used like this in SparkleFormation templates:
# templates/app.rb
user_data user_data_file!('app.erb', role: :worker)You can also use the joined_file! method which evaluates templates in templates/config/[file_name]. It is similar to user_data_file! but doesn't do base64 encoding. Example:
# templates/config/someconfig.conf.erb
my_variable=<%= ref!(:foo) %>
my_other_variable=<%= account_id! %>
# templates/ecs_task.rb
container_definitions array!(
-> {
command array!(
"-e",
joined_file!('someconfig.conf.erb')
)
...
}
)StackMaster allows you to separate your stack definitions and parameters from your templates by way of a template_dir key in your stack_master.yml.
You can also pass compiler-specific options to the template compiler to further customize SparkleFormation or CfnDsl's behavior. Combining the 2 lets you move your SFN templates away from your stack definitions. For example, if your project is laid out as:
project-root
|-- envs
|-- env-1
|-- stack_master.yml
|-- env-2
|-- stack_master.yml
|-- sparkle
|-- templates
|-- my-stack.rb
Your env-1/stack_master.yml files can reference common templates by setting:
template_dir: ../../sparkle/templates
stack_defaults:
compiler_options:
sparkle_path: ../../sparkle
stacks:
us-east-1:
my-stack:
template: my-stack.rbSparklePacks can be pre-loaded using compiler options. This requires the name of a rubygem to require followed by the name of the SparklePack, which is usually the same name as the Gem.
stacks:
us-east-1:
my-stack:
template: my-stack-with-dynamic.rb
compiler_options:
sparkle_packs:
- vpc-sparkle-packThe template can then simply load a dynamic from the sparkle pack like so:
SparkleFormation.new(:my_stack_with_dynamic) do
dynamic!(:sparkle_pack_dynamic)
endNote though that if a dynamic with the same name exists in your templates/dynamics/ directory it will get loaded since it has higher precedence.
Templates can be also loaded from sparkle packs by defining sparkle_pack_template. The name corresponds to the registered symbol rather than specific name. That means for a sparkle pack containing:
SparkleFormation.new(:template_name) do
...
endwe can use stack defined as follows:
stacks:
us-east-1:
my-stack:
template: template_name
compiler: sparkle_formation
compiler_options:
sparkle_packs:
- some-sparkle-pack
sparkle_pack_template: trueThe AWS account the command is executing in can be restricted to a specific list of allowed accounts. This is useful in reducing the possibility of applying non-production changes in a production account. Each stack definition can specify the allowed_accounts property with an array of AWS account IDs or aliases the stack is allowed to work with.
This is an opt-in feature which is enabled by specifying at least one account to allow.
Unlike other stack defaults, the allowed_accounts property values specified in the stack definition override values specified in the stack defaults (i.e., other stack property values are merged together with those specified in the stack defaults). This allows specifying allowed accounts in the stack defaults (inherited by all stacks) and override them for specific stacks. See below example config for an example.
stack_defaults:
allowed_accounts: '555555555'
stacks:
us-east-1:
myapp-vpc: # only allow account 555555555 (inherited from the stack defaults)
template: myapp_vpc.rb
tags:
purpose: front-end
myapp-db:
template: myapp_db.rb
allowed_accounts: # only allow these accounts (overrides the stack defaults)
- '1234567890'
- my-account-alias
tags:
purpose: back-end
myapp-web:
template: myapp_web.rb
allowed_accounts: [] # allow all accounts (overrides the stack defaults)
tags:
purpose: front-end
myapp-redis:
template: myapp_redis.rb
allowed_accounts: '888888888' # only allow this account (overrides the stack defaults)
tags:
purpose: back-endIn the cases where you want to bypass the account check, there is the StackMaster flag --skip-account-check that can be used.
stack_master help # Display up to date docs on the commands available
stack_master init # Initialises a directory structure and stack_master.yml file
stack_master list # Lists stack definitions
stack_master apply [region-or-alias] [stack-name] # Create or update a stack
stack_master apply [region-or-alias] [stack-name] [region-or-alias] [stack-name] # Create or update multiple stacks
stack_master apply [region-or-alias] # Create or update stacks in the given region
stack_master apply # Create or update all stacks
stack_master --changed apply # Create or update all stacks that have changed
stack_master --yes apply [region-or-alias] [stack-name] # Create or update a stack non-interactively (forcing yes)
stack_master diff [region-or-alias] [stack-name] # Display a stack template and parameter diff
stack_master drift [region-or-alias] [stack-name] # Detects and displays stack drift using the CloudFormation Drift API
stack_master delete [region-or-alias] [stack-name] # Delete a stack
stack_master events [region-or-alias] [stack-name] # Display events for a stack
stack_master outputs [region-or-alias] [stack-name] # Display outputs for a stack
stack_master resources [region-or-alias] [stack-name] # Display outputs for a stack
stack_master status # Displays the status of each stack
stack_master tidy # Find missing or extra templates or parameter files
stack_master compile # Print the compiled version of a given stack
stack_master validate # Validate a template
stack_master lint # Check the stack definition locally using cfn-lint
stack_master nag # Check the stack template with cfn_nagThe apply command does the following:
- Compiles the proposed stack template and resolves parameters.
- Fetches the current state of the stack from CloudFormation.
- Displays a diff of the current stack and the proposed stack.
- Creates a change set and displays the actions that CloudFormation will take to perform the update (if the stack already exists).
- Asks if the update should continue.
- If yes, the API calls are made to update or create the stack.
- Stack events are displayed until CloudFormation has finished applying the changes.
Demo:
stack_master drift us-east-1 mystack uses the CloudFormation APIs to trigger drift detection and display resources
that have changed outside of the CloudFormation stack. This can happen if a resource has been updated via the console or
CLI directly rather than via a stack update.
stack_master diff us-east-1 mystack displays whether the computed parameters or template differ to what was last
applied in CloudFormation. This can happen if the template or computed parameters have changed in code and the change
hasn't been applied to this stack.
StackMaster uses the MIT license. See LICENSE.txt for details.
- Fork it ( http://github.com/envato/stack_master/fork )
- Create your feature branch (
git checkout -b my-new-feature) - Commit your changes (
git commit -am 'Add some feature') - Push to the branch (
git push origin my-new-feature) - Create new Pull Request