Add ability for Terragrunt to download Terraform configurations #111

brikis98 · 2017-01-25T16:58:32Z

This PR adds a new element you can specify in the .terragrunt file:

terraform {
  source = "git::git@github.com:foo/bar.git//my-terraform-configurations?ref=v0.0.1"
}

When you specify this parameter, Terragrunt will download the Terraform configurations at the source URL to a temporary folder and run Terraform in that temporary folder.

This makes it easy to version all of your Terraform configurations and deploy a specific version automatically, similar to Kief Morris’ pipelines idea.

You can also override the source parameter with the --terragrunt-source command-line flag and TERRAGRUNT_SOURCE environment variable, which makes it easy to use a local checkout for iterative development.

See the Readme for full details.

…ote state

brikis98 · 2017-01-26T15:07:59Z

@josh-padnick Please take a look when you have a chance. I'd like to merge this in today.

brikis98 · 2017-01-26T15:25:40Z

Also, after using this for a bit with a real-world use case, here are a few follow-up tasks:

Instead of downloading the code into a new tmp folder each time, find a way to reuse an existing folder and intelligently update it. Otherwise, if your code includes lots of modules, having to run terraform get from scratch each time can be slow.
Find some way to override the source parameter for the spin-up and tear-down commands so you can spin up an entire stack from a local checkout.

Consider if Terragrunt should be able to read its configuration not only from a .terragrunt file, but also a terragrunt key within a terraform.tfvars file:

# Normal params in terraform.tfvars
foo = "bar"

# Terragrunt params
terragrunt {
  lock { ... }
  remote_state { ... }
}

The advantage of that is that you have one less file and folder to manage for each component:

infrastructure-live
  └ stage
    └ frontend-app.tfvars
    └ backend-app.tfvars
    └ search-app.tfvars
    └ mysql.tfvars
    └ redis.tfvars
    └ vpc.tfvars
  └ prod
    └ frontend-app.tfvars
    └ backend-app.tfvars
    └ search-app.tfvars
    └ mysql.tfvars
    └ redis.tfvars
    └ vpc.tfvars

Moreover, instead of worrying about a special "root" .terragrunt file, you can a root .tfvars file with shared configs (or even multiple shared .tfvars files), as Terraform merges configs together automatically.

josh-padnick · 2017-01-26T19:04:17Z

Looking now...

josh-padnick

This looks great overall. Most comments pertain to core ideas, not to code itself. Feel free to merge when ready.

josh-padnick · 2017-01-26T19:17:11Z

README.md

+especially useful if you have versioned all of your Terraform code and want to `apply` a specific version of it
+(see [Using Pipelines to Manage Environments with Infrastructure as 
+Code](https://medium.com/@kief/https-medium-com-kief-using-pipelines-to-manage-environments-with-infrastructure-as-code-b37285a1cbf5)).
+


This might be hard to understand as someone's first into to this concept. Consider first describing the problem that we're solving (e.g. "A common gripe of Terragrunt users is that launching a new environment is cumbersome. Much copy and pasting is required, and it can be cumbersome to identify the specific vars that change between environments. One approach to solving this is to...")

Good point. I just updated the README to make the problem clearer: cd3f23b

josh-padnick · 2017-01-26T19:19:23Z

README.md

+terraform {
+  source = "git::git@github.com:foo/bar.git//my-terraform-configurations?ref=v0.0.1"
+}
+```


+1 on this syntax, which mirrors Terraform's.

josh-padnick · 2017-01-26T19:22:01Z

README.md

+   [terraform init command](https://www.terraform.io/docs/commands/init.html), so the `source` parameter supports the
+   exact same syntax as the [module source](https://www.terraform.io/docs/modules/sources.html) parameter, including
+   local file paths, Git URLs, and Git URLs with `ref` parameters (useful for checking out a specific tag, commit, or
+   branch of Git repo).


Nice use of terraform init. Didn't even realize this command existed!

josh-padnick · 2017-01-26T19:23:44Z

README.md

+  └ mysql
+  └ redis
+  └ vpc
+```


This isn't the ideal place for this comment, but it occurs to me that this infra-modules folder structure should use folder groupings like:

- services - frontend-app - main.tf - backend-app - data-stores - mysql - redis

josh-padnick · 2017-01-26T19:25:09Z

README.md

+    └ mysql
+    └ redis
+    └ vpc
+```


Love this folder structure. I agree with your follow-up comment that combining .terragrunt and terraform.tfvars removes the need for a folder and keeps things simpler.

josh-padnick · 2017-01-26T19:27:59Z

README.md

+
+Note that you can also use the `--terragrunt-source` command-line option and `TERRAGRUNT_SOURCE` environment variable
+to override the `source` parameter. This is useful to point Terragrunt at a local checkout of your code so you can do 
+rapid, iterative, make-a-change-and-rerun development:


josh-padnick · 2017-01-26T19:32:44Z

README.md

+cd infrastructure-live/stage/frontend-app
+terragrunt apply --terragrunt-source ../../../infrastructure-modules/frontend-app
+```   
+


First, this is fantastic. I am really excited to start playing around with this.

Second, just a sanity check. So, let's say one of our customers is using our vpc modules. Then here's what the chain of calls looks like:

client/infrastructure-live: vpc

client/infrastructure-modules: vpc

gruntwork/module-vpc: vpc-app

So to state the obvious, this adds one more layer to our module calls than we had before. But I don't really see any downsides to this. Of course, this does raise the question of how we handle our traditional infrastructure-modules, like ecs-cluster. For example:

client/infrastructure-live: stage/ecs-cluster

client/infrastructure-modules: ecs-cluster

client/infrastructure-modules: modules/ecs-cluster

gruntwork/module-ecs: ecs-cluster

It means one infrastructure-module might want to make a versioned reference to another infrastructure-module. How are you addressing that? Might be better for offline or out-of-band discussion since it doesn't directly pertain to this PR.

So to state the obvious, this adds one more layer to our module calls than we had before.

It's not really one more layer. There is no Terraform code in infrastructure-live. The only Terraform code is in infrastructure-modules, which has a module dependency on module-vpc. The infrastructure-live repo just fills in the environment-specific variables for the infrastructure-modules code.

Of course, this does raise the question of how we handle our traditional infrastructure-modules, like ecs-cluster.

In your diagram, you had two entries for ecs-cluster in infrastructure-modules, but we'd really only have one. So in that regard, this code is simplified.

josh-padnick · 2017-01-26T19:35:13Z

cli/cli_app.go

+		if err := checkoutTerraformSource(conf.Terraform.Source, terragruntOptions); err != nil {
+			return err
+		}
+	}


Can you explain what condition this is meant to handle?

There are two ways to tell Terragrunt that it needs to download some Terraform templates:

The --terragrunt-source command-line option.

A terraform block in .terragrunt.

These if-statements pick them out in proper priority order. That said, this code was a bit convoluted for that, so I've refactored it as follows: 8ecd053

brikis98 · 2017-01-26T23:06:21Z

OK, I'm going to merge this one in.

josh-padnick · 2017-01-27T00:13:49Z

README.md

+frontend-app, backend-app, vpc, etc). As the number of components and environments grows, having to maintain more and 
+more code can become error prone. You can significantly reduce the amount of copy paste using [Terraform 
+modules](https://blog.gruntwork.io/how-to-create-reusable-infrastructure-with-terraform-modules-25526d65f73d), but even 
+the code to use the module module and set up input variables, output variables, providers, and remote state can still


s/module module/module.

Fixed that in a follow-up commit :)

josh-padnick · 2017-01-27T00:15:30Z

README.md


-Note that you can also use the `--terragrunt-source` command-line option and `TERRAGRUNT_SOURCE` environment variable
+Note that you can also use the `--terragrunt-source` command-line option or the `TERRAGRUNT_SOURCE` environment variable
 to override the `source` parameter. This is useful to point Terragrunt at a local checkout of your code so you can do 
 rapid, iterative, make-a-change-and-rerun development:



These docs are great.

brikis98 added 7 commits January 25, 2017 15:51

Add ability for Terragrunt to checkout Terraform code

6dcdcd3

Fix branch name in test fixture

03f0c06

Fix copy/paste error in test

dfaa64a

Fix CopyFolder function

13c98d3

Fix URL in remote checkout test case

64c4082

Add support for overriding checkout URL via cmd line option

be72dc0

Need to download source before downloading modules or configuring rem…

a6e615d

…ote state

josh-padnick approved these changes Jan 26, 2017

View reviewed changes

brikis98 added 5 commits January 26, 2017 22:47

Update README to make it clearer why Terragrunt can download code

cd3f23b

Shorten file structure example

3f7f267

Typo fix in README

796fae2

Fix a few more typos in the README

8b0d188

Simplify code for handling source URLs

8ecd053

brikis98 merged commit df5ccd1 into master Jan 26, 2017

brikis98 deleted the checkout branch January 26, 2017 23:06

josh-padnick reviewed Jan 27, 2017

View reviewed changes

brikis98 mentioned this pull request Jan 29, 2017

Cache temporary folder #114

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add ability for Terragrunt to download Terraform configurations #111

Add ability for Terragrunt to download Terraform configurations #111

brikis98 commented Jan 25, 2017

brikis98 commented Jan 26, 2017

brikis98 commented Jan 26, 2017

josh-padnick commented Jan 26, 2017

josh-padnick left a comment

josh-padnick Jan 26, 2017

brikis98 Jan 26, 2017

josh-padnick Jan 26, 2017

josh-padnick Jan 26, 2017

josh-padnick Jan 26, 2017

josh-padnick Jan 26, 2017

josh-padnick Jan 26, 2017

josh-padnick Jan 26, 2017

brikis98 Jan 26, 2017

josh-padnick Jan 26, 2017

brikis98 Jan 26, 2017

brikis98 commented Jan 26, 2017

josh-padnick Jan 27, 2017

brikis98 Jan 27, 2017

josh-padnick Jan 27, 2017

Add ability for Terragrunt to download Terraform configurations #111

Add ability for Terragrunt to download Terraform configurations #111

Conversation

brikis98 commented Jan 25, 2017

brikis98 commented Jan 26, 2017

brikis98 commented Jan 26, 2017

josh-padnick commented Jan 26, 2017

josh-padnick left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

brikis98 commented Jan 26, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment