Set AWS credentials in the environment so go-getter can use them

[jbergknoff-rival]

In going from 0.18.7 to 0.19.0, the method for downloading Terraform sources changed from terraform init to the hashicorp/go-getter library. This introduces a regression where an artifact in S3 can no longer be fetched using an assumed role.

.ssh/config

[ops]
aws_access_key_id=...
aws_secret_access_key=...

Failing output with Terragrunt 0.19.8 and Terraform 0.12.3:

$ export AWS_PROFILE=ops TERRAGRUNT_IAM_ROLE=arn:aws:iam::1234567890:role/my_role
$ terragrunt -version AWS_SDK_LOAD_CONFIG=true
terragrunt version v0.19.8
$ terragrunt init
[terragrunt] [/path/to/tfvars] 2019/07/03 17:29:27 Running command: /bin/terraform --version
[terragrunt] 2019/07/03 17:29:28 Reading Terragrunt config file at /path/to/tfvars/terragrunt.hcl
[terragrunt] 2019/07/03 17:29:28 Assuming IAM role arn:aws:iam::1234567890:role/my_role
[terragrunt] 2019/07/03 17:29:28 Downloading Terraform configurations from s3::https://s3-us-west-2.amazonaws.com/bucket/path/to/artifact.zip into /path/to/tfvars/.terragrunt-cache/uDnJlgn36Yz7t4-tE1QuPQJn2Ys/zTAc8OMcrkmYXjA_RJsARenWbMw
[terragrunt] 2019/07/03 17:29:28 Hit multiple errors:
AccessDenied: Access Denied
    status code: 403, request id: 06341E42B14F7178, host id: AxrAQ5qj61+r1buzdVlxun2Qr+GZhUUdME+OpqPibQDnjMKQl6lwSpCcn7oqbRIPsCW/pN5uQg8=
[terragrunt] 2019/07/03 17:29:28 Unable to determine underlying exit code, so Terragrunt will exit with error code 1

Output with Terragrunt built off of this PR:

$ terragrunt init 
[terragrunt] [/path/to/tfvars] 2019/07/03 17:16:00 Running command: /bin/terraform --version
[terragrunt] 2019/07/03 17:16:01 Reading Terragrunt config file at /path/to/tfvars/terragrunt.hcl
[terragrunt] 2019/07/03 17:16:01 Assuming IAM role arn:aws:iam::1234567890:role/my_role
[terragrunt] 2019/07/03 17:16:01 Downloading Terraform configurations from s3::https://s3-us-west-2.amazonaws.com/bucket/path/to/artifact.zip into /path/to/tfvars/.terragrunt-cache/uDnJlgn36Yz7t4-tE1QuPQJn2Ys/zTAc8OMcrkmYXjA_RJsARenWbMw
[terragrunt] 2019/07/03 17:16:02 Copying files from /path/to/tfvars into /path/to/tfvars/.terragrunt-cache/uDnJlgn36Yz7t4-tE1QuPQJn2Ys/zTAc8OMcrkmYXjA_RJsARenWbMw/modules
[terragrunt] 2019/07/03 17:16:02 Setting working directory to /path/to/tfvars/.terragrunt-cache/uDnJlgn36Yz7t4-tE1QuPQJn2Ys/zTAc8OMcrkmYXjA_RJsARenWbMw/modules
[terragrunt] 2019/07/03 17:16:02 Initializing remote state for the s3 backend
[terragrunt] 2019/07/03 17:16:02 Running command: /bin/terraform init -backend-config=region=us-west-2 -backend-config=bucket=... -backend-config=dynamodb_table=... -backend-config=encrypt=true -backend-config=key=...
Initializing modules...

This is likely actually an issue with the go-getter library not properly handling AWS credentials, but the change in this PR is a lower effort fix. I've tested that this fixes the issue in my setup. I've also tested separately that having Terragrunt pull in hashicorp/go-getter#185 (which looks related) does not fix my issue.

[ekini]

I was going through the terragrunt code yesterday and found very good comments in it, explaining why something was done. So for this, I would add a note as well, something like "it's for go-getter not doing the auth right".

Just my 2 cents.

[brikis98]

Thanks for the PR!

I'm grateful for the attempt at a simple fix, but I'm 95% sure this approach is going to cause other problems:

1. We run Terragrunt's tests in parallel, some of which use different auth strategies. If we set global env vars here, that will affect all tests, and cause strange, intermittent behavior depending on which tests happen to run in parallel.
2. Terragrunt itself runs things in parallel when you run xxx-all commands (e.g., apply-all). If you have different modules that use different auth settings, this again will cause strange race conditions based on timing.

So, I suspect that either we need a way to explicitly set auth values when executing an instance of go-getter, if it's API exposes that ability... And if not, fix the bug in go-getter itself.