Terraform S3 Remote State using only credential profiles (no default credentials)

[mikereinhold]

Terraform remote state S3 backend does not properly retrieve remote state using the profile config option alone if there is no default AWS credentials.

Example, consider a multiple account configuration using assumed administrative roles across accounts. I may not want to have a [default] profile configured in ~/.aws/credentials, set in Terraform, or in environment variables. For example, maybe because I want to prevent accidentally creating resources in the account that has the IAM account when the correct provider is not specified for each resource.

If I only specify the S3 config profile, the data source fails to find AWS credentials. If I add the provider line, terraform complains that the provider does not support terraform_remote_state.

Terraform Version

Terraform v0.7.3

Affected Resource(s)

• data source terraform_remote_state
• provider/aws

Terraform Configuration Files

data "terraform_remote_state" "shared-state"{
    provider = "aws.admin"
    backend = "s3"
    config {
        bucket = "${var.state-bucket}"
        key = "${var.group}/${var.project}/${var.environment}.tfstate"
        region = "${var.state-region}"
        profile = "admin"
    }
}

Expected Behavior

Terraform should use the specified provider for the Terraform remote state data source.

Alternatively, in order to avoid needing to specify the provider at all, the data source should consult the specific config (in this case S3 config) in order to determine the correct provider. This would probably be preferable for DRY.

Actual Behavior

If no provider is specified, just using the S3 config profile:

• No valid credential sources found for AWS Provider.
  Please see https://terraform.io/docs/providers/aws/index.html for more information on
  providing credentials for the AWS Provider

If the provider is specified (with or without the S3 config profile):

• data.terraform_remote_state.devyse-domains: Provider doesn't support data source: terraform_remote_state

Steps to Reproduce

Configure a terraform_remote_state as described above, with an S3 config profile (with or without the provider line, pending consideration for which approach is "correct").
Ensure an AWS provider using alias admin exists, referencing an AWS Config profile admin. No default AWS credentials should be found (no env vars, no default AWS config profile / credentials, no default Terraform provider that supplies the creds.

1. terraform plan

[mikereinhold]

I did some more testing on this - what I've found is that I configured my AWS credentials and config files based on Amazon's documented method for using cross account role assumption. Essentially the access key id and secret key go in the credentials file while the role assumption configuration goes into the config file.

Ex:
~/.aws/credentials

[admin]
aws_access_key_id = <redacted>
aws_secret_access_key = <redacted>

~/.aws/config

[default]
region = us-east-1
output = json

[preview]
cloudfront = true

[profile one]
role_arn = <redacted>
source_profile = admin

I misunderstood the Terraform AWS provider documentation - I was under the impression that it read both the credentials file and the config file, however my issue noted above is because it only reads the credentials file.

Using the assume role configuration block for the AWS provider works properly. It's too bad Terraform can't just read the config file in addition to the credentials file because then I wouldn't have to configure the role assumption as much (once for each Terraform project). Instead I could just configure the AWS CLI and let Terraform use that existing configuration.

[farvour]

Just ran into this issue when trying to access state from an s3 bucket defined in another AWS account. This would seem like important functionality for another account to export a root output, and be accessible via another account (e.g.: security group IDs or whatever else).

I believe the only workaround to this would be to write an IAM policy to allow access to the S3 bucket in the other account with the account/profile from the default aws provider. Not really a solution :(

[jbruett]

This happens in 0.9.2 when just trying to configure S3 Remote State storage as well. The terraform init commands fail if you're not using the default profile. It's as if it's completely not seeing the profile argument. Not sure why it wouldn't just use credentials passed by the provider.

[chadbean]

@farvour what I ended up doing is setting the role_arn from within the config {} block, like:

data "terraform_remote_state" "remote-state" {
  backend = "s3"
  config {
    bucket   = "${var.bucket_name}"
    key      = "base/terraform.state"
    region   = "${var.aws_region}"
    role_arn = "arn:aws:iam::XXXXXX:role/Administrator"
  }
}

@mikereinhold does this solve your issue as well?

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.