Dynamically-generated provider configurations based on a collection value

[dsegurag]

Terraform Version

v0.11.1

Terraform Configuration Files

variable "regions" {
  default = [
    "us-east-1",
    "us-east-2",
    "us-west-1",
    "us-west-2",
    "ca-central-1",
    "eu-central-1",
    "eu-west-1",
    "eu-west-2",
    "eu-west-3",
    "ap-northeast-1",
    "ap-northeast-2",
    "ap-southeast-1",
    "ap-southeast-2",
    "ap-south-1",
    "sa-east-1"
  ]
}

provider "aws" {
  count = "${length(var.regions)}"
  alias = "${element(var.regions, count.index)}"
  region = "${element(var.regions, count.index)}"
  profile = "defualt"
}

resource "aws_security_group" "http-https" {
  count = "${length(var.regions)}"
  provider = "aws.${element(var.regions, count.index)}"
  name = "http-https"
  ingress {
    from_port = 80
    to_port = 80
    protocol = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
  }
  ingress {
    from_port = 443
    to_port = 443
    protocol = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
  }
}

Expected Behavior

Creating a security group on each AWS regions.

Actual Behavior

Planning/Applying fails with

Error: Error asking for user input: 1 error(s) occurred:

* aws_security_group.http-https: configuration for aws.${element(var.regions, count.index)} is not present; a provider configuration block is required for all operations

Steps to Reproduce

1. terraform init
2. terraform apply

[jbardin]

Hi @dsegurag,

The count field isn't supported for providers, and interpolation is not supported in the alias field (provider interpolation). That should probably fail validation earlier, rather than continue on with a provider named aws.${element(var.regions, count.index)}.

Once we can support count on modules, that might be a good way to do something like this in the future. We can review whether having count on providers, or configuring them through modules makes more sense once we have the new configuration infrastructure in place.

[dmikalova]

@jbardin Is there any timeline for adding count to modules? It would solve some of the similar problems that we're experiencing.

[bflad]

Just for reference, a potential use case 😄 : hashicorp/terraform-provider-aws#3763

[hryamzik]

@jbardin do you have any new input on this? What's the supposed way to do this, modules or providers? Any ideas if it'll work with 0.12?

[sprutner]

+1 on this. Right now I have a module that wants to create a ACM Certificate in each region that we have defined in a list of regions. Instead of having to manually add alias providers for every region, this could allow us to easily scale up a region instead of needing to have these added to the module first.

[apparentlymart]

In Terraform v0.12.0 we've reserved the variable name provider so that in a future release module blocks can contain instance-specific provider configurations when using for_each or count:

module "regional" {
  source   = "./modules/regional"
  for_each = var.regions

  provider "aws" {
    region = each.value
  }
}

The above is intended to mean that within that module the default (unaliased) aws provider configuration is set to the one declared inside this block, thus allowing it to incorporate values from each. (If count were being used then that could be count.index instead.)

A design challenge with this is that it causes the same problem as having the provider declaration within the module itself: if you delete the module block then Terraform needs to delete all of the resources it was managing but no longer has the provider configuration that created them.

Therefore we need to do some more design iteration to figure out a suitable design here. The problem could potentially be avoided by allowing for_each on provider configurations themselves, but Terraform's internal handling of providers will need to be significantly reworked for that to be possible:

provider "aws" {
  for_each = var.regions

  region = each.value
}

module "regional" {
  source   = "./modules/regional"
  for_each = var.regions

  providers = {
    aws = aws[each.value]
  }
}

We already have some quite significant refactoring to do in order to support count and for_each on modules, so I expect we'll attack that problem first and while we are working on it look for opportunities to do foundational work for multi-instance provider configurations too. Aside from the complexity of the internal changes required, there is also the unfortunate UX oddity of having both provider aliases and multi-instance providers at the same time, so I think we'll probably also look for whether those two concepts can be combined somehow, because their interactions are likely to be confusing otherwise.

---

The discussion in #21612 is somewhat related to this, in that it is discussing whether region being a provider-wide setting in the AWS provider is actually a good idea anyway, and what it might do instead.

[kayrus]

@apparentlymart are there any plans to introduce this feature in terraform 0.12.x?

[aashitvyas]

+1 to this as I would like to create a CloudWatch Log Group for my implementation in all available AWS Regions.

[abeluck]

Another use case: Setting up standardized AWS Config compliance rules and Cloudwatch Alarms across multiple regions.

[robmoss2k]

Another use case: Setting up standardized State Manager associations across multiple/all regions.

[varunchandak]

Another use case: Setting up standardized GuardDuty across multiple/all regions.

[apparentlymart]

Thanks for continuing to share use-cases!

I think at this point we can safely generalize all of the variants of "Setting up [any AWS regionalized AWS object] across multiple/all AWS regions" as a single use-case, and so I don't think we need to fully enumerate every multi-region AWS service here.

I notice that we left this issue's summary specific to AWS and so I suspect that's causing this to only be visible to folks with AWS-related use-cases, and so I'm going to try to generalize the summary a little so we can see what some corresponding use-cases might look like in other providers which might have.

For example, a difference that might impact use-cases being shared are that some other providers allow setting location-related settings like "region" on a per-resource basis, but may have other provider configuration settings that aren't implemented that way and so would also benefit from dynamic provider configuration generation.

I'm sure there are other provider differences that I'm not thinking of too. If anyone would like to add real-world use-cases from other providers into the mix here I think that will help us to design a solution that generalizes well, vs. an AWS-specific solution implemented inside the AWS provider.

[skokado]

Another use case: Setting up standardized Security Hub across all regions.

[grahammills]

Another use case: Setting up standardized Security Hub across all regions.

I am watching this topic for that exact use case.

[apparentlymart]

So far this issue has, as far as I can see, focused entirely on the problem of dynamically selecting AWS regions. We have not seen any other situations which seem to have the same requirement, but admittedly that could be because this issue had an AWS-region-specific title for most of its life.

The focus on AWS regions makes me inclined to say that this is an AWS provider feature request rather than a Terraform Core one. Other providers for regionalized platforms, such as the Google Cloud Platform provider, treat region as a per-resource setting instead of a provider configuration setting -- the provider configuration setting serves as a convenient default for anything that didn't have an explicit region -- and so there is no corresponding problem in that provider: you can select regions dynamically by directly configuring the per-resource region.

Unless we find some stronger evidence that there's a more general problem that a provider cannot address in its own implementation here, I think we should consider this as feedback on the AWS provider and aim for it to be addressed there. I expect that the AWS provider team will find that it would be easier to meet this requirement with some help from a yet-to-be-designed plugin SDK feature to help handle this idea of an argument that applies to most of the resource types in the provider; the Google Cloud Platform provider uses code generation and so probably has an easier time with this sort of cross-cutting requirement without SDK help. (If there did end up being an SDK feature request under here, the AWS provider team can send that to the SDK team themselves after investigating what the requirements are.)

I'm going to leave this open because I don't think we've yet got enough information to say definitively that this ought to be a provider-specific improvement, but I'd like to renew my request that we try to move this discussion away from anything related to AWS regions to try to learn if there is a more general problem to be solved here, and if so what shape that problem has beyond just making certain settings be per-resource-configurable instead of whole-provider-configurable.

[apowers]

I think it is interesting to note that the desired behavior was possible in Terraform 0.13 via for_each, and then removed in Terraform 0.14.
https://www.terraform.io/language/modules/develop/providers
"Terraform v0.13 introduced the possibility for a module itself to use the for_each, count, and depends_on arguments, but the implementation of those unfortunately conflicted with the support for the legacy pattern. ... but a module with its own provider configurations is not compatible with for_each, count, or depends_on. Terraform will produce an error if you attempt to combine these features."

Since Terraform has apparently decided not to support for_each for providers it is likely up to the AWS provider to figure out how we should iterate over regions.

[apparentlymart]

That capability to declare providers in non-root modules was already deprecated for several versions before removal in v0.14. That mechanism was fundamentally broken because removing from the configuration any module instance which relies on a provider configuration declared inside that instance simultaneously removes the provider configuration, leaving Terraform no way to refresh, plan, or destroy the existing objects connected with that module.

My argument above is based on the observation that there's no particular reason why "region" in the AWS provider must be a provider-configuration-level setting, as demonstrated by other providers using different designs that don't run into this problem. The AWS provider is one of the original Terraform providers and so at the time of designing we didn't have enough experience with provider design to consider that region might actually work better as a per-resource setting instead; it is the way it is as a historical accident. The Google Cloud Platform provider had the advantage of coming later and learning from that (arguably) historical design error, and so it has a design which better reflects the behavior of the underlying system: the region an object belongs to is a property of that object, which sticks with that object even though the provider configuration might've changed.

It's also arguable that Terraform Core itself has a design error whereby it treats provider configurations as something exclusively in the configuration rather than something persisted in the state for future runs. However, we don't yet have have enough evidence to support that position because there are also a lot of provider configuration settings that describe the context where this particular Terraform run is happening. It wouldn't make sense for Terraform to remember the credentials used to create an object as part of its state, for example; that is naturally a per-run setting that must be re-provided for each run and will typically vary for each run in any situation where multiple different people are running Terraform, using their own credentials.

So again, this issue is still open in case we can find evidence that there's a Terraform Core design issue here. But as long as all of the use-cases we're identifying here are for the design of one aspect of one provider, it seems more appropriate to address the problem as part of that provider, particularly when we can see and learn from other providers that already have a different design for the same underlying concern which avoids the problem.

[papagalu]

Hello,

Another use case would be when using the Helm/Kubernetes provider with dynamically number of Kubernetes clusters.
It would be nice to instantiate Helm providers with count/for_each.

Thank you!

[ryandiamond23]

Any progress towards allowing more dynamic provider blocks?

[deitch]

@apparentlymart wrote that the following would be possible once modules support for_each:

provider "aws" {
  for_each = var.regions

  region = each.value
}

module "regional" {
  source   = "./modules/regional"
  for_each = var.regions

  providers = {
    aws = aws[each.value]
  }
}

As far as I know, the ability to loop over modules with count or for_each is solidly in place. Is there any progress on interpolation when declaring the providers { ... } block to pass to them? Do you mind linking to a tracking issue, if one exists?

Actually, even if there were no ability to loop in the provider declaration, just the ability to interpolate on the providers {...} block would mitigate much of the pain:

provider "aws" {
  region = "us-east-1"
  alias = "us-east-1"
}
provider "aws" {
  region = "us-west-1"
  alias = "us-west-1"
}

module "regional" {
  source   = "./modules/regional"
  for_each = var.regions

  providers = {
    aws = aws[each.value]
  }
}

[deitch]

This comment by @apparentlymart hits the nail on the head. It doesn't obviate the value of having the ability to interpolate during providers = { ... } blocks, but the majority of the issues (at least those I have seen in my brief searches) really are about the fact that an AWS provider only can deploy to a single region (whereas, as you pointed out, those learnings changed how you build the Google provider).

Is there tracking issue for it on the AWS provider? I did search in issues on that repo, but it wasn't possible to come up with a sane number of responses?

It would be much better to be able to do:

provider "aws" {
  region = "us-east-1" // this is the _default_ region, but override by resource, if desired
}

module "regional" {
  source   = "./modules/regional"
  for_each = var.regions
  region = each.value
}

and then within the module:

resource "aws_vpc" "main_vpc" {
  region = var.region
}

[julienbonastre]

We're all focusing on regions, how about multi/cross account deployments? Same issue here w.r.t to the nested assume_role block in aws provider.

[robmoss2k]

For the record, a workaround exists, although it's truly horrible. I have a source repository that uses the GitHub provider to generate the providers.tf file dynamically. But, yeah... horrible.