Plan-Time Resource Uniqueness Errors

[bflad]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request

Description

In any sufficiently large Terraform environment, there is potential for multiple resources to be planned for creation that cannot be successfully applied due to API uniqueness constraints. Operators have repeatedly asked for enhancements to Terraform and Terraform AWS Provider to provide this type of validation at plan-time. This issue serves as a meta-enhancement request to collect community interest around this for prioritization and prevent further issue sprawl with individual resource requests that are harder to collectively prioritize as part of the same effort.

An example Terraform configuration that should generate one of these types of errors:

# Trivial example of overlapping resource management.
# Usually these issues are spread across many Terraform
# configuration files or modules. :)

resource "aws_backup_vault" "example1" {
  name = "example"
}

resource "aws_backup_vault" "example2" {
  name = "example"
}

Resource uniqueness constraints are almost exclusively resource type dependent and in the case of AWS service APIs, may include any number of the following as potential constraints:

• Per AWS Partition
• Per AWS Account
• Per AWS Region
• Per resource attribute (e.g. identifier or name)
• Per parent resource attribute (e.g. one policy resource for a resource)

Some examples of these uniqueness constraints:

• aws_backup_vault (1 name value per AWS Region per AWS Partition per AWS Account ID)
• aws_dynamodb_table (1 name value per AWS Region per AWS Partition per AWS Account ID)
• aws_ebs_default_kms_key (1 per AWS Region per AWS Partition per AWS Account ID)
• aws_ebs_encryption_by_default (1 per AWS Region per AWS Partition per AWS Account ID)
• aws_iam_account_alias (1 per AWS Partition per AWS Account ID)
• aws_iam_account_password_policy (1 per AWS Partition per AWS Account ID)
• aws_iam_group_policy (1 name value per group value per AWS Partition per AWS Account ID)
• aws_iam_role_policy (1 name value per role value per AWS Partition per AWS Account ID)
• aws_iam_user_policy (1 name value per user value per AWS Partition per AWS Account ID)
• aws_s3_bucket (1 bucket value per AWS Partition)
• aws_s3_bucket_notification (1 bucket value per AWS Partition)
• aws_s3_bucket_policy (1 bucket value per AWS Partition)
• aws_sns_topic (1 name value per AWS Region per AWS Partition per AWS Account ID)

These uniqueness constraints must be checked in various lifecycles of two conflicting resources including:

• When none exist and both are being created
• When one exists and one is being created/updated
• When both exist and both are being updated/recreated

Only comparing resources in the new, expected Terraform state of any planning operation should cover these cases.

While a provider, such as the Terraform AWS Provider, could implement its own framework for handling this type of issue, it would require some additional amount effort due how Terraform core interoperates with provider resource planning today. Essentially, each provider instance would need to cache (in the meta) resource uniqueness keys for individual resources (per resource type) in their CustomizeDiff handling and return an error if the cached key already exists. Another consideration is that CustomizeDiff does not know about a resource's type, so this would need to be manually synthesized into the resource uniqueness handling.

A small effort could be done as a proof of concept implementation in the Terraform AWS Provider, however the Terraform Plugin SDK seems a much better fit to standardize the implementation into resource definitions and for standardizing the error handling. Given this, operators are encouraged to also 👍 the Terraform Plugin SDK issue below, if interested.

References

• Terraform core: Global IDs for representing relationships between resource objects (object containment, name collision detection, etc) terraform#22094
• Terraform Plugin SDK: Allow provider to define uniqueness of a resource terraform-plugin-sdk#224
• Similar but different enhancement to provide better error handling when a conflicting resource API error is found: Enhance Error Messaging for Resources Requiring Import #9223

Related Terraform AWS Provider feature requests:

• Multiple aws_s3_bucket with the same bucket name does not generate an error #367
• aws_s3_bucket_notification - lambda - append new instead of deleting all previous #501
• Multiple aws_sqs_queue_policy adds only one policy to queue #539
• Multiple codecommit triggers #3209
• Declaring 2 aws_ecr_repository_policy for a single repository overwrites the 1st one #3737
• More than 2 ECR lifecycle policies can't be defined for one ECR repository #6212
• creating multiple aws_sns_topic with the same name #6245
• Applying a plan with SQS queue with attributes matching an existing queue results in an import not a creation. #10824
• aws_iam_role_policy name collision #14387
• Creating/modifying multiple SQS queues with the same name doesn't fail #15429
• Warning when creating multiple aws_s3_bucket_notification resources on the same bucket #17842
• aws_apprunner_service is unable to detect an existing apprunner service deployed by another terraform #19776

[ewbankkit]

@bflad Do you see this issue also addressing cases like #15786, #10473 or #10930 where a resource is being renamed in code without the equivalent terraform state mv, causing resource deletion and creation (for the same underlying AWS resource) to occur concurrently?

[jorhett]

3 years now and all of those major issues linked to this and closed have no answers

[mcncl]

Keen on finding out the status of this too. We're looking at adding this feature in to our own provider and it doesn't look possible right now.

[jwieder]

This issue also impacts creation of aws_route resources. aws_route resources should be unique for combination of destination CIDR & rtbl ID. AWS API doesnt catch this, either, which can result in deletion of default routes.

[nronnei]

Feeling the pain with aws_s3_bucket_notification right now. I need a) a topic trigger and b) a lambda_function trigger. The lambda and topic are managed by separate teams with limited communication, so while I can trigger the lambda from the topic, they'd ideally be separate triggers so that the teams which own them could modify them without causing unintended problems for one another (e.g. by adding a filter_prefix).

The plan output looks like this (with names/ids redacted).

resource "aws_s3_bucket_notification" "my_bucket_triggers" {
  bucket      = "my-bucket"
  eventbridge = false

  lambda_function {
      events              = ["s3:ObjectCreated:*"]
      id                  = "MyLambdaEvent"
      lambda_function_arn = "arn:aws:lambda:us-west-2:<account_id>:function:<fn_name>"
    }

  topic {
      events    = ["s3:ObjectCreated:*"]
      id        = "MySnsEvent"
      topic_arn = "arn:aws:sns:us-west-2:<account_id>:<topic_name>"
    }
}

When I try to combine both into a single aws_s3_bucket_notification I get the following error:

api error InvalidArgument: Configurations overlap. Configurations on the same bucket cannot share a common event type.

When I try to separate aws_s3_bucket_notification resources, I get the perpetual difference problem described in the issue that led me here.