[Bug]: Provider fails to refresh aws_elasticache_replication_group if manually deleted

[Cyanopus]

Terraform Core Version

0.14.11

AWS Provider Version

5.5.0, 4.67.0

Affected Resource(s)

resource "aws_elasticache_replication_group"

Expected Behavior

When the resource is deleted in the cloud provider console, the next run in Terraform should refresh and suggest creating the group on the plan and applying it.

Actual Behavior

When the resource is deleted in the console, on the next run of terraform plan/refresh, the result is:

Error: unable to find ElastiCache Parameter Group (<name_of_the_resource>): couldn't find resource

It fails consistently in this situation when the group is deleted manually outside of Terraform.

Relevant Error/Panic Output Snippet

Error: unable to find ElastiCache Parameter Group (<name_of_the_resource>): couldn't find resource

Terraform Configuration Files

variable "product_name" {
  description = "(Required) This is the product name. This is usually required parameter."
  type        = string
}

variable "product_env" {
  description = "(Required) This is the product evironment in which this module is deployed. This is usually terraform.workspace."
  type        = string
}

variable "product_version" {
  description = "(Required) The version of the product"
  type        = string
}

variable "vpc_id" {
  description = "(Required) The ID of the VPC"
  type        = string
}

variable "security_group_ids" {
  description = "(Optional) One or more Amazon VPC security groups associated with this replication group."
  type        = list(string)
  default     = []
}

variable "subnet_ids" {
  description = "(Required) List of VPC Subnet IDs for the cache subnet group"
  type        = list(string)
}

variable "ec_subnet_name" {
  description = "(Optional) The name of the Elasticache subnet group"
  type        = string
  default     = "ec"
}

variable "ec_subnet_description" {
  description = "(Optional) Description for the Elasticache subnet group."
  type        = string
  default     = ""
}

variable "ec_pg_name" {
  description = "(Optional) The name of the Elasticache parameter group"
  type        = string
  default     = "ec"
}

variable "ec_pg_description" {
  description = "(Optional) The description of the ElastiCache parameter group."
  type        = string
  default     = ""
}

variable "ec_backend" {
  description = "(Required) The desired backend of the elasticache instance. The value should be between `redis` and `memcached`. Once the backend is set it cannot be changed"
  type        = string
}

variable "ec_cluster_id" {
  description = "(Required) The identifier of Elasticache cluster. Should be in lowercase."
  type        = string
}

variable "family" {
  description = "(Required) The family of the ElastiCache parameter group. Valid values are: memcached1.4, memcached1.5, memcached1.6, redis2.6, redis2.8, redis3.2, redis4.0, redis5.0, redis6.x"
  type        = string
}

variable "tags" {
  description = "(Optional) User-defined tags that will be merged to the standards."
  type        = map(string)
  default     = {}
}

variable "ec_parameters" {
  description = "(Optional) A list of Redis or Memcached parameters to apply. Note that parameters may differ from one Redis family to another"
  type = list(object({
    name  = string
    value = string
  }))
  default = []
}

variable "engine_version" {
  description = "(Optional) The version number of the cache engine to be used for the cache cluster."
  type        = string
}

variable "instance_type" {
  description = "(Required) The compute and memory capacity of the nodes in the node group. Elasticcache instance type"
  type        = string
  default     = "cache.t2.micro"
}

variable "ec_cluster_size" {
  description = "(Required when ec_cluster_mode_enabled == false) The number of cache clusters (primary and replicas) this replication group will have. If Multi-AZ is enabled, the value of this parameter must be at least 2. Updates will occur before other modifications."
  type        = number
  default     = 1
}

variable "port" {
  description = "(Optional) The port number on which each of the cache nodes will accept connections. For Memcache the default is 11211, and for Redis the default port is 6379."
  type        = number
}

variable "availability_zones" {
  description = "(Optional) A list of EC2 availability zones in which the replication group's cache clusters will be created. The order of the availability zones in the list is not important."
  type        = list(string)
  default     = []
}

variable "maintenance_window" {
  description = "(Optional) Specifies the weekly time range for when maintenance on the cache cluster is performed. The format is ddd:hh24:mi-ddd:hh24:mi (24H Clock UTC). The minimum maintenance window is a 60 minute period."
  type        = string
  default     = null
}

variable "notification_topic_arn" {
  description = "(Optional) An Amazon Resource Name (ARN) of an SNS topic to send ElastiCache notifications to. Example: arn:aws:sns:us-east-1:012345678999:my_sns_topic"
  type        = string
  default     = ""
}

variable "snapshot_window" {
  description = "(Optional) The daily time range (in UTC) during which ElastiCache will begin taking a daily snapshot of your cache cluster. The minimum snapshot window is a 60 minute period. Example 05:00-09:00"
  type        = string
  default     = null
}

variable "snapshot_retention_limit" {
  description = "(Optional, Redis Only) The number of days for which ElastiCache will retain automatic cache cluster snapshots before deleting them. For example, if you set SnapshotRetentionLimit to 5, then a snapshot that was taken today will be retained for 5 days before being deleted. If the value of SnapshotRetentionLimit is set to zero (0), backups are turned off. Please note that setting a snapshot_retention_limit is not supported on cache.t1.micro or cache.t2.* cache nodes"
  type        = number
  default     = 0
}

variable "apply_immediately" {
  description = "(Optional) Specifies whether any modifications are applied immediately, or during the next maintenance window."
  type        = bool
  default     = true
}
# Redis backend scpecific variables
variable "ec_cluster_mode_enabled" {
  description = "(Optional) Flag to enable/disable creation of a native redis cluster. `automatic_failover_enabled` must be set to `true`."
  type        = bool
  default     = false
}

variable "replication_group_description" {
  description = "(Required) A user-created description for the replication group"
  type        = string
  default     = "Managed by Terraform"
}

variable "replication_group_multi_az_enabled" {
  description = "(Optional) Specifies whether to enable Multi-AZ Support for the replication group. If true, automatic_failover_enabled must also be enabled. Defaults to false."
  type        = bool
  default     = false
}
variable "automatic_failover_enabled" {
  description = "(Optional) Specifies whether a read-only replica will be automatically promoted to read/write primary if the existing primary fails. If true, Multi-AZ is enabled for this replication group. If false, Multi-AZ is disabled for this replication group. Must be enabled for Redis (cluster mode enabled) replication groups.Automatic failover (Not available for T1/T2 instances)"
  type        = bool
  default     = false
}

variable "at_rest_encryption_enabled" {
  description = "(Optional) Whether to enable encryption at rest."
  type        = bool
  default     = true
}

variable "transit_encryption_enabled" {
  description = "(Optional) Whether to enable encryption in transit."
  type        = bool
  default     = true
}

variable "auth_token" {
  description = "(Optional) The password used to access a password protected server. Can be specified only if `transit_encryption_enabled` = `true`."
  type        = string
  default     = null
}

variable "kms_key_id" {
  description = "(Optional) The ARN of the key that you wish to use if encrypting at rest. If not supplied, uses service managed encryption. Can be specified only if at_rest_encryption_enabled = true"
  type        = string
  default     = null
}

variable "ec_cluster_mode_replicas_per_node_group" {
  description = "(Optional) Specify the number of replica nodes in each node group. Valid values are 0 to 5. Changing this number will force a new resource."
  type        = number
  default     = 0
}

variable "ec_cluster_mode_num_node_groups" {
  description = "(Optional) Specify the number of node groups (shards) for this Redis replication group. Changing this number will trigger an online resizing operation before other settings modifications."
  type        = number
  default     = 0
}


terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.0"
    }
  }
  required_version = ">= 0.14"
}

locals {
  common_tags = {
    Terraform = "true"
  }

  tags = merge(local.common_tags, var.tags)
}

resource "aws_elasticache_subnet_group" "ec" {
  name        = format("%s-%s-%s-subnet-group", var.product_name, var.product_env, var.ec_subnet_name)
  subnet_ids  = var.subnet_ids
  description = format("Managed by Terraform. %s", var.ec_subnet_description)
}

resource "aws_elasticache_parameter_group" "ec" {
  name        = format("%s-%s-%s-pg", var.product_name, var.product_env, var.ec_pg_name)
  family      = var.family
  description = format("Managed by Terraform. %s", var.ec_pg_description)

  dynamic "parameter" {
    for_each = var.ec_parameters
    content {
      name  = parameter.value.name
      value = parameter.value.value
    }
  }

  lifecycle {
    create_before_destroy = true
  }
}

resource "aws_elasticache_replication_group" "ec" {
  count = var.ec_backend == "redis" ? 1 : 0

  engine         = var.ec_backend
  engine_version = var.engine_version

  parameter_group_name = aws_elasticache_parameter_group.ec.name
  subnet_group_name    = aws_elasticache_subnet_group.ec.name
  security_group_ids   = var.security_group_ids
  replication_group_id = lower(var.ec_cluster_id)
  description          = var.replication_group_description

  multi_az_enabled            = var.replication_group_multi_az_enabled
  node_type                   = var.instance_type
  num_cache_clusters          = var.ec_cluster_mode_enabled ? null : var.ec_cluster_size
  port                        = var.port
  preferred_cache_cluster_azs = var.ec_cluster_mode_enabled ? null : slice(var.availability_zones, 0, var.ec_cluster_size)
  automatic_failover_enabled  = var.replication_group_multi_az_enabled ? true : var.automatic_failover_enabled

  at_rest_encryption_enabled = var.at_rest_encryption_enabled
  kms_key_id                 = var.at_rest_encryption_enabled ? var.kms_key_id : null
  transit_encryption_enabled = var.transit_encryption_enabled
  auth_token                 = var.transit_encryption_enabled ? var.auth_token : null

  notification_topic_arn   = var.notification_topic_arn
  maintenance_window       = var.maintenance_window
  snapshot_window          = var.snapshot_window
  snapshot_retention_limit = var.snapshot_retention_limit
  apply_immediately        = var.apply_immediately

  replicas_per_node_group = var.ec_cluster_mode_enabled ? var.ec_cluster_mode_replicas_per_node_group : null
  num_node_groups         = var.ec_cluster_mode_enabled ? var.ec_cluster_mode_num_node_groups : null
  tags                    = local.tags
}

Steps to Reproduce

1. Terraform Apply the resource
2. Go to the AWS console for Elasticache and delete the Elasticache cluster and the Replication Group
3. Terraform apply the resource again
4. Observe the error:

Error: unable to find ElastiCache Parameter Group (<name_of_the_resource>): couldn't find resource

Debug Output

No response

Panic Output

No response

Important Factoids

No response

References

No response

Would you like to implement a fix?

None

[github-actions]

Community Note

Voting for Prioritization

• Please vote on this issue by adding a 👍 reaction to the original post to help the community and maintainers prioritize this request.
• Please see our prioritization guide for information on how we prioritize.
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.

Volunteering to Work on This Issue

• If you are interested in working on this issue, please leave a comment.
• If this would be your first contribution, please review the contribution guide.

[github-actions]

This functionality has been released in v5.10.0 of the Terraform AWS Provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template. Thank you!

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.