Elasticache redis 6.x support

[ghost]

This issue was originally opened by @kaushik-prasanna as hashicorp/terraform#26570. It was migrated here as a result of the provider split. The original body of the issue is below.

---

With the recent release of Redis 6 by AWS Elasticache, AWS has moved to a single version model for its redis engine.

From the docs :

Beginning with Redis 6, ElastiCache for Redis will offer a single version for each Redis OSS major release, rather than offering multiple minor versions. Versionless engine support is designed to minimize confusion and ambiguity on having to choose from multiple minor versions. ElastiCache for Redis will also automatically keep your cache cluster up to date on the selected major version (Redis 6 and above). By providing the updated engine version, it ensures improved performance and enhanced security.

You specify the engine version by using 6.x. ElastiCache for Redis will automatically invoke the preferred minor version of Redis 6 that is available.

What this means is that the version to be specified in such case is 6.x while creating Elasticache rather than something like 6.0.5.

Coming to the issue, the creation of elasticache works fine when specified as 6.x, but the same thing when run again after creation leads to the problem due to terraform not recognising the version and trying to update it as such.

Sample terraform 0.13.4 code :

resource "aws_elasticache_replication_group" "redis" {
...
  engine                        = "redis"
  engine_version                = "6.x"
  parameter_group_name          = "default.redis6.x"
  port                          = "6379"
...
}

On the first run of terraform apply, the resource gets created. But on second run (with not changes), terraform sees that the redis version created was 6.0.5 and tries to update it to 6.x.

Output of terraform plan :

 # aws_elasticache_replication_group.redis will be updated in-place
  ~ resource "aws_elasticache_replication_group" "redis" {
...
        at_rest_encryption_enabled    = false
        auto_minor_version_upgrade    = true
        automatic_failover_enabled    = false
        engine                        = "redis"
      ~ engine_version                = "6.0.5" -> "6.x"
        parameter_group_name          = "default.redis6.x"
...
    }

And on apply, we get the error :

Error: error updating Elasticache Replication Group : InvalidParameterCombination: Upgrading minor engine version for redis '6.x' using modify API is not allowed, please use Service Updates
status code: 400

From an inital glance, it looks like terraform currently is not aligned with Redis 6.x version from AWS.

Also, we cannot specify specific versions. Hence it's forcing us to downgrade back to Redis 5. Hope this gets solved soon.

Error: Error creating Elasticache Replication Group: InvalidParameterValue: Specific version selection is not allowed for '6.0.5', please use '6.x'

[HaroonSaid]

Any workarounds!

[shuheiktgw]

I've created a PR so please upvote it for a faster review! #15702

[Krule]

Any workarounds!

@HaroonSaid got it to work by accident, before seeing this thread, by first setting engine_version = "6.x" and applying. Afterwards for unrelated change, plan reported it will update version to 6.0.5 (which was the version already). So I updated the terraform code and set engine_version = "6.0.5" explicit.

This resulted in No changes. Infrastructure is up-to-date..

[edvsuperbrave]

I've got these ElastiCache instances that are 5.0.3 and via Terraform we try to update them to 6.

When I use 6.x , terraform plan/apply croaks with the message:

Error: Malformed version: 6.x

When I use 6.0.5, terraform thinks that's fine, but when actually applying it AWS starts to complain:

Error updating ElastiCache cluster (eks-acceptance-bc-payment-service), error: InvalidParameterValue: Specific version selection is not allowed for '6.0.5', please use '6.x' status code: 400, request id: jaddajaddajadda

Now that's fun. Damned if you do, damned if you don't!

[marcincuber]

# module.study_resources["test_name"].module.redis[0].aws_elasticache_replication_group.redis will be updated in-place
  ~ resource "aws_elasticache_replication_group" "redis" {
        apply_immediately             = true
        at_rest_encryption_enabled    = true
        auth_token                    = (sensitive value)
        auto_minor_version_upgrade    = true
        automatic_failover_enabled    = true
        engine                        = "redis"
      ~ engine_version                = "6.0.5" -> "6.x"
     ...

Getting exactly the same issue :(. AWS naming convention again being amazing

[kaushik-prasanna]

@Krule , that's because your state matches what's already deployed (you are inadvertently making it match manually) and hence it does not try to update the redis cache. Try this on a fresh environment and you'll the issue.

[HaroonSaid]

There are two approaches until a fix is created

1. If existing upgrade, then manually upgrade redis to 6.x (6.0.5) using AWS. Follow up running terraform with version 6.0.5

2. New Redis - set version to 6.x run terraform, set version to 6.0.5

[Strainy]

Given you don't really care what the engine value is, you could just add a lifecycle block to ignore the changes as follows:

resource "aws_elasticache_replication_group" "redis" {
  auto_minor_version_upgrade = true

  ...

  # `auto_minor_version_upgrade` is set to true by default, so this engine version may drift
  lifecycle {
    ignore_changes = [engine_version]
  }
}

[HaroonSaid]

Any Updates?

[LozanoMatheus]

Hey, I'm not sure if I should create a new ticket for this or not, but I think there is a bug when it's creating a 6* Cluster.

If you create the cluster using:

resource "aws_elasticache_replication_group" "main" {
  ...
  engine_version             = "6.x"
  ...
}

Then later you can change it to a specific version and it will ok.

resource "aws_elasticache_replication_group" "main" {
  ...
  engine_version             = "6.0.5"
  ...
}

But if you recreate the cluster using the engine_version = "6.0.5", you'll get this error:

Error: Error creating Elasticache Replication Group: InvalidParameterValue: Specific version selection is not allowed for '6.0.5', please use '6.x'
	status code: 400, request id: 3dd4a3f3-9cbc-4fdb-9ad1-b7c0c3ebbd6f

I'm using:

Terraform v0.12.29
+ provider.aws v3.24.1

[rabidscorpio]

@bflad @jbardin @gdavison This issue breaks Redis 6.x support in elasticache and seems like a fairly easy fix(?). Any way to get some love for this issue? As noted before, there's a PR (#15702) but that doesn't take into account Redis versions below 6.

[RRosalia]

He there i have a temporary workaround we currently use in case you don't care about exactly which engine version is running for version 6. You can simply create your cache cluster and ommit the paramater engine_version Below is an example of our working terraform code.

resource` "aws_elasticache_cluster" "cache" {
  cluster_id           = "redis-cluster-${var.environment}"
  engine               = "redis"
  node_type            = var.node_type
  num_cache_nodes      = var.amount_of_nodes
  parameter_group_name = "default.redis6.x"
//  engine_version       = "6.x"
  port                 = 6379
  security_group_ids = var.security_groups
  subnet_group_name = aws_elasticache_subnet_group.redis_subnet_group.name
  apply_immediately = true
  tags = {
    ManagedBy = "Terraform"
    Environment = var.environment
  }
}

resource "aws_elasticache_subnet_group" "redis_subnet_group" {
  name       = "tf-cache-subnet"
  subnet_ids = var.subnets
}

It works both for creating/updating the resource. For us it created the cluster with the newest version. It recognizes version 6 by using the paramater_group_name

[dodge245]

just recently hit this exact issue, does anyone know if its fixed in the latest AWS provider release? Or scheduled to go into one?

[NapalmCodes]

With this closed what release will we see this fix in?

[billputer]

With this closed what release will we see this fix in?

It looks like it'll be in the next 3.38.0 release. 795d147

[ghost]

This has been released in version 3.38.0 of the Terraform AWS provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template for triage. Thanks!

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.