Terraform gives error while trying to upgrade PostgreSQL primary and read-replica in same module

[satish-chef]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform CLI and Terraform AWS Provider Version

Terraform v0.12.20
provider.aws v3.37.0

Affected Resource(s)

• aws_db_instance

Terraform Configuration Files

Child module:
resource aws_db_parameter_group this {
  name   = "abc"
  family = var.db_parameter_group_family


  lifecycle {
    create_before_destroy = true
  }
}


resource aws_db_instance this {
  identifier                = "abc"
  final_snapshot_identifier = var.final_snapshot_identifier

  lifecycle {
    ignore_changes = [
      snapshot_identifier,
      password
    ]
  }
  username                        = local.username
  password                        = local.password
  name                            = var.db_name
  engine                          = "postgres"
  engine_version                  = var.engine_version
  allocated_storage               = var.allocated_storage
  storage_type                    = var.storage_type
  auto_minor_version_upgrade      = var.auto_minor_version_upgrade
  instance_class                  = var.instance_class
  multi_az                        = var.multi_az
  parameter_group_name            = aws_db_parameter_group.this.id
  backup_retention_period         = var.backup_retention_period
  vpc_security_group_ids          = [data.aws_security_group.this.id]
  db_subnet_group_name            = var.db_subnet_group_name
  apply_immediately               = var.apply_immediately
  snapshot_identifier             = var.snapshot_identifier
  skip_final_snapshot             = var.skip_final_snapshot
  tags                            = var.defaults.default_tags
  deletion_protection             = var.deletion_protection
  enabled_cloudwatch_logs_exports = var.enabled_cloudwatch_logs_exports
  storage_encrypted               = var.storage_encrypted
  kms_key_id                      = local.kms_key_arn
  allow_major_version_upgrade     = var.allow_major_version_upgrade

  publicly_accessible = false
}


resource aws_db_parameter_group read_replica {
  name   = "${var.rds_read_replica}-rr-pg12"
  family = postgres11

  lifecycle {
    create_before_destroy = true
  }
}

resource aws_db_instance read_replica {
  identifier                      = "${var.rds_read_replica}-rr"
  replicate_source_db             = aws_db_instance.this.id
  storage_type                    = var.storage_type
  auto_minor_version_upgrade      = var.auto_minor_version_upgrade
  parameter_group_name            = aws_db_parameter_group.read_replica.id
  vpc_security_group_ids          = [var.aws_security_group.this.id]
  apply_immediately               = "true"
  skip_final_snapshot             = "true"
  tags                            = var..rds_tags
  storage_encrypted               = var.storage_encrypted
  kms_key_id                      = local.kms_key_arn
  enabled_cloudwatch_logs_exports = var.enabled_cloudwatch_logs_exports
  depends_on                      = [aws_db_instance.this]
}

Parent module
module qa_rds {
  source = "path_to_child_module"

  short_name                                    = "qa"
  db_name                                       = "abc"
  instance_class                                = var.instance_class
  allocated_storage                             = var.allocated_storage
  engine_version                                = var.engine_version
  db_parameter_group_family                     = var.db_parameter_group_family
  storage_type                                  = var.storage_type
  multi_az                                      = var.multi_az
  backup_retention_period                       = var.backup_retention_period
  auto_minor_version_upgrade                    = var.auto_minor_version_upgrade
  apply_immediately                             = var.apply_immediately
  skip_final_snapshot                           = var.skip_final_snapshot
  final_snapshot_identifier                     = var.final_snapshot_identifier
  deletion_protection                           = var.deletion_protection
  enabled_cloudwatch_logs_exports               = var.enabled_cloudwatch_logs_exports
  storage_encrypted                             = false
  allow_major_version_upgrade                   = "true"
}

Please include all Terraform configurations required to reproduce the bug. Bug reports without a functional reproduction may be closed without investigation.

# Copy-paste your Terraform configurations here - for large Terraform configs,
# please use a service like Dropbox and share a link to the ZIP file. For
# security, you can also encrypt the files using our GPG public key: https://keybase.io/hashicorp

Debug Output

Panic Output

Expected Behavior

Terraform should have upgraded PostgreSQL primary and read-replica from version 9.6.20 to version 12.5. To workaround, I upgraded the primary manually which upgraded both primary and read-replia RDS instanes.

Actual Behavior

Terraform gave below error:

Error: Error modifying DB Instance qa-abc-rep-rr: InvalidParameterCombination: The Parameter Group qa-abc-rr-pg12 with DBParameterGroupFamily postgres12 cannot be used for this instance. Please use a Parameter Group with DBParameterGroupFamily postgres9.6
	status code: 400, request id: 4590cb12-88a1-411a-a128-3b31bb0337fe

As per AWS document, primary and read-replica instanes are upgraded simultaneously but I am not sure if there is a way to execute two resources in parallel in terraform.

During a major version upgrade, Amazon RDS also upgrades all of your in-Region read replicas along with the primary DB instance.

I have put lifecycle in DB parameter group as per suggestion in this comment - #6448 (comment)

Steps to Reproduce

1. terraform apply

Important Factoids

I think this is a typical "Chicken-and-egg" problem in which terraform cannot decide which resource to apply first since AWS in the background is upgrading both RDS instances parallelly. I tried depends_on in read-replica but it gave same error.

References

I am creating this issue after suggestion from James Bardin in this issue - hashicorp/terraform#29334

• #0000

[ewbankkit]

@satish-chef Thanks for raising this issue.
Please see this comment for some potential solutions.

[github-actions]

Marking this issue as stale due to inactivity. This helps our maintainers find and focus on the active issues. If this issue receives no comments in the next 30 days it will automatically be closed. Maintainers can also remove the stale label.

If this issue was automatically closed and you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thank you!

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.