RDS Aurora Cross-Region replication for encrypted cluster failing

[hashibot]

This issue was originally opened by @asadhu as hashicorp/terraform#12985. It was migrated here as part of the provider split. The original body of the issue is below.

---

Hi there,

Thank you for opening an issue. Please note that we try to keep the Terraform issue tracker reserved for bug reports and feature requests. For general usage questions, please see: https://www.terraform.io/community.html.

Terraform Version

v0.9.1_1

Affected Resource(s)

RDS Cross-Region Cluster replication

Terraform Configuration Files

resource "aws_rds_cluster" "replica-west" {
  cluster_identifier      = "aurora-replica-cluster-demo"
  replication_source_identifier = "arn:aws:rds:us-east-1:*:cluster:aurora-cluster"
  #"${aws_rds_cluster.default-east.id}"
  availability_zones      = ["us-west-2a"]
  backup_retention_period = 5
  preferred_backup_window = "07:00-09:00"
  storage_encrypted       = "true"
  kms_key_id              = "arn:aws:kms:us-east-1:*:key/[key_name]"
  db_cluster_parameter_group_name = "default.aurora5.6"
  #source-region           = "us-east-1"
}

Debug Output

• aws_rds_cluster.replica-west: InvalidParameterCombination: Source cluster arn:aws:rds:us-east-1:*:cluster:aurora-cluster is encrypted; pre-signed URL has to be specified
  status code: 400

Panic Output

Expected Behavior

What should have happened?

I have an AWS RDS Aurora Cluster in US East and am trying to create a Cross-Region Cluster on the west. The Replica Cluster should have been created.

Actual Behavior

What actually happened?

The East Cluster is encrypted and requires kms_key_id and source_region/presigned_url. 'source-region/presigned_url' is not supported in Terraform 0.9.1_1.
https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Aurora.Replication.CrossRegion.html

Steps to Reproduce

Please list the steps required to reproduce the issue, for example:

1. terraform apply

Important Factoids

References

[asadhu]

@twofivetechnology @grubernaut Any work around that can help to remediate the problem??

[asadhu]

Is the SourceRegion being specified

aws_rds_cluster.replica: InvalidParameterCombination: Source cluster arn:aws:rds:us-east-1:<id>:cluster:<cluster_name> is encrypted; pre-signed URL has to be specified status code: 400, request id: b59ab0b3-812b-11e7-8467-c192637e53bf

[timboven]

Looking quickly through the code, I believe it is the next change that needs to happen:

--- a/vendor/github.com/terraform-providers/terraform-provider-aws/aws/resource_aws_rds_cluster.go
+++ b/vendor/github.com/terraform-providers/terraform-provider-aws/aws/resource_aws_rds_cluster.go
@@ -222,6 +222,11 @@ func resourceAwsRDSCluster() *schema.Resource {
                                Type:     schema.TypeString,
                                Optional: true,
                        },
+                       
+                       "source_region": {
+                               Type:     schema.TypeString,
+                               Optional: true,
+                       },
 
                        "iam_roles": {
                                Type:     schema.TypeSet,
@@ -356,6 +361,7 @@ func resourceAwsRDSClusterCreate(d *schema.ResourceData, meta interface{}) error
                        Engine:                      aws.String(d.Get("engine").(string)),
                        StorageEncrypted:            aws.Bool(d.Get("storage_encrypted").(bool)),
                        ReplicationSourceIdentifier: aws.String(d.Get("replication_source_identifier").(string)),
+                       SourceRegion:                aws.String(d.Get("source_region").(string)),
                        Tags: tags,
                }

I also thought to add it in the describe-output in the function resourceAwsRDSClusterRead - but looks like this param isn't returned by the SDK.

I'm however very new to go - and the first time that I look into terraform-code and I'm also not familiar yet with the aws api - so would be great if somebody could check this to see if I am on the right track here.
I'm also a little bit puzzled on where this change now actually needs to happen - in this repo or in the terraform-repo? (as you see I made it in my clone of terraform as it looks like then the compiler picked it)

[timboven]

I tried my proposed change - and while it looks like he takes the change, I still get an error (but a different one this time): PreSignedUrl could not be authenticated (what seems to be related to aws/aws-sdk-go#1098)

[asadhu]

@timboven Yes it initially seemed like the source-region is the issue but I too have been getting the same issue trying to run it with AWS CLI. If we can get a working AWS CLI request I suppose we would know the exact parameters required. The documentation doesn't seem to be consistent.

[twofivetechnology]

@asadhu I have been using CLI to work around. I will redact and post the working cli bash script tonight.

[twofivetechnology]

#!bin/bash


echo "Checking to see if cluster exists"
cluster=`aws rds describe-db-clusters --region ${backup_region} --db-cluster-identifier ${db_identifier}-${app_env}-cluster || echo 'UNDEFINED'`

if [ "${cluster}" == "UNDEFINED" ]; then
  echo "Creating replica cluster"
  # create the replica cluster if it does not already exist
  aws rds create-db-cluster \
	  --region ${backup_region} \
	  --db-cluster-identifier ${db_identifier}-${app_env}-cluster \
	  --replication-source-identifier arn:aws:rds:${region}:${account_id}:cluster:${db_identifier}-${app_env}-cluster \
	  --kms-key-id ${BACKUP_KMS_KEY_ID} \
	  --storage-encrypted \
	  --source-region ${region} \
    --availability-zones ${backup_aurora_avl_zones} \
	  --vpc-security-group-ids ${backup_db_security_group} \
	  --db-subnet-group-name ${backup_db_subnet_name} \
    --db-cluster-parameter-group-name ${db_identifier}-${app_env}-cluster-pg \
	  --tags <your tags here> \
    --engine aurora \
    --port ${db_port}
fi

echo "Waiting for replica cluster to become available"
cluster_status=`aws rds describe-db-clusters --region ${backup_region} --db-cluster-identifier ${db_identifier}-${app_env}-cluster --query 'DBClusters[*].Status' | grep \" | sed 's/.*"\(.*\)".*/\1/g'`
count=0
while [ "${cluster_status}" != "available" ]
do
	echo "Cluster Status: ${cluster_status}"
	echo "sleeping for 10 minutes..."
	sleep 600
	cluster_status=`aws rds describe-db-clusters --region ${backup_region} --db-cluster-identifier ${db_identifier}-${app_env}-cluster --query 'DBClusters[*].Status' | grep \" | sed 's/.*"\(.*\)".*/\1/g'`
	# wait at most 4 hours for cluster to be available
	((count++)) && ((count>=24)) && break
done

if [ "${cluster_status}" != "available" ]; then
  echo "Replica cluster never became available"
  exit 1
fi

# capture the cluster in tfstate to make it easier once it is supported
terraform import $terraform_vars -state=${TERRAFORM_STATE_FILE} aws_rds_cluster.backup ${db_identifier}-${app_env}-cluster

[dev-usa]

@twofivetechnology - Hi ... thanks for the bash script. New to Terraform here. Any inputs on how to use it in conjunction with TF ? and if the backend is S3, can I just pass the state file name to the TERRAFORM_STATE_FILE variable?

[twofivetechnology]

There is a nice way actually to sync state direct to S3 backend that I haven't updated to yet (all my pipelines have a sync from S3 to begin with then a sync back to S3 in a finally block from before it was a first class citizen).

As for the script above the terraform import at the end then gets the provisioned replica into your state file (so for me that gets syncd to S3 in the finally block of my groovy pipeline). The reason to consider doing that is that when the issue here is fixed (note issue might actually be in Go SDK and not terraform) you can seemlessly switch to using terraform for your replica management because it will already be tracked in your state file.

[jude-pieries]

I can confirm that sourceregion needs to be added to "aws_rds_cluster" to get around the encrypted error " cluster:aurora-cluster is encrypted; pre-signed URL has to be specified status code: 400" for encrypted cross region replicas. Hope this gets added soon

I managed recreate above error with the New-rdsdbcluster in powershell , adding the -sourceregion resolved the issue.

New-RDSDBCluster -SourceRegion $var_source_Region -Region $var_des_Region -DBClusterIdentifier $var_des_DBClusterIdentifier -ReplicationSourceIdentifier $var_source_ReplicationSourceIdentifier -KmsKeyId $var_des_KmsKeyId -StorageEncrypted $var_des_StorageEncrypted -DBSubnetGroupName $var_des_DBSubnetGroupName -DBClusterParameterGroupName $var_des_DBClusterParameterGroupName -Engine $var_des_Engine -Port $var_des_Port

[mitchelldavis44]

Have there been any updates on this? I'm having the same issue of not being able to create a cross region read replica for an encrypted cluster.

[viggy28]

Thanks @twofivetechnology for the work around. In the end you added the resource to the TF state . Is that sufficient for terraform to handle any future changes on the replica cluster (for eg. change the instance size or master password) can terraform handle that?

Team is there any timeline for this issue to get fixed?

[bflad]

This has been released in version 1.10.0 of the AWS provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks!