Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

provider/aws: Catch 400 error from rds_cluster #11795

Merged
merged 1 commit into from
Feb 8, 2017

Conversation

grubernaut
Copy link
Contributor

Previously, an aws_rds_cluster that contains active instance groups would timeout on a destroy, if the destroy was able to only target the rds_cluster and not include the instance groups.
This would result in a 400 response from AWS, and Terraform would sit in a wait-loop until a 15-minute timeout while waiting for the cluster to be destroyed.

This catches the error returned from the DeleteDBCluster function call such that the proper error case can be returned to the user.

400 from the AWS API:

2017/02/08 13:40:47 [DEBUG] plugin: terraform: ---[ RESPONSE ]--------------------------------------
2017/02/08 13:40:47 [DEBUG] plugin: terraform: HTTP/1.1 400 Bad Request
2017/02/08 13:40:47 [DEBUG] plugin: terraform: Connection: close
2017/02/08 13:40:47 [DEBUG] plugin: terraform: Content-Length: 337
2017/02/08 13:40:47 [DEBUG] plugin: terraform: Content-Type: text/xml
2017/02/08 13:40:47 [DEBUG] plugin: terraform: Date: Wed, 08 Feb 2017 18:40:46 GMT
2017/02/08 13:40:47 [DEBUG] plugin: terraform: X-Amzn-Requestid: 1b4a76cc-ee2e-11e6-867d-2311ebaffd3e
2017/02/08 13:40:47 [DEBUG] plugin: terraform:
2017/02/08 13:40:47 [DEBUG] plugin: terraform: <ErrorResponse xmlns="http://rds.amazonaws.com/doc/2014-10-31/">
2017/02/08 13:40:47 [DEBUG] plugin: terraform:   <Error>
2017/02/08 13:40:47 [DEBUG] plugin: terraform:     <Type>Sender</Type>
2017/02/08 13:40:47 [DEBUG] plugin: terraform:     <Code>InvalidDBClusterStateFault</Code>
2017/02/08 13:40:47 [DEBUG] plugin: terraform:     <Message>Cluster cannot be deleted, it still contains DB instances in non-deleting state.</Message>
2017/02/08 13:40:47 [DEBUG] plugin: terraform:   </Error>
2017/02/08 13:40:47 [DEBUG] plugin: terraform:   <RequestId>1b4a76cc-ee2e-11e6-867d-2311ebaffd3e</RequestId>
2017/02/08 13:40:47 [DEBUG] plugin: terraform: </ErrorResponse>
2017/02/08 13:40:47 [DEBUG] plugin: terraform:
2017/02/08 13:40:47 [DEBUG] plugin: terraform: -----------------------------------------------------

Error returns now, as expected:

Error applying plan:

2017/02/08 13:40:47 [DEBUG] plugin: waiting for all plugin processes to complete...
1 error(s) occurred:

* aws_rds_cluster.jake (destroy): 1 error(s) occurred:

2017/02/08 13:40:47 [DEBUG] plugin: terraform: aws-provider (internal) 2017/02/08 13:40:47 [DEBUG] plugin: waiting for all plugin processes to complete...
* aws_rds_cluster.jake: RDS Cluster cannot be deleted: Cluster cannot be deleted, it still contains DB instances in non-deleting state.

Fixes: #11378

Previously, an `aws_rds_cluster` that contains active instance groups would timeout on a destroy, if the destroy was able to only target the rds_cluster and not include the instance groups.
This would result in a `400` response from AWS, and Terraform would sit in a wait-loop until a 15-minute timeout while waiting for the cluster to be destroyed.

This catches the error returned from the `DeleteDBCluster` function call such that the proper error case can be returned to the user.

`400` from the AWS API:
```
2017/02/08 13:40:47 [DEBUG] plugin: terraform: ---[ RESPONSE ]--------------------------------------
2017/02/08 13:40:47 [DEBUG] plugin: terraform: HTTP/1.1 400 Bad Request
2017/02/08 13:40:47 [DEBUG] plugin: terraform: Connection: close
2017/02/08 13:40:47 [DEBUG] plugin: terraform: Content-Length: 337
2017/02/08 13:40:47 [DEBUG] plugin: terraform: Content-Type: text/xml
2017/02/08 13:40:47 [DEBUG] plugin: terraform: Date: Wed, 08 Feb 2017 18:40:46 GMT
2017/02/08 13:40:47 [DEBUG] plugin: terraform: X-Amzn-Requestid: 1b4a76cc-ee2e-11e6-867d-2311ebaffd3e
2017/02/08 13:40:47 [DEBUG] plugin: terraform:
2017/02/08 13:40:47 [DEBUG] plugin: terraform: <ErrorResponse xmlns="http://rds.amazonaws.com/doc/2014-10-31/">
2017/02/08 13:40:47 [DEBUG] plugin: terraform:   <Error>
2017/02/08 13:40:47 [DEBUG] plugin: terraform:     <Type>Sender</Type>
2017/02/08 13:40:47 [DEBUG] plugin: terraform:     <Code>InvalidDBClusterStateFault</Code>
2017/02/08 13:40:47 [DEBUG] plugin: terraform:     <Message>Cluster cannot be deleted, it still contains DB instances in non-deleting state.</Message>
2017/02/08 13:40:47 [DEBUG] plugin: terraform:   </Error>
2017/02/08 13:40:47 [DEBUG] plugin: terraform:   <RequestId>1b4a76cc-ee2e-11e6-867d-2311ebaffd3e</RequestId>
2017/02/08 13:40:47 [DEBUG] plugin: terraform: </ErrorResponse>
2017/02/08 13:40:47 [DEBUG] plugin: terraform:
2017/02/08 13:40:47 [DEBUG] plugin: terraform: -----------------------------------------------------
```

Error returns now,  as expected:
```
Error applying plan:

2017/02/08 13:40:47 [DEBUG] plugin: waiting for all plugin processes to complete...
1 error(s) occurred:

* aws_rds_cluster.jake (destroy): 1 error(s) occurred:

2017/02/08 13:40:47 [DEBUG] plugin: terraform: aws-provider (internal) 2017/02/08 13:40:47 [DEBUG] plugin: waiting for all plugin processes to complete...
* aws_rds_cluster.jake: RDS Cluster cannot be deleted: Cluster cannot be deleted, it still contains DB instances in non-deleting state.
```
Copy link
Contributor

@stack72 stack72 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM - 1 question

@@ -611,6 +611,13 @@ func resourceAwsRDSClusterDelete(d *schema.ResourceData, meta interface{}) error

log.Printf("[DEBUG] RDS Cluster delete options: %s", deleteOpts)
_, err := conn.DeleteDBCluster(&deleteOpts)
if err != nil {
if awsErr, ok := err.(awserr.Error); ok {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am presuming we can't put a cluster into a state to test this?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Test case is as follows:

# main.tf
variable "mysql_username" { default = "dummy_user" }
variable "mysql_password" { default = "password12345" }

provider "aws" {
  region = "us-west-2"
}


resource "aws_rds_cluster" "cluster" {
  cluster_identifier     = "cluster-test-cluster"
  availability_zones     = ["us-west-2a","us-west-2b"]
  database_name          = "test-db"
  master_username        = "${var.mysql_username}"
  master_password        = "${var.mysql_password}"
  apply_immediately      = true
}

resource "aws_rds_cluster_instance" "group_0" {
  identifier           = "group-0-testing"
  cluster_identifier   = "${aws_rds_cluster.cluster.id}"
  instance_class       = "db.t2.medium"
  publicly_accessible  = true

  tags {
    Name = "group-0-testing"
  }
}

resource "aws_rds_cluster_instance" "group_1" {
  identifier           = "group-1-testing"
  cluster_identifier   = "${aws_rds_cluster.cluster.id}"
  instance_class       = "db.t2.medium"
  publicly_accessible  = true

  tags {
    Name = "group-1-testing"
  }
}

Followed by:

terraform apply
mkdir import; cd import
terraform import aws_rds_cluster.cluster cluster-test-cluster
terraform destroy
# observe error from dangling rds_cluster without instance groups being destroyed

@grubernaut grubernaut merged commit 327b9e2 into master Feb 8, 2017
@grubernaut grubernaut deleted the b-respect-400-rds-deletion branch February 8, 2017 19:26
@ghost
Copy link

ghost commented Apr 17, 2020

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

@ghost ghost locked and limited conversation to collaborators Apr 17, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

400 response when destroying aws_rds_cluster not respected
2 participants