Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

resource/aws_dynamodb_table: Allow simultaneous region deletion retry of 5 minutes to better handle global table deletions #5518

Merged
merged 1 commit into from
Aug 13, 2018

Conversation

bflad
Copy link
Contributor

@bflad bflad commented Aug 12, 2018

When DynamoDB is deleting multiple regions of a global table, the deletion call can get stuck waiting for another deletion call to finish. This can be seen with the very consistently failing acceptance test:

--- FAIL: TestAccAWSDynamoDbGlobalTable_multipleRegions (284.43s)
	testing.go:579: Error destroying resource! WARNING: Dangling resources
		may exist. The full state and error is shown below.

		Error: Error applying: 2 errors occurred:
			* aws_dynamodb_table.us-east-1 (destroy): 1 error occurred:
			* aws_dynamodb_table.us-east-1: ResourceInUseException: Attempt to change a resource which is still in use: Table is being deleted: tf-acc-test-bavzb
			status code: 400, request id: 8UEJN22QSQF434KNQF9IJ63J1NVV4KQNSO5AEMVJF66Q9ASUAAJG

			* aws_dynamodb_table.us-west-2 (destroy): 1 error occurred:
			* aws_dynamodb_table.us-west-2: ResourceInUseException: Attempt to change a resource which is still in use: Table is being deleted: tf-acc-test-bavzb
			status code: 400, request id: T73SL4E6R3EUUE7E2012Q11DCJVV4KQNSO5AEMVJF66Q9ASUAAJG

Changes proposed in this pull request:

  • Increase the retry threshold in aws_dynamodb_table resource deletion function to allow DynamoDB to get around to deleting other regions simultaneously. Increasing the threshold to 2 minutes still was flakey with our empty DynamoDB global table, so decided on 5 minutes as the retry threshold since the retry conditions are fairly safe. In some regards this is a guess that it shouldn't instead be d.Timeout(schema.TimeoutDelete), which is already used with the wait for deletion StateChangeConf.

Output from acceptance testing:

$ make testacc TEST=./aws TESTARGS='-run=TestAccAWSDynamoDbGlobalTable_multipleRegions'
==> Checking that code complies with gofmt requirements...
TF_ACC=1 go test ./aws -v -run=TestAccAWSDynamoDbGlobalTable_multipleRegions -timeout 120m
=== RUN   TestAccAWSDynamoDbGlobalTable_multipleRegions
--- PASS: TestAccAWSDynamoDbGlobalTable_multipleRegions (301.57s)
PASS
ok  	github.com/terraform-providers/terraform-provider-aws/aws	302.214s

…dle global table deletions

When DynamoDB is deleting multiple regions of a global table, the deletion call can get stuck waiting for another deletion call to finish. This can be seen with the very consistently failing acceptance test:

--- FAIL: TestAccAWSDynamoDbGlobalTable_multipleRegions (284.43s)
	testing.go:579: Error destroying resource! WARNING: Dangling resources
		may exist. The full state and error is shown below.

		Error: Error applying: 2 errors occurred:
			* aws_dynamodb_table.us-east-1 (destroy): 1 error occurred:
			* aws_dynamodb_table.us-east-1: ResourceInUseException: Attempt to change a resource which is still in use: Table is being deleted: tf-acc-test-bavzb
			status code: 400, request id: 8UEJN22QSQF434KNQF9IJ63J1NVV4KQNSO5AEMVJF66Q9ASUAAJG

			* aws_dynamodb_table.us-west-2 (destroy): 1 error occurred:
			* aws_dynamodb_table.us-west-2: ResourceInUseException: Attempt to change a resource which is still in use: Table is being deleted: tf-acc-test-bavzb
			status code: 400, request id: T73SL4E6R3EUUE7E2012Q11DCJVV4KQNSO5AEMVJF66Q9ASUAAJG

Here we increase the retry threshold to allow DynamoDB to get around to deleting all other regions simultaneously.

make testacc TEST=./aws TESTARGS='-run=TestAccAWSDynamoDbGlobalTable_multipleRegions'
==> Checking that code complies with gofmt requirements...
TF_ACC=1 go test ./aws -v -run=TestAccAWSDynamoDbGlobalTable_multipleRegions -timeout 120m
=== RUN   TestAccAWSDynamoDbGlobalTable_multipleRegions
--- PASS: TestAccAWSDynamoDbGlobalTable_multipleRegions (301.57s)
PASS
ok  	github.com/terraform-providers/terraform-provider-aws/aws	302.214s
@bflad bflad added bug Addresses a defect in current functionality. service/dynamodb Issues and PRs that pertain to the dynamodb service. labels Aug 12, 2018
@bflad bflad requested a review from a team August 12, 2018 00:34
@ghost ghost added the size/XS Managed by automation to categorize the size of a PR. label Aug 12, 2018
Copy link
Contributor

@tombuildsstuff tombuildsstuff left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM 👍

@bflad bflad added this to the v1.32.0 milestone Aug 13, 2018
@bflad bflad merged commit 36e53b5 into master Aug 13, 2018
@bflad bflad deleted the b-aws_dynamodb_table-multiple-deletion-retry branch August 13, 2018 15:22
bflad added a commit that referenced this pull request Aug 13, 2018
@bflad
Copy link
Contributor Author

bflad commented Aug 16, 2018

This has been released in version 1.32.0 of the AWS provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

@ghost
Copy link

ghost commented Apr 4, 2020

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks!

@ghost ghost locked and limited conversation to collaborators Apr 4, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
bug Addresses a defect in current functionality. service/dynamodb Issues and PRs that pertain to the dynamodb service. size/XS Managed by automation to categorize the size of a PR.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants