[Bug]: Error emptying S3 Bucket when objects are in it, even though force-destroy is enabled.

[porrige51122]

Terraform Core Version

1.5.1

AWS Provider Version

5.6.0, 5,6,1

Affected Resource(s)

• aws_s3_bucket.bucket

Expected Behavior

Terraform init an S3 bucket module which contains all the things to make an S3 bucket secure and with force_destroy = true

Terraform apply to an account.

Add objects to bucket.

Terraform destroy, empties the bucket and then destroys it and all corresponding resources.

Actual Behavior

When running terraform destroy, it throws an error and doesn't delete the bucket or objects inside of it.

This is different to the behaviour in the version prior.

Relevant Error/Panic Output Snippet

╷
│ Error: emptying S3 Bucket (test-bucket-...): listing S3 Bucket (test-bucket-...) object versions: InvalidArgument: Invalid attribute name specified.
│       status code: 400, request id: JD08VXT7P4Y19J58, host id: .../...
│ 
│ 

Terraform Configuration Files

https://github.com/porrige51122/terraform-aws-secure-s3-bucket

Steps to Reproduce

terraform init

terraform apply

add an object to the bucket

terraform destroy

Debug Output

https://gist.github.com/porrige51122/b63e062cf5f670e84412c93d91830445

Panic Output

No response

Important Factoids

This is on an arm chip, and also failed on aws codebuild on an arm chip.

This also worked on versions <5.6.0

References

No response

Would you like to implement a fix?

No

[github-actions]

Community Note

Voting for Prioritization

• Please vote on this issue by adding a 👍 reaction to the original post to help the community and maintainers prioritize this request.
• Please see our prioritization guide for information on how we prioritize.
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.

Volunteering to Work on This Issue

• If you are interested in working on this issue, please leave a comment.
• If this would be your first contribution, please review the contribution guide.

[porrige51122]

I am going to see how minimal I can make the example. Hopefully that will narrow it down a bit more

[porrige51122]

Okay, I've removed all the extra infrastructure, and just deployed the s3 bucket resource alone and the error still occurs.

[ewbankkit]

Can reproduce with one of our unit tests (v5.6.1):

% AWS_REGION=us-west-2 go test -v ./internal/service/s3 -run=TestEmptyBucket -b ewbankkit-test-empty-bucket-001 -f
=== RUN   TestEmptyBucket
=== PAUSE TestEmptyBucket
=== CONT  TestEmptyBucket
    delete_test.go:33: error emptying S3 bucket (ewbankkit-test-empty-bucket-001): listing S3 Bucket (ewbankkit-test-empty-bucket-001) object versions: InvalidArgument: Invalid attribute name specified.
        	status code: 400, request id: YWJBA8NQBVRPCPMN, host id: fC3v2YQGsnRTjPsUh+g1WolqlDiyDBFrYSoASfAfaX7/tB+xVeLLlPye4YkqX8RdkpTdqwtNlfQWICS894j2yw==
--- FAIL: TestEmptyBucket (0.52s)
FAIL
FAIL	github.com/hashicorp/terraform-provider-aws/internal/service/s3	6.342s
FAIL

The same test passes with v5.5.0:

% AWS_REGION=us-west-2 go test -v ./internal/service/s3 -run=TestEmptyBucket -b ewbankkit-test-empty-bucket-002 -f
=== RUN   TestEmptyBucket
=== PAUSE TestEmptyBucket
=== CONT  TestEmptyBucket
    delete_test.go:36: 0 S3 objects deleted
--- PASS: TestEmptyBucket (0.51s)
PASS
ok  	github.com/hashicorp/terraform-provider-aws/internal/service/s3	5.788s

Suspicious:

• Amazon S3 provides restore status of S3 Glacier objects using the S3 LIST API
• AWS SDK for Go v1.44.292: The S3 LISTObjects, ListObjectsV2 and ListObjectVersions API now supports a new optional header x-amz-optional-object-attributes.

Downgrade to AWS SDK for Go v1.44.291 and the test passes:

% go get github.com/aws/aws-sdk-go@v1.44.291 && go mod tidy
go: downgraded github.com/aws/aws-sdk-go v1.44.292 => v1.44.291
% AWS_REGION=us-west-2 go test -v ./internal/service/s3 -run=TestEmptyBucket -b ewbankkit-test-empty-bucket-002 -f
=== RUN   TestEmptyBucket
=== PAUSE TestEmptyBucket
=== CONT  TestEmptyBucket
    delete_test.go:36: 0 S3 objects deleted
--- PASS: TestEmptyBucket (0.51s)
PASS
ok  	github.com/hashicorp/terraform-provider-aws/internal/service/s3	6.175s

Relates aws/aws-sdk-go#4897.
Relates aws/aws-sdk-go#4898.

Fixed in AWS SDK for Go v1.44.293: #32305.

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.