S3 Replication Configuration returns 400 error but thinks it has applied anyway

[ChadScott]

Terraform Version

$ terraform -v
Terraform v0.8.2

Affected Resource(s)

• aws_s3_bucket

Terraform Configuration Files

resource "aws_iam_role" "usw2_state_bucket_replication_role" {
  name = "terraform-usw2_state_bucket_replication_role"
  assume_role_policy = <<POLICY
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Action": "sts:AssumeRole",
      "Principal": {
        "Service": "s3.amazonaws.com"
      },
      "Effect": "Allow",
      "Sid": ""
    }
  ]
}
POLICY
}

resource "aws_iam_policy" "usw2_state_bucket_replication_policy" {
  policy = <<POLICY
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Action": [
        "s3:GetReplicationConfiguration",
        "s3:ListBucket"
      ],
      "Effect": "Allow",
      "Resource": [
        "${aws_s3_bucket.terraform-usw2_state_bucket.arn}"
      ]
    },
    {
      "Action": [
        "s3:GetObjectVersion",
        "s3:GetObjectVersionAcl"
      ],
      "Effect": "Allow",
      "Resource": [
        "${aws_s3_bucket.terraform-usw2_state_bucket.arn}/*"
      ]
    },
    {
      "Action": [
        "s3:ReplicateObject",
        "s3:ReplicateDelete"
      ],
      "Effect": "Allow",
      "Resource": "${data.terraform_remote_state.us_east_1.state_bucket_arn}/*"
    }
  ]
}
POLICY
}

resource "aws_iam_policy_attachment" "usw2_state_bucket_replication" {
  name = "usw2_state_bucket_replication_role_policy"
  roles = ["${aws_iam_role.usw2_state_bucket_replication_role.name}"]
  policy_arn = "${aws_iam_policy.usw2_state_bucket_replication_policy.arn}"
}

resource "aws_s3_bucket" "terraform-usw2_state_bucket" {
  bucket = "the_bucket"
  acl = "private"

  versioning {
    enabled = true
  }

  lifecycle_rule {
    id = "whole_bucket"
    prefix = "/"
    enabled = true

    noncurrent_version_transition {
      days = 90
      storage_class = "STANDARD_IA"
    }
    noncurrent_version_transition {
      days = 180
      storage_class = "GLACIER"
    }
    noncurrent_version_expiration {
      days = 366
    }
  }

  replication_configuration {
    role = "${aws_iam_role.usw2_state_bucket_replication_role.arn}"
    rules {
      id = "replication_configuration"
      prefix = ""
      status = "Enabled"

      destination {
        bucket = "${data.terraform_remote_state.us_east_1.state_bucket_arn}"
      }
    }
  }

  lifecycle {
    prevent_destroy = true
  }
}

Debug Output

There's proprietary info in the debug output. I can provide this separately.

Expected Behavior

Replication should have been enabled on the bucket with the role given without error.

Actual Behavior

A 400 error is returned from Amazon. In addition, the error seems to be ignored as the state file believes replication is enabled when it is not.

[Ninir]

Hi @ChadScott

Could you please run TF_LOG=DEBUG terraform apply and put the output here please?
Would be better to have the 400 error message returned to debug :)

Also, I tried again the documentation from the website, and all works fine.

Ninir

[ChadScott]

As I mentioned in the report, there's a lot of proprietary info in there. I can provide the output via keybase.io or similar, if that works.

[Ninir]

Then, just output the errors you got, with the body (i.e. the related XML), should be fine!

[ChadScott]

2016/12/22 12:39:06 [DEBUG] plugin: terraform: ---[ REQUEST POST-SIGN ]-----------------------------
2016/12/22 12:39:06 [DEBUG] plugin: terraform: PUT /<redacted>?replication= HTTP/1.1
2016/12/22 12:39:06 [DEBUG] plugin: terraform: Host: s3-us-west-2.amazonaws.com
2016/12/22 12:39:06 [DEBUG] plugin: terraform: User-Agent: APN/1.0 HashiCorp/1.0 Terraform/0.8.2
2016/12/22 12:39:06 [DEBUG] plugin: terraform: Content-Length: 349
2016/12/22 12:39:06 [DEBUG] plugin: terraform: Authorization: AWS4-HMAC-SHA256 Credential=<redacted>/20161222/us-west-2/s3/aws4_request, SignedHeaders=content-length;content-md5;host;x-amz-content-sha256;x-amz-date, Signature=b8dc0957381498897c0a11a9d15b356667d6ebde45181b26e510cb2c3beba4b8
2016/12/22 12:39:06 [DEBUG] plugin: terraform: Content-Md5: gX52xQlKJV3ACNxzhP8O3w==
2016/12/22 12:39:06 [DEBUG] plugin: terraform: X-Amz-Content-Sha256: 47dbf34a5761ef2b73949503a9de085f86e2e29f88a4175a1a658c9b78f1de2e
2016/12/22 12:39:06 [DEBUG] plugin: terraform: X-Amz-Date: 20161222T203906Z
2016/12/22 12:39:06 [DEBUG] plugin: terraform: Accept-Encoding: gzip
2016/12/22 12:39:06 [DEBUG] plugin: terraform: 
2016/12/22 12:39:06 [DEBUG] plugin: terraform: <ReplicationConfiguration><Role>arn:aws:iam::<redacted>:role/terraform-usw2_state_bucket_replication_role</Role><Rule><Destination><Bucket>arn:aws:s3:::<redacted></Bucket><StorageClass></StorageClass></Destination><ID>replication_configuration</ID><Prefix></Prefix><Status>Enabled</Status></Rule></ReplicationConfiguration>
2016/12/22 12:39:06 [DEBUG] plugin: terraform: -----------------------------------------------------
2016/12/22 12:39:06 [DEBUG] plugin: terraform: aws-provider (internal) 2016/12/22 12:39:06 [DEBUG] [aws-sdk-go] DEBUG: Response s3/PutBucketReplication Details:
2016/12/22 12:39:06 [DEBUG] plugin: terraform: ---[ RESPONSE ]--------------------------------------
2016/12/22 12:39:06 [DEBUG] plugin: terraform: HTTP/1.1 400 Bad Request
2016/12/22 12:39:06 [DEBUG] plugin: terraform: Connection: close
2016/12/22 12:39:06 [DEBUG] plugin: terraform: Transfer-Encoding: chunked
2016/12/22 12:39:06 [DEBUG] plugin: terraform: Content-Type: application/xml
2016/12/22 12:39:06 [DEBUG] plugin: terraform: Date: Thu, 22 Dec 2016 20:39:06 GMT
2016/12/22 12:39:06 [DEBUG] plugin: terraform: Server: AmazonS3
2016/12/22 12:39:06 [DEBUG] plugin: terraform: X-Amz-Id-2: ZKRJebrIuoXjq7zRm8S9yka9cmHZjc7WYRk8taM2eK9mDgaejPMc9+QAV/56LBxsF/IxiBlFv04=
2016/12/22 12:39:06 [DEBUG] plugin: terraform: X-Amz-Request-Id: C58BE3B51D322637
2016/12/22 12:39:06 [DEBUG] plugin: terraform: 
2016/12/22 12:39:06 [DEBUG] plugin: terraform: 13f
2016/12/22 12:39:06 [DEBUG] plugin: terraform: <?xml version="1.0" encoding="UTF-8"?>
2016/12/22 12:39:06 [DEBUG] plugin: terraform: <Error><Code>MalformedXML</Code><Message>The XML you provided was not well-formed or did not validate against our published schema</Message><RequestId>C58BE3B51D322637</RequestId><HostId>ZKRJebrIuoXjq7zRm8S9yka9cmHZjc7WYRk8taM2eK9mDgaejPMc9+QAV/56LBxsF/IxiBlFv04=</HostId></Error>
2016/12/22 12:39:06 [DEBUG] plugin: terraform: 0
2016/12/22 12:39:06 [DEBUG] plugin: terraform: 
2016/12/22 12:39:06 [DEBUG] plugin: terraform: 
2016/12/22 12:39:06 [DEBUG] plugin: terraform: -----------------------------------------------------

[ChadScott]

I think I've diagnosed the problem.

The documentation states:

storage_class - (Optional) The class of storage used to store the object.

This appears to be compulsory, however. Leaving it out produces the 400 error. Further, despite Terraform completing successfully, replication is not enabled on the bucket. Both the GUI and API show it as not enabled.

[ChadScott]

Okay, I finally got it to work properly. I defined storage_class, ran Terraform, commented out the entire replication_configuration section, ran Terraform, then uncommented the same section and ran Terraform yet again.

This seems to have properly enabled it on the bucket with the configuration I've specified.

[Ninir]

Hey @ChadScott

With your inputs, I was able to replicate the issue. As you said, the issue is with the StorageClass, which is added to the request even if the value is empty (the <StorageClass></StorageClass> part in your request).

I will investigate this and come back with a fix ASAP!

Sorry for the issue,
Ninir

[ChadScott]

I think the more troubling part is that Terraform stored the completed state in the state file and thought it was enabled when it wasn't.

[Ninir]

Hey @ChadScott

Just made #10921 to fix it. Sorry for the noise here!

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.