New s3 subresources in 3.75.1 are unstable

[benjamin-wright]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform CLI and Terraform AWS Provider Version

➜ terraform -v      
Terraform v1.1.8
on darwin_amd64
+ provider registry.terraform.io/hashicorp/aws v3.75.1

Affected Resource(s)

• aws_s3_bucket
• aws_s3_bucket_server_side_encryption_configuration
• aws_s3_bucket_lifecycle_configuration

Terraform Configuration Files

Please include all Terraform configurations required to reproduce the bug. Bug reports without a functional reproduction may be closed without investigation.

terraform {
    required_version = "1.1.8"

    required_providers {
        aws = "3.75.1"
    }
}

provider "aws" {
    region = "eu-central-1"
}

resource "aws_s3_bucket" "global_test_ouput" {
  bucket        = "ben-test-config-bucket"
  force_destroy = true
}

resource "aws_s3_bucket_acl" "global_test_ouput" {
  bucket = aws_s3_bucket.global_test_ouput.bucket
  acl    = "private"
}

resource "aws_s3_bucket_server_side_encryption_configuration" "global_test_ouput" {
  bucket = aws_s3_bucket.global_test_ouput.bucket

  rule {
    apply_server_side_encryption_by_default {
      sse_algorithm = "AES256"
    }
  }
}


resource "aws_s3_bucket_lifecycle_configuration" "global_test_ouput" {
  depends_on = [aws_s3_bucket.global_test_ouput]
  bucket     = aws_s3_bucket.global_test_ouput.bucket

  rule {
    id = "config"

    transition {
      days          = 30
      storage_class = "STANDARD_IA"
    }

    transition {
      days          = 90
      storage_class = "GLACIER"
    }

    expiration {
      days = 180
    }

    status = "Enabled"
  }
}

resource "aws_s3_bucket_versioning" "global_test_ouput" {
  depends_on = [aws_s3_bucket.global_test_ouput]
  bucket     = aws_s3_bucket.global_test_ouput.bucket
  versioning_configuration {
    status = "Enabled"
  }
}

Expected Behavior

• Initial apply should create the bucket and associated sub-resources
• subsequent plan / apply should make no changes

Actual Behavior

• Initial apply should create the bucket and associated sub-resources
• second apply:

Note: Objects have changed outside of Terraform

Terraform detected the following changes made outside of Terraform since the last "terraform apply":

  # aws_s3_bucket.global_test_ouput has changed
  ~ resource "aws_s3_bucket" "global_test_ouput" {
        id                          = "ben-test-config-bucket"
        tags                        = {}
        # (11 unchanged attributes hidden)

      + lifecycle_rule {
          + abort_incomplete_multipart_upload_days = 0
          + enabled                                = true
          + id                                     = "config"
          + tags                                   = {}

          + expiration {
              + days                         = 180
              + expired_object_delete_marker = false
            }

          + transition {
              + days          = 30
              + storage_class = "STANDARD_IA"
            }
          + transition {
              + days          = 90
              + storage_class = "GLACIER"
            }
        }

      + server_side_encryption_configuration {
          + rule {
              + bucket_key_enabled = false

              + apply_server_side_encryption_by_default {
                  + sse_algorithm = "AES256"
                }
            }
        }

        # (1 unchanged block hidden)
    }

  # aws_s3_bucket_server_side_encryption_configuration.global_test_ouput has changed
  ~ resource "aws_s3_bucket_server_side_encryption_configuration" "global_test_ouput" {
        id     = "ben-test-config-bucket"
        # (1 unchanged attribute hidden)

      + rule {
          + bucket_key_enabled = false

          + apply_server_side_encryption_by_default {
              + sse_algorithm = "AES256"
            }
        }
      - rule {
          - apply_server_side_encryption_by_default {
              - sse_algorithm = "AES256" -> null
            }
        }
    }


Unless you have made equivalent changes to your configuration, or ignored the relevant attributes using ignore_changes, the following plan may include actions to undo or respond to
these changes.

────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  ~ update in-place

Terraform will perform the following actions:

  # aws_s3_bucket.global_test_ouput will be updated in-place
  ~ resource "aws_s3_bucket" "global_test_ouput" {
        id                          = "ben-test-config-bucket"
        tags                        = {}
        # (11 unchanged attributes hidden)

      - lifecycle_rule {
          - abort_incomplete_multipart_upload_days = 0 -> null
          - enabled                                = true -> null
          - id                                     = "config" -> null
          - tags                                   = {} -> null

          - expiration {
              - days                         = 180 -> null
              - expired_object_delete_marker = false -> null
            }

          - transition {
              - days          = 30 -> null
              - storage_class = "STANDARD_IA" -> null
            }
          - transition {
              - days          = 90 -> null
              - storage_class = "GLACIER" -> null
            }
        }

      - server_side_encryption_configuration {
          - rule {
              - bucket_key_enabled = false -> null

              - apply_server_side_encryption_by_default {
                  - sse_algorithm = "AES256" -> null
                }
            }
        }

        # (1 unchanged block hidden)
    }

Plan: 0 to add, 1 to change, 0 to destroy.

Third apply:

Note: Objects have changed outside of Terraform

Terraform detected the following changes made outside of Terraform since the last "terraform apply":

  # aws_s3_bucket_lifecycle_configuration.global_test_ouput has been deleted
  - resource "aws_s3_bucket_lifecycle_configuration" "global_test_ouput" {
      - bucket = "ben-test-config-bucket" -> null
      - id     = "ben-test-config-bucket" -> null

      - rule {
          - id     = "config" -> null
          - status = "Enabled" -> null

          - expiration {
              - days                         = 180 -> null
              - expired_object_delete_marker = false -> null
            }

          - filter {
            }

          - transition {
              - days          = 30 -> null
              - storage_class = "STANDARD_IA" -> null
            }
          - transition {
              - days          = 90 -> null
              - storage_class = "GLACIER" -> null
            }
        }
    }

  # aws_s3_bucket_server_side_encryption_configuration.global_test_ouput has been deleted
  - resource "aws_s3_bucket_server_side_encryption_configuration" "global_test_ouput" {
      - bucket = "ben-test-config-bucket" -> null
      - id     = "ben-test-config-bucket" -> null

      - rule {
          - bucket_key_enabled = false -> null

          - apply_server_side_encryption_by_default {
              - sse_algorithm = "AES256" -> null
            }
        }
    }


Unless you have made equivalent changes to your configuration, or ignored the relevant attributes using ignore_changes, the following plan may include actions to undo or respond to
these changes.

────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # aws_s3_bucket_lifecycle_configuration.global_test_ouput will be created
  + resource "aws_s3_bucket_lifecycle_configuration" "global_test_ouput" {
      + bucket = "ben-test-config-bucket"
      + id     = (known after apply)

      + rule {
          + id     = "config"
          + status = "Enabled"

          + expiration {
              + days                         = 180
              + expired_object_delete_marker = (known after apply)
            }

          + transition {
              + days          = 30
              + storage_class = "STANDARD_IA"
            }
          + transition {
              + days          = 90
              + storage_class = "GLACIER"
            }
        }
    }

  # aws_s3_bucket_server_side_encryption_configuration.global_test_ouput will be created
  + resource "aws_s3_bucket_server_side_encryption_configuration" "global_test_ouput" {
      + bucket = "ben-test-config-bucket"
      + id     = (known after apply)

      + rule {
          + apply_server_side_encryption_by_default {
              + sse_algorithm = "AES256"
            }
        }
    }

Plan: 2 to add, 0 to change, 0 to destroy.

...and the cycle repeats from there

Steps to Reproduce

1. terraform apply multiple times

Important Factoids

NA

References

NA

[gai6948]

Same issue, every time terraform apply gives me different result

[justinretzolk]

Hey y'all 👋 It looks like the issue you're running into here is similar to what was reported in #23758. This behavior is noted on the aws_s3_bucket_server_side_encryption_configuration and aws_s3_bucket_lifecycle_configuration documentation pages under the "Usage Notes" section. To correct this, the following should be added to the aws_s3_bucket.global_test_ouput resource definition:

lifecycle {
  ignore_changes = [
    lifecycle_rule,
    server_side_encryption_configuration
  ]
}

[benjamin-wright]

Hi @justinretzolk, thanks for the workaround, and sorry for missing that note in the docs! Obviously we don't get the drift detection with ignore_changes, and we're stuck down at 3.X for the time being by a third-party dependency that is taking a while to upgrade to 4. Is there any plan to support this properly in 3.X in the near future?

[benjamin-wright]

Not obviously, the bucket will ignore the changes but the sub-resources will manage them properly, right? In which case, disregard my previous :)

[justinretzolk]

Hey @benjamin-wright -- you're correct on your follow up note 🙂. The sub resources will handle any configuration drift/changes, so the aws_s3_bucket resource doesn't need to, so the ignore_changes won't have any "real" impact as far as managing the resources in AWS goes.

[liviudm]

Are there any plans to make a bugfix release for this? The problem seems fixed in 4.0, but we still use modules that depend on v3

[justinretzolk]

Hi @liviudm 👋 For v3 versions of the sub resources, this behavior is expected, and will not be changing in any future 3.x releases. Initially, the sub resources were not included in 3.x, but were backported with the notes about needing ignore_changes to make it a bit easier to transition from 3.x to 4.x. There's quite a bit more detail around some of this in #23106, where most of the conversation around the 4.x S3 bucket changes has taken place.

[dan-petty]

@justinretzolk Just another example of how this fracturing of resources makes use of the provider unnecessarily complex. Adding gotchas to a working implementation is not progress.

[dan-petty]

You didn't scrub 20 pages of documentation to configure your s3 bucket the new way?!? GOTCHA!!!

[github-actions]

Marking this issue as stale due to inactivity. This helps our maintainers find and focus on the active issues. If this issue receives no comments in the next 30 days it will automatically be closed. Maintainers can also remove the stale label.

If this issue was automatically closed and you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thank you!

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.