r/s3_bucket_lifecycle_configuration: update value set in state for an empty filter argument

[anGie44]

Community Note

• Please vote on this pull request by adding a 👍 reaction to the original pull request comment to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for pull request followers and do not help prioritize the request

Closes #23228

Description

• Perhaps we don't need the special handling if the tests still pass? @YakDriver

Output from acceptance testing:

$ make testacc TESTARGS='-run=TestAccS3BucketLifecycleConfiguration' PKG=s3 ACCTEST_PARALLELISM=5
--- PASS: TestAccS3BucketLifecycleConfiguration_basic (197.31s)
--- PASS: TestAccS3BucketLifecycleConfiguration_EmptyFilter_NonCurrentVersions (197.80s)
--- PASS: TestAccS3BucketLifecycleConfiguration_TransitionZeroDays_intelligentTiering (197.94s)
--- PASS: TestAccS3BucketLifecycleConfiguration_RuleExpiration_emptyBlock (197.94s)
--- PASS: TestAccS3BucketLifecycleConfiguration_TransitionUpdateBetweenDaysAndDate_intelligentTiering (329.40s)
--- PASS: TestAccS3BucketLifecycleConfiguration_TransitionStorageClassOnly_intelligentTiering (194.99s)
--- PASS: TestAccS3BucketLifecycleConfiguration_nonCurrentVersionExpiration (198.93s)
--- PASS: TestAccS3BucketLifecycleConfiguration_ruleAbortIncompleteMultipartUpload (275.39s)
--- PASS: TestAccS3BucketLifecycleConfiguration_RuleExpiration_expireMarkerOnly (279.81s)
--- PASS: TestAccS3BucketLifecycleConfiguration_prefix (192.03s)
--- PASS: TestAccS3BucketLifecycleConfiguration_disappears (55.15s)
--- PASS: TestAccS3BucketLifecycleConfiguration_nonCurrentVersionTransition (192.90s)
--- PASS: TestAccS3BucketLifecycleConfiguration_multipleRules (192.55s)
--- PASS: TestAccS3BucketLifecycleConfiguration_disableRule (326.82s)
--- PASS: TestAccS3BucketLifecycleConfiguration_filterWithPrefix (272.41s)
--- PASS: TestAccS3BucketLifecycleConfiguration_TransitionDate_intelligentTiering (194.79s)
--- PASS: TestAccS3BucketLifecycleConfiguration_TransitionDate_standardIa (203.78s)

[YakDriver]

Import is not happy with this right now.

My 3.74 Config

resource "aws_s3_bucket" "example" {
  bucket = "barrameda-994617143"

  lifecycle_rule {
    id      = "log-expiration"
    enabled = true
    prefix  = ""

    transition {
      days          = 30
      storage_class = "STANDARD_IA"
    }

    transition {
      days          = 180
      storage_class = "GLACIER"
    }
  }
}

resource "aws_s3_bucket" "example2" {
  bucket = "valladolid-817999296"

  lifecycle_rule {
    id      = "Keep previous version 30 days, then in Glacier another 60"
    enabled = true

    noncurrent_version_transition {
      days          = 30
      storage_class = "GLACIER"
    }

    noncurrent_version_expiration {
      days = 90
    }
  }

  lifecycle_rule {
    id                                     = "Delete old incomplete multi-part uploads"
    enabled                                = true
    abort_incomplete_multipart_upload_days = 7
  }
}

resource "aws_s3_bucket" "example3" {
  bucket = "granada-2433251135"

  lifecycle_rule {
    id      = "log-expiration"
    enabled = true
    prefix = "log/"

    transition {
      days          = 30
      storage_class = "STANDARD_IA"
    }

    transition {
      days          = 180
      storage_class = "GLACIER"
    }
  }
}

resource "aws_s3_bucket" "example4" {
  bucket = "marbella-1354962447"

  lifecycle_rule {
    id      = "log-expiration"
    enabled = true

    transition {
      days          = 30
      storage_class = "STANDARD_IA"
    }

    transition {
      days          = 180
      storage_class = "GLACIER"
    }
  }
}

Import/v4 Config

resource "aws_s3_bucket" "example" {
  bucket = "barrameda-994617143"
}

resource "aws_s3_bucket_lifecycle_configuration" "example" {
  bucket = aws_s3_bucket.example.id

  rule {
    id     = "log-expiration"
    status = "Enabled"
    prefix = ""

    transition {
      days          = 30
      storage_class = "STANDARD_IA"
    }

    transition {
      days          = 180
      storage_class = "GLACIER"
    }
  }
}

resource "aws_s3_bucket" "example2" {
  bucket = "valladolid-817999296"
}

resource "aws_s3_bucket_lifecycle_configuration" "example2" {
  bucket = aws_s3_bucket.example2.id

  rule {
    id     = "Keep previous version 30 days, then in Glacier another 60"
    status = "Enabled"

    noncurrent_version_transition {
      noncurrent_days = 30
      storage_class   = "GLACIER"
    }

    noncurrent_version_expiration {
      noncurrent_days = 90
    }
  }

  rule {
    id     = "Delete old incomplete multi-part uploads"
    status = "Enabled"

    abort_incomplete_multipart_upload {
      days_after_initiation = 7
    }
  }
}

resource "aws_s3_bucket" "example3" {
  bucket = "granada-2433251135"
}

resource "aws_s3_bucket_lifecycle_configuration" "example3" {
  bucket = aws_s3_bucket.example3.id

  rule {
    id     = "log-expiration"
    status = "Enabled"
    
    filter {
        prefix = "log/"
    }

    transition {
      days          = 30
      storage_class = "STANDARD_IA"
    }

    transition {
      days          = 180
      storage_class = "GLACIER"
    }
  }
}

resource "aws_s3_bucket" "example4" {
  bucket = "marbella-1354962447"
}

resource "aws_s3_bucket_lifecycle_configuration" "example4" {
  bucket = aws_s3_bucket.example4.id

  rule {
    id     = "log-expiration"
    status = "Enabled"

    transition {
      days          = 30
      storage_class = "STANDARD_IA"
    }

    transition {
      days          = 180
      storage_class = "GLACIER"
    }
  }
}

Importing with this PR

aws_s3_bucket.example: Refreshing state... [id=barrameda-994617143]
aws_s3_bucket.example4: Refreshing state... [id=marbella-1354962447]
aws_s3_bucket.example2: Refreshing state... [id=valladolid-817999296]
aws_s3_bucket.example3: Refreshing state... [id=granada-2433251135]
aws_s3_bucket_lifecycle_configuration.example2: Refreshing state... [id=valladolid-817999296]
aws_s3_bucket_lifecycle_configuration.example: Refreshing state... [id=barrameda-994617143]
aws_s3_bucket_lifecycle_configuration.example3: Refreshing state... [id=granada-2433251135]
aws_s3_bucket_lifecycle_configuration.example4: Refreshing state... [id=marbella-1354962447]

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following
symbols:
  ~ update in-place

Terraform will perform the following actions:

  # aws_s3_bucket_lifecycle_configuration.example will be updated in-place
  ~ resource "aws_s3_bucket_lifecycle_configuration" "example" {
        id     = "barrameda-994617143"
        # (1 unchanged attribute hidden)

      ~ rule {
            id     = "log-expiration"
            # (1 unchanged attribute hidden)

          - filter {
            }

            # (2 unchanged blocks hidden)
        }
    }

  # aws_s3_bucket_lifecycle_configuration.example2 will be updated in-place
  ~ resource "aws_s3_bucket_lifecycle_configuration" "example2" {
        id     = "valladolid-817999296"
        # (1 unchanged attribute hidden)

      ~ rule {
            id     = "Keep previous version 30 days, then in Glacier another 60"
            # (1 unchanged attribute hidden)

          - filter {
            }


            # (2 unchanged blocks hidden)
        }
      ~ rule {
            id     = "Delete old incomplete multi-part uploads"
            # (1 unchanged attribute hidden)


          - filter {
            }
            # (1 unchanged block hidden)
        }
    }

  # aws_s3_bucket_lifecycle_configuration.example4 will be updated in-place
  ~ resource "aws_s3_bucket_lifecycle_configuration" "example4" {
        id     = "marbella-1354962447"
        # (1 unchanged attribute hidden)

      ~ rule {
            id     = "log-expiration"
            # (1 unchanged attribute hidden)

          - filter {
            }

            # (2 unchanged blocks hidden)
        }
    }

Plan: 0 to add, 3 to change, 0 to destroy.

Importing with v4.1.0

aws_s3_bucket.example4: Refreshing state... [id=marbella-1354962447]
aws_s3_bucket.example2: Refreshing state... [id=valladolid-817999296]
aws_s3_bucket.example3: Refreshing state... [id=granada-2433251135]
aws_s3_bucket.example: Refreshing state... [id=barrameda-994617143]
aws_s3_bucket_lifecycle_configuration.example3: Refreshing state... [id=granada-2433251135]
aws_s3_bucket_lifecycle_configuration.example2: Refreshing state... [id=valladolid-817999296]
aws_s3_bucket_lifecycle_configuration.example: Refreshing state... [id=barrameda-994617143]
aws_s3_bucket_lifecycle_configuration.example4: Refreshing state... [id=marbella-1354962447]

No changes. Your infrastructure matches the configuration.

Terraform has compared your real infrastructure against your configuration and found no differences, so no changes are needed.

Apply complete! Resources: 0 added, 0 changed, 0 destroyed.

[anGie44]

My current observations:

If we apply aws_s3_bucket.example using provider v3.74.1, when we run aws s3api get-bucket-lifecycle-configuration barrameda-994617143 it returns:

{
    "Rules": [
        {
            "ID": "example",
            "Filter": {
                "Prefix": ""
            },
            "Status": "Enabled",
            "Transitions": [
                {
                    "Days": 30,
                    "StorageClass": "STANDARD_IA"
                },
                {
                    "Days": 180,
                    "StorageClass": "GLACIER"
                }
            ]
        }
    ]
}

We can see prefix is housed within filter.
Thus to translate this into v4.0 aws_s3_bucket_lifecycle_configuration, we include filter:

resource "aws_s3_bucket_lifecycle_configuration" "example" {
  bucket = aws_s3_bucket.example.id

  rule {
    id = "example"
    filter {
      prefix = ""
    }

   # ... etc. ...
}

Similarly if we take example2 where prefix is never specified, the AWS CLI returns:

{
    "Rules": [
        {
            "ID": "Keep previous version 30 days, then in Glacier another 60",
            "Filter": {
                "Prefix": ""
            },
            "Status": "Enabled",
            "NoncurrentVersionTransitions": [
                {
                    "NoncurrentDays": 30,
                    "StorageClass": "GLACIER"
                }
            ],
            "NoncurrentVersionExpiration": {
                "NoncurrentDays": 90
            }
        },
        {
            "ID": "Delete old incomplete multi-part uploads",
            "Filter": {
                "Prefix": ""
            },
            "Status": "Enabled",
            "AbortIncompleteMultipartUpload": {
                "DaysAfterInitiation": 7
            }
        }
    ]
}

Here again we see prefix inside filter, so the translation can look like:

resource "aws_s3_bucket_lifecycle_configuration" "example2" {
  bucket = aws_s3_bucket.example2.id

  rule {
    id     = "Keep previous version 30 days, then in Glacier another 60"
    status = "Enabled"

    filter {
       prefix = ""
    }
  
    noncurrent_version_transition {
      noncurrent_days = 30
      storage_class   = "GLACIER"
    }

    noncurrent_version_expiration {
      noncurrent_days = 90
    }
  }

  rule {
    id     = "Delete old incomplete multi-part uploads"
    status = "Enabled"

    filter {
      prefix = ""
    }

    abort_incomplete_multipart_upload {
      days_after_initiation = 7
    }
  }
}

This is all to say that I believe (a) v4.0 was doing the right thing with how filter was getting set back in state; I just didn't communicate it well in the upgrade guide how to consider this translation of prefix to filter {} (b) we need to reinstate setting the filter block with an empty prefix string as suggested in the flex.go code change.

Perhaps now the question is, if buckets are old, and the lifecycle configuration was configured in V1 format, does the s3api still return V2 format as seen above OR is V1 returned? something to ask the OP.

[YakDriver]

Not a fan of having to set anything to empty or an empty string to make it work. Seems messy. I'd prefer a diff suppress. But, if needs must.

[anGie44]

@YakDriver one last experiment to show the change IMO represents reality.

Let's say i create a bucket in the console, and the aws s3api get-bucket-lifecycle-configuration --bucket its-me-bb-bucket returns:

{
    "Rules": [
        {
            "ID": "test",
            "Filter": {},
            "Status": "Enabled",
            "NoncurrentVersionTransitions": [
                {
                    "NoncurrentDays": 30,
                    "StorageClass": "GLACIER_IR"
                }
            ]
        }
    ]
}

and then the AWS Go SDK returns <Filter/>:

<LifecycleConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Rule><ID>test</ID><Filter/><Status>Enabled</Status><NoncurrentVersionTransition><NoncurrentDays>30</NoncurrentDays><StorageClass>GLACIER_IR</StorageClass></NoncurrentVersionTransition></Rule></LifecycleConfiguration>

so for terraform to be representative of what exists in S3, the resource would look like:

resource "aws_s3_bucket_lifecycle_configuration" "example_bb" {
  bucket = "its-me-bb-bucket"
  rule {
    id = "test"
    status = "Enabled"

    filter {}

    noncurrent_version_transition {
      noncurrent_days = 30
      storage_class = "GLACIER_IR"
    }
  }
}

and we should thus expect terraform to recognize the empty filter{} as valid and set it in state, result in

$ terraform show
# aws_s3_bucket_lifecycle_configuration.example_bb:
resource "aws_s3_bucket_lifecycle_configuration" "example_bb" {
    bucket = "its-me-bb-bucket"
    id     = "its-me-bb-bucket"

    rule {
        id     = "test"
        status = "Enabled"

        filter {
        }

        noncurrent_version_transition {
            newer_noncurrent_versions = 0
            noncurrent_days           = 30
            storage_class             = "GLACIER_IR"
        }
    }
}

[YakDriver]

Ah, I see the difference in opinion: viewing what AWS does as a source of correctness vs. not. I like a layer of correctness on top of the API to smooth out the various inconsistencies. But, there's something to be said for adhereing to "the way."

[anGie44]

Ahh yes that's a good way to put it 👍 I think in this case it's important contextual data to keep in terraform state but we can keep discussing 😃 . I'm gonna mark this as ready in the meantime.

[anGie44]

Documentation may need an edit but i'll double check with @justinretzolk if that's already in the works since we somewhat strayed from the original plan in Slack thread

[justinretzolk]

I've not started on documentation edits as of yet @anGie44 - happy to do so, but wanted to keep an eye on what the final outcome was first.

[YakDriver]

MPMB! 🎉 I especially like the branch "fitler!"

As long as I add in the fitler {}, er, filter {} my import tests pass.

Output from acceptance tests (us-west-2):

% make testacc TESTS=TestAccS3BucketLifecycleConfig PKG=s3
==> Checking that code complies with gofmt requirements...
TF_ACC=1 go test ./internal/service/s3/... -v -count 1 -parallel 20 -run='TestAccS3BucketLifecycleConfig'  -timeout 180m
--- PASS: TestAccS3BucketLifecycleConfiguration_disappears (62.22s)
--- PASS: TestAccS3BucketLifecycleConfiguration_basic (201.36s)
--- PASS: TestAccS3BucketLifecycleConfiguration_prefix (206.86s)
--- PASS: TestAccS3BucketLifecycleConfiguration_nonCurrentVersionExpiration (206.87s)
--- PASS: TestAccS3BucketLifecycleConfiguration_TransitionDate_standardIa (206.87s)
--- PASS: TestAccS3BucketLifecycleConfiguration_RuleExpiration_emptyBlock (206.95s)
--- PASS: TestAccS3BucketLifecycleConfiguration_TransitionDate_intelligentTiering (207.01s)
--- PASS: TestAccS3BucketLifecycleConfiguration_TransitionStorageClassOnly_intelligentTiering (207.01s)
--- PASS: TestAccS3BucketLifecycleConfiguration_multipleRules (207.04s)
--- PASS: TestAccS3BucketLifecycleConfiguration_TransitionZeroDays_intelligentTiering (207.10s)
--- PASS: TestAccS3BucketLifecycleConfiguration_nonCurrentVersionTransition (207.11s)
--- PASS: TestAccS3BucketLifecycleConfiguration_EmptyFilter_NonCurrentVersions (208.66s)
--- PASS: TestAccS3BucketLifecycleConfiguration_RuleExpiration_expireMarkerOnly (281.83s)
--- PASS: TestAccS3BucketLifecycleConfiguration_filterWithPrefix (286.26s)
--- PASS: TestAccS3BucketLifecycleConfiguration_ruleAbortIncompleteMultipartUpload (286.55s)
--- PASS: TestAccS3BucketLifecycleConfiguration_disableRule (339.27s)
--- PASS: TestAccS3BucketLifecycleConfiguration_TransitionUpdateBetweenDaysAndDate_intelligentTiering (339.45s)
PASS
ok  	github.com/hashicorp/terraform-provider-aws/internal/service/s3	341.317s

[github-actions]

This functionality has been released in v4.2.0 of the Terraform AWS Provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template. Thank you!

[github-actions]

I'm going to lock this pull request because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems related to this change, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.