[Bug]: aws_backup_plan unable to apply in GovCloud

[tmccombs]

Terraform Core Version

1.6.2

AWS Provider Version

5.33.0

Affected Resource(s)

• aws_backup_plan

Expected Behavior

Able to create a plan with a rule with a lifecycle in in GovCloud

Actual Behavior

It fails with the error seen below

Relevant Error/Panic Output Snippet

│ Error: creating Backup Plan: InvalidParameterValueException: Invalid lifecycle. EBS Cold Tier is not yet supported.
│ {
│   RespMetadata: {
│     StatusCode: 400,
│     RequestID: "REDACTED"
│   },
│   Message_: "Invalid lifecycle. EBS Cold Tier is not yet supported."
│ }
│ 
│   with module.tfe.aws_backup_plan.main,
│   on ../modules/terraform_enterprise/backup.tf line 15, in resource "aws_backup_plan" "main":
│   15: resource "aws_backup_plan" "main" {
│

Terraform Configuration Files

resource "aws_backup_plan" "main" {
  name = var.name

  rule {
    rule_name                = "tfe-db"
    target_vault_name        = aws_backup_vault.main.name
    schedule                 = "cron(0 8 * * ? *)" # Schedule needed due to https://github.com/hashicorp/terraform-provider-aws/issues/22627
    enable_continuous_backup = true
    recovery_point_tags      = local.base_tags

    copy_action {
      destination_vault_arn = aws_backup_vault.backup_region.arn

      lifecycle {
        delete_after = 30
      }
    }

    lifecycle {
      delete_after = 30
    }
  }

  tags = local.base_tags
}

Notice that I do not set any parameters related to cold storage.

Steps to Reproduce

I suspect the problem is that the request includes OptInToArchiveForSupportedResources (even though I don't supply a value for that). And the AWS API is rejecting that.

This might be a bug in the AWS SDK, or even the service itself.

Debug Output

2024-01-26T14:59:53.650-0700 [DEBUG] provider.terraform-provider-aws_v5.33.0_x5: HTTP Request Sent: http.flavor=1.1 rpc.method=CreateBackupPlan tf_req_id=REDACTED @module=aws http.url=https://backup-fips.us-gov-west-1.amazonaws.com/backup/plans/ tf_resource_type=aws_backup_plan tf_provider_addr=registry.terraform.io/hashicorp/aws tf_rpc=ApplyResourceChange @caller=github.com/hashicorp/aws-sdk-go-base/v2/awsv1shim/v2@v2.0.0-beta.47/logger.go:109 aws.region=us-gov-west-1 http.method=PUT
  http.request.body=
  | {"BackupPlan":{"AdvancedBackupSettings":[],"BackupPlanName":"tfe","Rules":[{"CompletionWindowMinutes":180,"CopyActions":[{"DestinationBackupVaultArn":"arn:aws-us-gov:backup:us-gov-east-1:REDACTED:backup-vault:tfe","Lifecycle":{"DeleteAfterDays":30,"OptInToArchiveForSupportedResources":false}}],"EnableContinuousBackup":true,"Lifecycle":{"DeleteAfterDays":30,"OptInToArchiveForSupportedResources":false},"RecoveryPointTags":{"BillingGroup":"tfe","lucid:criticality":"critical","lucid:owner":"SRE","lucid:service":"tfe","lucid:usage":"terraform"},"RuleName":"tfe-db","ScheduleExpression":"cron(0 8 * * ? *)","StartWindowMinutes":60,"TargetBackupVaultName":"tfe"}]},"BackupPlanTags":{"BillingGroup":"tfe","lucid:criticality":"critical","lucid:owner":"SRE","lucid:service":"tfe","lucid:usage":"terraform"}}
   http.request.header.x_amz_security_token="*****" rpc.service=Backup http.request.header.authorization="AWS4-HMAC-SHA256 Credential=************/20240126/us-gov-west-1/backup/aws4_request, SignedHeaders=content-length;content-type;host;x-amz-date;x-amz-security-token, Signature=*****" tf_mux_provider="*schema.GRPCProviderServer" rpc.system=aws-api tf_aws.sdk=aws-sdk-go http.request.header.content_type=application/json http.request.header.x_amz_date=20240126T215953Z http.request_content_length=807 http.user_agent="APN/1.0 HashiCorp/1.0 Terraform/1.6.2 (+https://www.terraform.io) terraform-provider-aws/5.33.0 (+https://registry.terraform.io/providers/hashicorp/aws) aws-sdk-go/1.49.19 (go1.20.12; linux; amd64)" net.peer.name=backup-fips.us-gov-west-1.amazonaws.com timestamp=2024-01-26T14:59:53.650-0700
2024-01-26T14:59:54.088-0700 [DEBUG] provider.terraform-provider-aws_v5.33.0_x5: HTTP Response Received: @module=aws http.duration=437 http.response.header.date="Fri, 26 Jan 2024 21:59:54 GMT" tf_req_id=REDACTED http.response.header.content_type=application/json http.response.header.x_amzn_errortype=InvalidParameterValueException rpc.service=Backup rpc.system=aws-api tf_provider_addr=registry.terraform.io/hashicorp/aws tf_mux_provider="*schema.GRPCProviderServer" @caller=github.com/hashicorp/aws-sdk-go-base/v2/awsv1shim/v2@v2.0.0-beta.47/logger.go:157 aws.region=us-gov-west-1
  http.response.body=
  | {"Code":null,"Context":null,"Message":"Invalid lifecycle. EBS Cold Tier is not yet supported.","Type":null}
   http.response.header.x_amzn_requestid=REDACTED http.status_code=400 tf_aws.sdk=aws-sdk-go tf_resource_type=aws_backup_plan tf_rpc=ApplyResourceChange http.response_content_length=107 rpc.method=CreateBackupPlan timestamp=2024-01-26T14:59:54.088-0700
2024-01-26T14:59:54.088-0700 [ERROR] provider.terraform-provider-aws_v5.33.0_x5: Response contains error diagnostic: tf_rpc=ApplyResourceChange tf_resource_type=aws_backup_plan tf_req_id=REDACTED @module=sdk.proto diagnostic_detail="" diagnostic_severity=ERROR tf_proto_version=5.4 @caller=github.com/hashicorp/terraform-plugin-go@v0.20.0/tfprotov5/internal/diag/diagnostics.go:62
  diagnostic_summary=
  | creating Backup Plan: InvalidParameterValueException: Invalid lifecycle. EBS Cold Tier is not yet supported.
  | {
  |   RespMetadata: {
  |     StatusCode: 400,
  |     RequestID: "REDACTED"
  |   },
  |   Message_: "Invalid lifecycle. EBS Cold Tier is not yet supported."
  | }
   tf_provider_addr=registry.terraform.io/hashicorp/aws timestamp=2024-01-26T14:59:54.088-0700
2024-01-26T14:59:54.090-0700 [DEBUG] State storage *remote.State declined to persist a state snapshot
2024-01-26T14:59:54.090-0700 [ERROR] vertex "module.tfe.aws_backup_plan.main" error: creating Backup Plan: InvalidParameterValueException: Invalid lifecycle. EBS Cold Tier is not yet supported.
{
  RespMetadata: {
    StatusCode: 400,
    RequestID: "REDACTED"
  },
  Message_: "Invalid lifecycle. EBS Cold Tier is not yet supported."
}

Panic Output

No response

Important Factoids

This might be a bug in the AWS SDK, or even the service itself.

I suspect it has to do with the OptInToArchiveForSupportedResources option that is being passed in the Rules.CopyActions.Lifecycle part of the request.

Possibly Terraform would be able to work around this passing null instead of false to lifecycle.OptInToArchiveForSupportedResources if that option is null or unspecified.

References

• https://docs.aws.amazon.com/aws-backup/latest/devguide/whatisbackup.html (mentions that cold storage isn't available in GovCloud)
• https://pkg.go.dev/github.com/aws/aws-sdk-go-v2/service/backup@v1.31.3/types#Lifecycle (notice that OptInToArchiveForSupportedResoruces is a pointer to a bool instead of a bool and can thus be null).

Would you like to implement a fix?

None

[github-actions]

Community Note

Voting for Prioritization

• Please vote on this issue by adding a 👍 reaction to the original post to help the community and maintainers prioritize this request.
• Please see our prioritization guide for information on how we prioritize.
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.

Volunteering to Work on This Issue

• If you are interested in working on this issue, please leave a comment.
• If this would be your first contribution, please review the contribution guide.

[acwwat]

According to the AWS Backup Developer Guide, EBS cold storage archive tier is indeed not supported in the GovCloud regions:

Amazon EBS cold storage archive tier is not available in is not currently available in China (Beijing), China (Ningxia), AWS GovCloud (US-East) and AWS GovCloud (US-West) Regions.

Thus the AWS API is failing as expected.

[tmccombs]

But I'm not using the cold storage tier. Why is it expected that if I pass false to an option related to cold storage, the API will fail?

[acwwat]

In the earlier comment I was just making note of the EBS code storage support statement more explicitly. I was still digging into this interesting problem.

I believe that the conversion is a result how the Terraform Plugin SDK is handling schema attributes. hashicorp/terraform-plugin-sdk#817 describes the problem and a seeming lack of support for a tri-state boolean (any primitive value for that matter). There are some workarounds but I am not sure which one is the most appropriate architecturally speaking. Terraform is highly discouraging the use of d.getOkExists but also not providing an alternative. The next best thing is d.GetRawConfig but it's considered experimental. Either way, it would also be interesting to see how these can be used in the context of configuration blocks.

I've also seen in some other AWS resource code that related attributes are used to determine the setting. For example, one could pass in opt_in_to_archive_for_supported_resources only if cold_storage_after is set (or > 0) during expansion. But this doesn't feel exactly right.

Just leaving some research notes here - I think we'll need a core contributor to chime in.

[justinretzolk]

Hey @tmccombs 👋 Thanks for another great bug report! Did this come up after upgrading to 5.33.0? Looking at whether this needs a regression label.

[tmccombs]

I don't know if it is a regression, this is the first time I've tried creating this resource in a GovCloud region.

[justinretzolk]

Related #34994

☝️ With that (and the more recent comments on it), labeling this a regression. @tmccombs with that in mind, if you happen to be able to use 5.32.1, you should be able to work around this for now.

[tmccombs]

ok thanks.

Another thought on this:

During creation, if the value is false set the OptInToArchiveForSupportedResources parameter to nil instead.

During update, if the value is false and the previous value was also false (or nil) , then don't set OptInToArchiveForSupportedResources. (or set it to nil).

[tmccombs]

Using 5.32.1 does work for me.

[tmccombs]

Thanks for fixing that so quickly!

[github-actions]

This functionality has been released in v5.35.0 of the Terraform AWS Provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template. Thank you!

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.