ElasticSearch Domain: can't switch back to standard instance storage type after configuring EBS

[walterdolce]

Terraform Version

Terraform v0.10.4

Affected Resource(s)

aws_elasticsearch_domain

Terraform Configuration Files

Note: the IP specified in the access policy below is fictional. Please amend accordingly if necessary.

resource "aws_elasticsearch_domain" "ls_esearch_domain" {
  domain_name = "logstash-elasticsearch"

  cluster_config {
    instance_count = "2"
    instance_type = "i2.xlarge.elasticsearch"
    dedicated_master_enabled = true
    dedicated_master_count = 3
  }

  ebs_options {
    ebs_enabled = false
  }

  access_policies = <<CONFIG
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": "es:*",
            "Principal": "*",
            "Effect": "Allow",
            "Condition": {
                "IpAddress": {
                    "aws:SourceIp": ["0.0.0.0"]
                }
            }
        }
    ]
}
CONFIG

  snapshot_options {
    automated_snapshot_start_hour = 05
  }
}

Debug Output

https://gist.github.com/walterdolce/569829b99cf621cc8e0525d6255e2d19

Panic Output

N/A

Expected Behavior

Terraform should delete/revert all the EBS-related instance options and switch back to "non-EBS standard"

Actual Behavior

Terraform errors with:

ValidationException: EBSEnabled must be set to true to use any of the EBS options.

Steps to Reproduce

1. terraform apply

Important Factoids

N/A

References

N/A

[walterdolce]

More information regarding this as a FYI for you in relation to the exchange of data between Terraform and AWS APIs. Please note this comes straight from AWS Enterprise Support.

“EBSOption” part of describeElasticsearchInstance API response keeps the value of the last configuration that the domain had during “EBSEnabled” was true even after it’s switched to false.
That means, “VolumeSize” or “VolumeType” in the response won’t be removed nor overwritten with the Instance store information even after EBS configuration is disabled.
I understand that Terraform is not working with this API response.
As Terraform is not developed by AWS, we’re not sure what it expects as an API response.

[walterdolce]

Another update from AWS:

The service engineering team of Elasticsearch service have confirmed that the API is not resetting "EBSOptions" information after disabling EBS volumes.
This is current behavior of describeElasticsearchDomain API and we don't have a plan to update it at this point unfortunately.
Apologies for your inconvenience with this behavior but please consider to ignore other values of "EBSOptions" when "EBSEnabled" is false for a workaround.

[christianmeyer]

Is there any update to the issue?
We're facing the same issue, as described above.

12:59:46 module.apps.module.elk.aws_elasticsearch_domain.elk-legacy: Modifying... (ID: arn:aws:es:eu-central-1:***:domain/s01-elk-legacy)
12:59:46 cluster_config.0.instance_type: "r4.large.elasticsearch" => "i3.large.elasticsearch"
12:59:47
12:59:47 Error: Error applying plan:
12:59:47
12:59:47 1 error(s) occurred:
12:59:47
12:59:47 * module.apps.module.elk.aws_elasticsearch_domain.elk-legacy: 1 error(s) occurred:
12:59:47
12:59:47 * aws_elasticsearch_domain.elk-legacy: ValidationException: Instance type i3.large.elasticsearch does not support EBS storage
12:59:47 status code: 400, request id: ***
12:59:47

With the current amount of instance storage class instances, it might make sense to give this a try again.
Funny enough, a out of band change and plan+apply afterwards is a good workaround, but nothing for our CI chain.

[rigregs]

Good morning any update on this? We have a ES cluster using EBS and we want to switch it live using the AWS i3.* instances.

We cannot apply the change via terraform.

This is the error we're getting:

• aws_elasticsearch_domain.aws-elasticsearch-: ValidationException: EBSEnabled must be set to true to use any of the EBS options.
  status code: 400, request id: 12345**

We try to change the instances manually and after ran terraform and we have the same issue.

[bflad]

A fix for this should be released with version 1.23.0 of the AWS provider, likely tomorrow.

[bflad]

This has been released in version 1.23.0 of the AWS provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks!