Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

aws_elasticsearch_domain: opensearch support #20853

Closed
aviau opened this issue Sep 9, 2021 · 18 comments · Fixed by #23902
Closed

aws_elasticsearch_domain: opensearch support #20853

aviau opened this issue Sep 9, 2021 · 18 comments · Fixed by #23902
Assignees
Labels
enhancement Requests to existing resources that expand the functionality or scope. service/elasticsearch Issues and PRs that pertain to the elasticsearch service.
Milestone

Comments

@aviau
Copy link
Contributor

aviau commented Sep 9, 2021

Community Note

  • Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
  • Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
  • If you are interested in working on this issue or have submitted a pull request, please leave a comment

Description

OpenSearch was now released. The terraform module could consider renaming resources.

Also, OpenSearch supports an Enable compatibility mode option. This should probably be exposed from the module.

New or Affected Resource(s)

  • aws_elasticsearch_domain
@aviau aviau added the enhancement Requests to existing resources that expand the functionality or scope. label Sep 9, 2021
@github-actions github-actions bot added needs-triage Waiting for first response or review from a maintainer. service/elasticsearch Issues and PRs that pertain to the elasticsearch service. labels Sep 9, 2021
@ewbankkit ewbankkit removed the needs-triage Waiting for first response or review from a maintainer. label Sep 9, 2021
@joshedneyaerian
Copy link

Any movement or ETA on this feature request ?

@mqchau
Copy link

mqchau commented Oct 4, 2021

FWIW, I upgrade terraform-provider-aws to version 3.60 and was able to use string OpenSearch_1.0 in the field of elasticsearch_version.

PR to support OpenSearch in aws-sdk-go: aws/aws-sdk-go#4101

The PR to pull version 1.45 of aws-sdk-go into terraform-provider-aws: #20975

@mqchau
Copy link

mqchau commented Oct 4, 2021

In terms of Enable compatibility mode option, I haven't tried this yet but I think you can add

resource "aws_elasticsearch_domain" "es" {
  ...
  advanced_options = {
    "override_main_response_version" = "true"
  }
}

Source:

@palmajano
Copy link

palmajano commented Oct 5, 2021

FWIW, I upgrade terraform-provider-aws to version 3.60 and was able to use string OpenSearch_1.0 in the field of elasticsearch_version.

terraform-provider-aws version 3.57 also allows upgrading with string OpenSearch_1.0

@Sebor
Copy link

Sebor commented Oct 15, 2021

In terms of Enable compatibility mode option, I haven't tried this yet but I think you can add

resource "aws_elasticsearch_domain" "es" {
  ...
  advanced_options = {
    "override_main_response_version" = "true"
  }
}

Unfortunately it does not work:

Error: ValidationException: Unrecognized advanced option 'override_main_response_version' passed in advancedOptions.

terraform-provider-aws version 3.63.0
terraform version 1.0.9

@mqchau
Copy link

mqchau commented Oct 16, 2021

I'm still using ElasticSearch 7.10 and not OpenSearch 1.0 because of cross cluster replication. But I tried to create a new OpenSearch 1.0 cluster like this

provider "aws" {
  region              = "us-west-2"
}

terraform {
  required_version = "1.0.7"

  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "3.63.0"
    }
  }
}

resource "aws_elasticsearch_domain" "search_engine_2" {
  domain_name = "test-opensearch"

  elasticsearch_version = "OpenSearch_1.0"

  cluster_config {
    instance_type            = "t3.medium.elasticsearch"
    instance_count           = 1
    dedicated_master_enabled = false
    zone_awareness_enabled = false
  }

  ebs_options {
    volume_size = 50
    volume_type = "gp2"
    ebs_enabled = true
  }

  snapshot_options {
    automated_snapshot_start_hour = 23
  }

  advanced_options = {
    "override_main_response_version" = "true"
  }
}

I was able to run terraform apply on it

@mqchau
Copy link

mqchau commented Oct 16, 2021

oh I see the error now

aws_elasticsearch_domain.search_engine_2: Modifying... [id=arn:aws:es:us-west-2:410444354559:domain/quan-test-opensearch]
╷
│ Error: ValidationException: Unrecognized advanced option 'override_main_response_version' passed in advancedOptions.
│
│   with aws_elasticsearch_domain.search_engine_2,
│   on main.tf line 17, in resource "aws_elasticsearch_domain" "search_engine_2":
│   17: resource "aws_elasticsearch_domain" "search_engine_2" {
│
╵

I could not run terraform apply the 2nd time with that setting.

@Sebor
Copy link

Sebor commented Oct 16, 2021

With upgrade from elasticseaarch it is not working either

@mqchau
Copy link

mqchau commented Oct 16, 2021

I dig deeper and learn these:

  1. This is the correct request to update OpenSearch domain

https://docs.aws.amazon.com/opensearch-service/latest/developerguide/configuration-api.html#configuration-api-actions-updatedomainconfig

POST /2021-01-01/opensearch/domain/domain-name/config
  1. terraform is using aws-sdk-go v1, which uses old API endpoint

https://github.com/aws/aws-sdk-go/blob/bdbaf04864f503518c19b5f6d83e62ebf5d6bbe4/service/elasticsearchservice/api.go#L4094
The code in terraform calling aws-sdk-go:

input := elasticsearch.UpdateElasticsearchDomainConfigInput{

POST /2015-01-01/es/domain/test-opensearch/config HTTP/1.1
...

{"AdvancedOptions":{"override_main_response_version":"true"}}
  1. The new API endpoint is available at aws-sdk-go v2

https://github.com/aws/aws-sdk-go-v2/blob/3a8ba5eae1d5246ab4a835a1ca4b4dc4a94d1fd1/codegen/sdk-codegen/aws-models/opensearch.2021-01-01.json#L6485

  1. Version 4.x of this provider will use aws-sdk-go-v2 Upcoming Changes in Version 4.0 of the AWS Provider #20433

Basically to support this setting, I would wait until version 4.x of the provider. From now until then, I suggest to manually do the upgrade and/or the advanced options and ignore that in terraform.

@Gowiem
Copy link

Gowiem commented Nov 3, 2021

Just wanted to highlight that this issue of not supporting override_main_response_version as an advanced option blocks a workaround to a perpetual drift issue within the provider as well #21318. Ideally that advanced option would be supported by the v3.x provider so this isn't a "wait until 4.x and deal with the upgrade" type of thing.

@nnovaes
Copy link
Contributor

nnovaes commented Dec 13, 2021

Would closing #20433 be sufficient to allow an upgrade to 1.x via terraform or support for this needs to be added specifically for aws opensearch?

@tmccombs
Copy link
Contributor

the opensearch API is avaliable in v1 as well: https://pkg.go.dev/github.com/aws/aws-sdk-go@v1.42.33/service/opensearchservice

@breathingdust
Copy link
Member

Hi all 👋 Just letting you know that this is issue is featured on this quarters roadmap. If a PR exists to close the issue a maintainer will review and either make changes directly, or work with the original author to get the contribution merged. If you have written a PR to resolve the issue please ensure the "Allow edits from maintainers" box is checked. Thanks for your patience and we are looking forward to getting this merged soon!

@YakDriver
Copy link
Member

There are several problems with adding OpenSearch support to the existing Elasticsearch resources in TF AWS provider:

  • It looks sort of wonky to have AWS console & docs be predominantly OpenSearch but TF AWS provider being all Elasticsearch
  • There are enough differences that it's not clean to simply add the functionality. E.g., Elasticsearch versions are 6.9, 7.1 while OpenSearch versions are OpenSearch_1.1, Elasticsearch_6.9, Elasticsearch_7.1. It's also called engine_version now instead of elasticsearch_version.
  • From a practitioner perspective, ARNs, endpoints, policies are different. AWS makes these backward-compatible mostly but that may change if/when Elasticsearch is deprecated as a service and absorbed by OpenSearch.

We can avoid breaking changes and support OpenSearch by simply following AWS's lead and re-implementing the Elasticsearch service as the OpenSearch service (see #23902). Resources will be named correctly, and the structural changes are not a problem with new resources. It will be clear to practitioners that they are using the OpenSearch service and can, within that, use Elasticsearch or OpenSearch, with the appropriate ARNs, endpoints, and policies.

@YakDriver YakDriver self-assigned this Mar 28, 2022
@tmccombs
Copy link
Contributor

That seems like the right move to me as well. Thanks for making PR for that @YakDriver

@github-actions
Copy link

github-actions bot commented Apr 7, 2022

This functionality has been released in v4.9.0 of the Terraform AWS Provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template. Thank you!

@lizsnyder
Copy link

@github-actions
Copy link

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

@github-actions github-actions bot locked as resolved and limited conversation to collaborators May 11, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
enhancement Requests to existing resources that expand the functionality or scope. service/elasticsearch Issues and PRs that pertain to the elasticsearch service.
Projects
None yet
Development

Successfully merging a pull request may close this issue.