added opensearch serverless support #92

vasyaxparfenov · 2023-10-09T18:21:35Z

Description

Added Opensearch Serverless support by:

Extending getClient func with configration necessary for working with serverless version.
OSVersion and Flavor values are set explicitly since serverless variant returns empty response on ping request.
Added AWSv4 signing client RoundTripper wrapper to support required header.

Issues Resolved

Closes #90

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

vasyaxparfenov · 2023-10-09T18:38:29Z

I would like to mention that I'm fully aware that dragging hash header creation logic into this solution is a bad idea from maintenance perspective, but, since neither elastic client nor awsv4 signing client don't support this, it seems like the only way besides either creating a set of related PRs in dependant packages repositories or, which would probably be THE BEST WAY, completely switching to native go-opensearch client maintained by opensearch team which seems to support aforementioned requirement by default.

vasyaxparfenov · 2023-10-09T18:39:47Z

We've tested this changes against our private opensearch serverless collection and it successfully created resources.

vasyaxparfenov · 2023-10-09T20:56:25Z

I've run tests locally and they all passed. Can we re-try? It seems strange that flavor was set to 0 (Unknown) as it seems like docker hosted opensearch instance returned a valid version info response

vasyaxparfenov · 2023-10-16T15:33:58Z

@prudhvigodithi could you please approve test run?

prudhvigodithi · 2023-10-19T19:05:32Z

Thanks for your contribution @vasyaxparfenov.

lcapillera · 2023-10-20T14:52:53Z

@vasyaxparfenov Thank you for this. Any idea when this could be released?

prudhvigodithi · 2023-10-23T15:28:37Z

Hey @vasyaxparfenov can you please add some unit tests to the change.
Thank you

vasyaxparfenov · 2023-10-26T07:35:45Z

Thanks for your contribution @vasyaxparfenov.

Done

lcapillera · 2023-10-30T18:33:48Z

@vasyaxparfenov I encounter an error while using this branch in a project. The creation of an index works fine, but when tags are updated in other resources, the opensearch_index resource throws a HEAD healthcheck failed error.

It looks like there are some scenarios where the setting healthcheck = false is not getting into consideration.

The steps are:

Write terraform code to create an opensearch serverless collection with default tags
Write terraform code to create an index
Run apply
Change some tags
Run apply again

Actual Result:

│ Error: HEAD healthcheck failed: This is usually due to network or permission issues. The underlying error isn't accessible, please debug by disabling healthchecks.
│ 
│   with opensearch_index.events_index,
│   on opensearch.tf line 117, in resource "opensearch_index" "events_index":
│  117: resource "opensearch_index" "events_index" {
│

Expected Result:

Update tags and do nothing in relation to the index.

My current configs

Terraform version: 1.6.2
OS: linux
hashicorp/aws version: 5.22

provider "opensearch" {
  url         = data.aws_opensearchserverless_collection.example.collection_endpoint
  healthcheck = false
}

resource "opensearch_index" "events_index" {
  name     = "events"
  mappings = file("${path.module}/opensearch_index_mappings.json")
  force_destroy = false
}

vasyaxparfenov · 2023-10-31T09:20:30Z

@vasyaxparfenov I encounter an error while using this branch in a project. The creation of an index works fine, but when tags are updated in other resources, the opensearch_index resource throws a HEAD healthcheck failed error.

It looks like there are some scenarios where the setting healthcheck = false is not getting into consideration.

The steps are:

Write terraform code to create an opensearch serverless collection with default tags

Write terraform code to create an index

Run apply

Change some tags

Run apply again

Actual Result:
│ Error: HEAD healthcheck failed: This is usually due to network or permission issues. The underlying error isn't accessible, please debug by disabling healthchecks.
│ 
│   with opensearch_index.events_index,
│   on opensearch.tf line 117, in resource "opensearch_index" "events_index":
│  117: resource "opensearch_index" "events_index" {
│ 
Expected Result:

Update tags and do nothing in relation to the index.

My current configs

Terraform version: 1.6.2 OS: linux hashicorp/aws version: 5.22
provider "opensearch" {
  url         = data.aws_opensearchserverless_collection.example.collection_endpoint
  healthcheck = false
}

resource "opensearch_index" "events_index" {
  name     = "events"
  mappings = file("${path.module}/opensearch_index_mappings.json")
  force_destroy = false
}

Thank you for a feedback, @lcapillera. By looking at source code of provider and elasticsearch client used in this solution I've noticed that aforementioned error is returned by provider itself while hiding actual error during client creation process. Could you please enable info level logging and run your setup? We should get a real error which happens during setup rather than generic one.

lcapillera · 2023-10-31T09:58:48Z

@vasyaxparfenov The client error is no active connection found: no Elasticsearch node available:

opensearch_index.events_index: Refreshing state... [id=events]
2023-10-31T10:37:47.788+0100 [INFO]  provider.terraform-provider-opensearch: 2023/10/31 10:37:47 [INFO] couldn't create client: *errors.withStack, no active connection found: no Elasticsearch node available, *errors.withMessage: timestamp="2023-10-31T10:37:47.787+0100"
2023-10-31T10:37:47.788+0100 [WARN]  unexpected data:
  registry.terraform.io/my-opensearch-project/opensearch:stderr=
  | {"@caller":"/home/leo/.asdf/installs/golang/1.20/packages/pkg/mod/github.com/hashicorp/terraform-plugin-sdk/v2@v2.12.0/internal/logging/helper_schema.go:21","@level":"trace","@message":"Calling downstream","@module":"sdk.helper_schema","@timestamp":"2023-10-31T10:37:47.787720+01:00"}
  | {"@caller":"/home/leo/.asdf/installs/golang/1.20/packages/pkg/mod/github.com/hashicorp/terraform-plugin-sdk/v2@v2.12.0/internal/logging/helper_schema.go:21","@level":"trace","@message":"Called downstream","@module":"sdk.helper_schema","@timestamp":"2023-10-31T10:37:47.787876+01:00"}
  
2023-10-31T10:37:47.788+0100 [ERROR] vertex "opensearch_index.events_index" error: HEAD healthcheck failed: This is usually due to network or permission issues. The underlying error isn't accessible, please debug by disabling healthchecks.
2023-10-31T10:37:47.788+0100 [ERROR] vertex "opensearch_index.events_index (expand)" error: HEAD healthcheck failed: This is usually due to network or permission issues. The underlying error isn't accessible, please debug by disabling healthchecks.

The OpenSearch Serverless cluster is up and running because I can recreate the index by doing:

DELETE /events
terraform state rm opensearch_index.events_index
Run terraform apply.
The index is created successfully

vasyaxparfenov · 2023-10-31T14:08:01Z

@vasyaxparfenov The client error is no active connection found: no Elasticsearch node available:

opensearch_index.events_index: Refreshing state... [id=events]
2023-10-31T10:37:47.788+0100 [INFO]  provider.terraform-provider-opensearch: 2023/10/31 10:37:47 [INFO] couldn't create client: *errors.withStack, no active connection found: no Elasticsearch node available, *errors.withMessage: timestamp="2023-10-31T10:37:47.787+0100"
2023-10-31T10:37:47.788+0100 [WARN]  unexpected data:
  registry.terraform.io/my-opensearch-project/opensearch:stderr=
  | {"@caller":"/home/leo/.asdf/installs/golang/1.20/packages/pkg/mod/github.com/hashicorp/terraform-plugin-sdk/v2@v2.12.0/internal/logging/helper_schema.go:21","@level":"trace","@message":"Calling downstream","@module":"sdk.helper_schema","@timestamp":"2023-10-31T10:37:47.787720+01:00"}
  | {"@caller":"/home/leo/.asdf/installs/golang/1.20/packages/pkg/mod/github.com/hashicorp/terraform-plugin-sdk/v2@v2.12.0/internal/logging/helper_schema.go:21","@level":"trace","@message":"Called downstream","@module":"sdk.helper_schema","@timestamp":"2023-10-31T10:37:47.787876+01:00"}
  
2023-10-31T10:37:47.788+0100 [ERROR] vertex "opensearch_index.events_index" error: HEAD healthcheck failed: This is usually due to network or permission issues. The underlying error isn't accessible, please debug by disabling healthchecks.
2023-10-31T10:37:47.788+0100 [ERROR] vertex "opensearch_index.events_index (expand)" error: HEAD healthcheck failed: This is usually due to network or permission issues. The underlying error isn't accessible, please debug by disabling healthchecks.

The OpenSearch Serverless cluster is up and running because I can recreate the index by doing:

DELETE /events
terraform state rm opensearch_index.events_index
Run terraform apply.
The index is created successfully

Thank you, @lcapillera ! So this happens mainly when terraform performs requests to refresh its state? I made this assumption based on your original scenario and based on a fact you were able to remove resource from state and re-create it. Are you using this exact branch?

lcapillera · 2023-10-31T15:51:41Z

@vasyaxparfenov It looks like the refreshing state only fails during the apply if there are modification on the tags of any resource. If I change anything other than a tag, there index refreshing works fine.

For instance, if I add this some_tag to the OpenSearch collection, the index refreshing fails:

resource "aws_opensearchserverless_collection" "events_opensearch_collection" {
  name = "example-${var.environment}"
  type = "TIMESERIES"

  tags = {
    some_tag = "some_tag_value"
  }

  depends_on = [
    aws_opensearchserverless_security_policy.events_opensearch_encryption_policy,
    aws_opensearchserverless_security_policy.events_opensearch_network_policy,
    aws_opensearchserverless_access_policy.events_opensearch_access_policy
  ]
}

But the tag change can be on any resource, not related to OpenSearch, for the index refreshing to fail. It's pretty strange.
And yes, I'm using this branch. I'll rebuild the provider object again just in case.

lcapillera · 2023-10-31T16:50:43Z

@vasyaxparfenov I narrowed down the problem a bit more, my current setup is this:


provider "opensearch" {
  url         = data.aws_opensearchserverless_collection.example.collection_endpoint
  healthcheck = false
}

resource "aws_opensearchserverless_collection" "example_opensearch_collection" {
  name = "example"
  type = "TIMESERIES"

  depends_on = [
    aws_opensearchserverless_security_policy.events_opensearch_encryption_policy,
    aws_opensearchserverless_security_policy.events_opensearch_network_policy,
    aws_opensearchserverless_access_policy.events_opensearch_access_policy
  ]
}

data "aws_opensearchserverless_collection" "example" {
  name = "example"
  depends_on = [aws_opensearchserverless_collection.example_opensearch_collection]
}

The index error started to happen when I added depends_on to the data resource, so changing from

data "aws_opensearchserverless_collection" "example" {
  name = "example"
}

to

data "aws_opensearchserverless_collection" "example" {
  name = "example"
  depends_on = [aws_opensearchserverless_collection.example_opensearch_collection]
}

I did that because it wasn't waiting for the collection to be created in the first run, so the provider was getting a null url.

If I remove depends_on = [aws_opensearchserverless_collection.example_opensearch_collection], the index works fine when tags are changed.

So it looks like I have 2 bugs and I can't fix both at the same time :D.

vasyaxparfenov · 2023-10-31T22:12:45Z

@vasyaxparfenov I narrowed down the problem a bit more, my current setup is this:
provider "opensearch" {
  url         = data.aws_opensearchserverless_collection.example.collection_endpoint
  healthcheck = false
}

resource "aws_opensearchserverless_collection" "example_opensearch_collection" {
  name = "example"
  type = "TIMESERIES"

  depends_on = [
    aws_opensearchserverless_security_policy.events_opensearch_encryption_policy,
    aws_opensearchserverless_security_policy.events_opensearch_network_policy,
    aws_opensearchserverless_access_policy.events_opensearch_access_policy
  ]
}

data "aws_opensearchserverless_collection" "example" {
  name = "example"
  depends_on = [aws_opensearchserverless_collection.example_opensearch_collection]
}
The index error started to happen when I added depends_on to the data resource, so changing from
data "aws_opensearchserverless_collection" "example" {
  name = "example"
}
to
data "aws_opensearchserverless_collection" "example" {
  name = "example"
  depends_on = [aws_opensearchserverless_collection.example_opensearch_collection]
}
I did that because it wasn't waiting for the collection to be created in the first run, so the provider was getting a null url.

If I remove depends_on = [aws_opensearchserverless_collection.example_opensearch_collection], the index works fine when tags are changed.

So it looks like I have 2 bugs and I can't fix both at the same time :D.

Thank you for a detailed explanation. It seems like issue is related to aws provider resources. Did you try using collection_endpoint attribute? It will be same as using data_source from logical perspective, but it might impose better dependency relationship between opensearch provider and collection and help terraform understand your intention better.

vasyaxparfenov · 2023-10-31T22:41:08Z

@lcapillera I would actually recommend you to use terragrunt to resolve this issue. On our project we use terragrunt and separate opensearch configuration and setup in two different modules with terragrunt explicit dependency which makes this modules run in specific order due to chicken and egg problem. Even despite my suggestion to use direct dependency to resource rather than using data_source, it won't solve a first run problem due to terraform not being able to handle such thing since it needs all provider to be configured before running refresh (unless you run terraform with target argument first time avoiding opensearch provider configuration until collection is created). One of the reasons why your configuration starts to fail after you add depends_on might be related to a lack of terraform knowledge when running refresh of data_source data due to pending update (new tags for example).

lcapillera · 2023-11-01T10:34:15Z

@vasyaxparfenov Thank you for the links and suggestions. What you said makes sense. But I wonder if the issue I'm facing does occur with OpenSearch Service (as opposed to Serverless), because I don't see any issues in this repo and I guess not everybody uses terragrunt.

vasyaxparfenov · 2023-11-01T13:07:25Z

@vasyaxparfenov Thank you for the links and suggestions. What you said makes sense. But I wonder if the issue I'm facing does occur with OpenSearch Service (as opposed to Serverless), because I don't see any issues in this repo and I guess not everybody uses terragrunt.

@lcapillera could you please try using attribute from collection resource rather than data_source? Based on what is mentioned here we can't be sure that it provides previously used url to opensearch provider when you change inputs on dependant resource. I'm actually curious what it configure provider with. Did your run with logging enabled had any details regarding provider setup? Maybe if you try log level debug we could get some information. Nevertheless I would first try using resource reference explicitly

lcapillera · 2023-11-01T14:28:25Z

@vasyaxparfenov I just tried using the attribute from the collection resource and it works in both cases, first run and later updates. The reason I added the data resource was because this wasn't working, but maybe I had something in a inconsistent state while trying different things. Thanks for your time, I'll let you know if I find anything else, but I guess this PR is ready to go 👏

vasyaxparfenov · 2023-11-01T14:42:05Z

@vasyaxparfenov I just tried using the attribute from the collection resource and it works in both cases, first run and later updates. The reason I added the data resource was because this wasn't working, but maybe I had something in a inconsistent state while trying different things. Thanks for your time, I'll let you know if I find anything else, but I guess this PR is ready to go 👏

@lcapillera I actually have an explanation why it doesn't work when you use data_source. So, it can't determine what attributes data_source is going to have because it depends on collection which is about to change - so it just passes empty string (default value) to provider and golang url.Parse API is ok with empty string being passed and returns url object with default fields. Then when you try to refresh your resource it calls getClient which when creating client checks url, if it doesn't have valid schema it's not added to connection list and then it has logic which checks active connection and that's where it fails because previous step skips empty url object. Mystery solved 🤣

prudhvigodithi · 2023-11-02T19:26:35Z

Thanks for the feedback and the contribution, I will review this PR soon and lets this merged so that this feature is part of the upcoming release. Thanks

prudhvigodithi · 2023-11-02T19:38:53Z

Hey @vasyaxparfenov do you think its useful to add anything in the documentation? https://github.com/opensearch-project/terraform-provider-opensearch/blob/main/CONTRIBUTING.md#generate-documentation-using-tfplugindocs

Signed-off-by: vasyaxparfenov <vasyaxparfenov@gmail.com>

vasyaxparfenov · 2023-11-04T23:39:16Z

Hey @vasyaxparfenov do you think its useful to add anything in the documentation? https://github.com/opensearch-project/terraform-provider-opensearch/blob/main/CONTRIBUTING.md#generate-documentation-using-tfplugindocs

tfplugindocs generate didn't produce any changes, so I guess there is nothing important to add, the usage pattern hasn't changed so to use serverless variant you would just provide url which normally contains all necessary information

prudhvigodithi · 2023-11-06T23:15:26Z

Thanks @vasyaxparfenov, please make sure the CI runs are green and we can proceed to merge the PR.
Error:

         with opensearch_user.test,
          on terraform_plugin_test.tf line 2, in resource "opensearch_user" "test":
           2: resource "opensearch_user" "test" {

vasyaxparfenov · 2023-11-07T13:03:55Z

@prudhvigodithi I've been running tests locally and I don't have similar issue. It might be a bug in tests logic since I can't find a logic path which would lead to Flavor being set to 0.

prudhvigodithi · 2023-11-07T17:29:06Z

provider/awsv4.go

+	"net/http"
+)
+
+var emptyStringSHA256 = "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"


Thanks, can we have the logic so that emptyStringSHA256 is generated dynamically ?

Thanks for your contribution @vasyaxparfenov, just one comment rest the PR LGTM.

Signed-off-by: vasyaxparfenov <vasyaxparfenov@gmail.com>

prudhvigodithi · 2023-11-08T16:47:28Z

LGTM, @vasyaxparfenov, thanks for your contribution.

vasyaxparfenov requested review from peterzhuamazon, prudhvigodithi, jackson-theisen and phillbaker as code owners October 9, 2023 18:21

vasyaxparfenov force-pushed the opensearch_serverless_support branch from 371a713 to a30f48d Compare October 9, 2023 18:25

vasyaxparfenov force-pushed the opensearch_serverless_support branch from a30f48d to f526e6f Compare October 12, 2023 21:54

vasyaxparfenov force-pushed the opensearch_serverless_support branch 2 times, most recently from 9546c59 to 63a3b59 Compare October 19, 2023 13:05

vasyaxparfenov force-pushed the opensearch_serverless_support branch from 0278ef7 to d42fac5 Compare October 24, 2023 15:15

vasyaxparfenov force-pushed the opensearch_serverless_support branch from b62f986 to 7d22515 Compare November 2, 2023 14:17

vasiliyparfenov and others added 2 commits November 5, 2023 01:36

added opensearch serverless support

443640b

Signed-off-by: vasyaxparfenov <vasyaxparfenov@gmail.com>

added unit tests

cce6435

Signed-off-by: vasyaxparfenov <vasyaxparfenov@gmail.com>

vasyaxparfenov force-pushed the opensearch_serverless_support branch from 7d22515 to cce6435 Compare November 4, 2023 23:37

prudhvigodithi reviewed Nov 7, 2023

View reviewed changes

prudhvigodithi mentioned this pull request Nov 8, 2023

When the provider assumes a given role, don't use the default profile… #87

Merged

applied suggestions

dc2947d

Signed-off-by: vasyaxparfenov <vasyaxparfenov@gmail.com>

vasyaxparfenov requested a review from prudhvigodithi November 8, 2023 12:50

prudhvigodithi approved these changes Nov 8, 2023

View reviewed changes

prudhvigodithi merged commit 2245e1d into opensearch-project:main Nov 8, 2023
5 checks passed

prudhvigodithi mentioned this pull request Nov 8, 2023

[Release] Provider version 2.1.0 #122

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

added opensearch serverless support #92

added opensearch serverless support #92

vasyaxparfenov commented Oct 9, 2023 •

edited

Loading

vasyaxparfenov commented Oct 9, 2023

vasyaxparfenov commented Oct 9, 2023 •

edited

Loading

vasyaxparfenov commented Oct 9, 2023

vasyaxparfenov commented Oct 16, 2023

prudhvigodithi commented Oct 19, 2023

lcapillera commented Oct 20, 2023

prudhvigodithi commented Oct 23, 2023

vasyaxparfenov commented Oct 26, 2023

lcapillera commented Oct 30, 2023

vasyaxparfenov commented Oct 31, 2023 •

edited

Loading

Actual Result:

Expected Result:

My current configs

lcapillera commented Oct 31, 2023

vasyaxparfenov commented Oct 31, 2023 •

edited

Loading

lcapillera commented Oct 31, 2023

lcapillera commented Oct 31, 2023

vasyaxparfenov commented Oct 31, 2023 •

edited

Loading

vasyaxparfenov commented Oct 31, 2023 •

edited

Loading

lcapillera commented Nov 1, 2023

vasyaxparfenov commented Nov 1, 2023 •

edited

Loading

lcapillera commented Nov 1, 2023

vasyaxparfenov commented Nov 1, 2023 •

edited

Loading

prudhvigodithi commented Nov 2, 2023

prudhvigodithi commented Nov 2, 2023

vasyaxparfenov commented Nov 4, 2023 •

edited

Loading

prudhvigodithi commented Nov 6, 2023

vasyaxparfenov commented Nov 7, 2023

prudhvigodithi Nov 7, 2023

prudhvigodithi Nov 7, 2023

vasyaxparfenov Nov 8, 2023

prudhvigodithi commented Nov 8, 2023

added opensearch serverless support #92

added opensearch serverless support #92

Conversation

vasyaxparfenov commented Oct 9, 2023 • edited Loading

Description

Issues Resolved

vasyaxparfenov commented Oct 9, 2023

vasyaxparfenov commented Oct 9, 2023 • edited Loading

vasyaxparfenov commented Oct 9, 2023

vasyaxparfenov commented Oct 16, 2023

prudhvigodithi commented Oct 19, 2023

lcapillera commented Oct 20, 2023

prudhvigodithi commented Oct 23, 2023

vasyaxparfenov commented Oct 26, 2023

lcapillera commented Oct 30, 2023

Actual Result:

Expected Result:

My current configs

vasyaxparfenov commented Oct 31, 2023 • edited Loading

Actual Result:

Expected Result:

My current configs

lcapillera commented Oct 31, 2023

vasyaxparfenov commented Oct 31, 2023 • edited Loading

lcapillera commented Oct 31, 2023

lcapillera commented Oct 31, 2023

vasyaxparfenov commented Oct 31, 2023 • edited Loading

vasyaxparfenov commented Oct 31, 2023 • edited Loading

lcapillera commented Nov 1, 2023

vasyaxparfenov commented Nov 1, 2023 • edited Loading

lcapillera commented Nov 1, 2023

vasyaxparfenov commented Nov 1, 2023 • edited Loading

prudhvigodithi commented Nov 2, 2023

prudhvigodithi commented Nov 2, 2023

vasyaxparfenov commented Nov 4, 2023 • edited Loading

prudhvigodithi commented Nov 6, 2023

vasyaxparfenov commented Nov 7, 2023

prudhvigodithi Nov 7, 2023

Choose a reason for hiding this comment

prudhvigodithi Nov 7, 2023

Choose a reason for hiding this comment

vasyaxparfenov Nov 8, 2023

Choose a reason for hiding this comment

prudhvigodithi commented Nov 8, 2023

vasyaxparfenov commented Oct 9, 2023 •

edited

Loading

vasyaxparfenov commented Oct 9, 2023 •

edited

Loading

vasyaxparfenov commented Oct 31, 2023 •

edited

Loading

vasyaxparfenov commented Oct 31, 2023 •

edited

Loading

vasyaxparfenov commented Oct 31, 2023 •

edited

Loading

vasyaxparfenov commented Oct 31, 2023 •

edited

Loading

vasyaxparfenov commented Nov 1, 2023 •

edited

Loading

vasyaxparfenov commented Nov 1, 2023 •

edited

Loading

vasyaxparfenov commented Nov 4, 2023 •

edited

Loading