[BUG] ISM index rollover actions fail because of missing setting for otel-v1-apm-span-* indices

[JannikBrand]

Describe the bug
In the Trace Analytics OpenSearch setup, index rollovers for the otel-v1-apm-span-* indices are not working with Data Prepper's official index template.

To Reproduce

• Prerequisite is a running OpenSearch cluster.
• Configure Data Prepper for the trace analytics use case. At least the raw-pipeline should be connected to an OpenSearch Sink.
• When Data Prepper has been started and is running it initializes the OpenSearch sink. In this step it creates the otel-v1-apm-span-000001 index and an ISM policy raw-span-policy which is targeting those kind of indices.
• In OpenSearch Index Management -> Managed Indices you should see this index. It takes some time until the ISM policy is initialized for this index.
• When the first index rollover operation is performed by the ISM policy (after 24h), this operation fails with the error "message": "Missing rollover_alias index setting [index=otel-v1-apm-span-000001]" and after the retry operations were performed the Job Status will also be "failed". (In order to test this within multiple minutes you can manually change the raw-span-policy to perform the rollover after e.g. 3 minutes. Do not forget that policies don't get applied to existing indices, so you would either need to delete the otel-v1-apm-span-000001 index and restart Data Prepper or apply this policy to the existing index.)

Expected behavior
Rollover actions from the ISM policy should not fail. According to this OpenSearch documentation the error message is clear and you need to add the "plugins.index_state_management.rollover_alias": "otel-v1-apm-span" index setting to the otel-v1-apm-span-index-template. I tested this for our setup and this solved the problem.

I know that the rollover actions worked in previous setups (older Data Prepper and OpenSearch versions) even though this index setting was not set.I could not figure out the reason so far. Also, I would assume that this is a problem also for newer Data Prepper / OpenSearch versions (compared to our setup) because I do not see that this index setting is configured somewhere.

Environment (please complete the following information):

• Data Prepper version: 2.3.2
• OpenSearch version: 1.3.12

[dlvenable]

@JannikBrand , Thank you for reporting this issue. We have had some significant changes to the Data Prepper code for managing indexes. It appears that they introduced a regression. Would you able to contribute toward a PR on this?

It may be good to start by creating an integration test in OpenSearchIT which reproduces this issue.

[JannikBrand]

Hi @dlvenable, I can provide a PR, however I am not immediately able to work on it.
I've just created the draft PR #3580 to show how I think the solution would look like after having a brief look.

I will have a look in the integration tests, because to reproduce it we would need a rollover action performed by ISM policy. A manual rollover command will not do it.

[dlvenable]

@JannikBrand , Thank you for the PR. I was working to try to reproduce this yesterday and been unable to reproduce it. I tested this against the current Data Prepper on main (unreleased) along with the latest published OpenSearch version. I changed one file which was to update the rollover time to 2 minutes. In this setup the indexes were able to rollover without any issue.

I also tried it on Data Prepper 2.3.2 and OpenSearch 1.13.2 as you reported above. Because I can't make changes to this version, I made a manual change to update the index after the fact as you suggested. Again, I'm seeing the rollover complete successfully.

Data Prepper does set the rollover_alias using the deprecated opendistro value so that it can work with OpenSearch and OpenDistro distributions. The code that sets it is here:

data-prepper/data-prepper-plugins/opensearch/src/main/java/org/opensearch/dataprepper/plugins/sink/opensearch/index/AbstractIndexManager.java

Line 309 in 73a80a1

indexTemplate.putCustomSetting(IndexConstants.ISM_ROLLOVER_ALIAS_SETTING, rolloverAlias);

And here you can see the constant we use:

data-prepper/data-prepper-plugins/opensearch/src/main/java/org/opensearch/dataprepper/plugins/sink/opensearch/index/IndexConstants.java

Line 23 in 48d8bc3

public static final String ISM_ROLLOVER_ALIAS_SETTING = "opendistro.index_state_management.rollover_alias";

Indeed, I can confirm the settings.

GET _template/otel-v1-apm*

{
  "otel-v1-apm-span-index-template" : {
    "order" : 0,
    "version" : 1,
    "index_patterns" : [
      "otel-v1-apm-span-*"
    ],
    "settings" : {
      "index" : {
        "opendistro" : {
          "index_state_management" : {
            "rollover_alias" : "otel-v1-apm-span"
          }
        }
      }
    },
...

And the index has them as expected.

GET otel-v1-apm-span-000001/_settings

{
  "otel-v1-apm-span-000001" : {
    "settings" : {
      "index" : {
        "opendistro" : {
          "index_state_management" : {
            "rollover_alias" : "otel-v1-apm-span"
          }
        },
        "number_of_shards" : "1",
        "provided_name" : "otel-v1-apm-span-000001",
        "creation_date" : "1698939403541",
        "number_of_replicas" : "1",
        "uuid" : "y926m1IDQBuMwZdr_k2Lpw",
        "version" : {
          "created" : "135249027"
        }
      }
    }
  }
}

And after a rollover:

GET otel-v1-apm-span-000002/_settings

{
  "otel-v1-apm-span-000002" : {
    "settings" : {
      "index" : {
        "opendistro" : {
          "index_state_management" : {
            "rollover_alias" : "otel-v1-apm-span"
          }
        },
        "number_of_shards" : "1",
        "provided_name" : "otel-v1-apm-span-000002",
        "creation_date" : "1698940724097",
        "number_of_replicas" : "1",
        "uuid" : "DLi67FZOTw6X_U9oDA7N4A",
        "version" : {
          "created" : "135249027"
        }
      }
    }
  }
}

[dlvenable]

@JannikBrand , I have been able to reproduce this issue by setting template_type: index-template on the opensearch sink. It does appear that Data Prepper is not setting this value when using the composable index templates.

[JannikBrand]

@dlvenable, thanks for looking into this!
This also answers the question why it worked in a previous setup for us, were we used legacy templates most likely.
Thanks for pointing to the deprecated opendistro rollover_alias setting, I was just looking for the new one. Would it make sense to set both settings simultaneously, in case the deprecated one stops working for a future OpenSearch version? Don't think a test case would discover the problem so far.
I will also close my PR as yours will solve the issue.

[dlvenable]

@JannikBrand ,

I'd like to update Data Prepper to detect the OpenSearch version and then decide which setting to use. This came up in a discussion in another PR:

#3560 (comment)

I'm not sure if setting both will have an impact or not on how ISM works.