Opensearch: [Segment Replication] Add new Test manifest to test all plugin compatibility with Segment Replication enabled. #3345

Rishikesh1159 · 2023-03-23T21:52:18Z

Is your feature request related to a problem? Please describe

Coming from: opensearch-project/OpenSearch#6602

Leading up to GA of segment replication in opensearch 2.7, we need to do a round of sanity testing with plugins.

To do this testing we need a new test-manifest exactly same as https://github.com/opensearch-project/opensearch-build/blob/main/manifests/2.7.0/opensearch-2.7.0-test.yml but with below values added to each plugin:

additional-cluster-configs:
   opensearch.experimental.feature.replication_type.enabled: true
   cluster.indices.replication.strategy: "SEGMENT"

Describe the solution you'd like

To enable segment replication by default, we need to set feature flag and a cluster setting. Both of these must be passed/set from opensearch.yml

-> To enable segment replication feature flag, we have to pass:
opensearch.experimental.feature.replication_type.enabled: true in opensearch.yml. (4959).
-> To set cluster setting of replication type, we have to pass: cluster.indices.replication.strategy: "SEGMENT" in opensearch.yml. (6791).

Describe alternatives you've considered

No response

Additional context

No response

The text was updated successfully, but these errors were encountered:

gaiksaya · 2023-03-23T22:07:25Z

Hi @Rishikesh1159,

I believe we want to keep the actual integration testing separate from seg-Rep feature testing.
We can add one more test-manifest exactly same as https://github.com/opensearch-project/opensearch-build/blob/main/manifests/2.7.0/opensearch-2.7.0-test.yml but with below values added to each plugin:

additional-cluster-configs:
   opensearch.experimental.feature.replication_type.enabled: true
   cluster.indices.replication.strategy: "SEGMENT"

This will add above values to opensearch.yml to each cluster that is deployed for testing. I assume this is what we want?
cc: @bbarani

Rishikesh1159 · 2023-03-23T22:21:46Z

Hi @gaiksaya, yes you are correct. Sorry for confusing description, the request here is to add one more test-manifest to add those values to opensearch.yml. Let me change the description of this issue to make it clear

gaiksaya · 2023-03-23T23:02:26Z

Thanks! So I tried adding the running with above configuration and looks like cluster is unable to start.

Error

[2023-03-23T15:56:40,947][ERROR][o.o.b.OpenSearchUncaughtExceptionHandler] [node_name_9200] uncaught exception in thread [main]
org.opensearch.bootstrap.StartupException: java.lang.IllegalArgumentException: unknown setting [cluster.indices.replication.strategy] please check that any required plugins are installed, or check the breaking changes documentation for removed settings
	at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:184) ~[opensearch-2.7.0.jar:2.7.0]
	at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:171) ~[opensearch-2.7.0.jar:2.7.0]
	at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:104) ~[opensearch-2.7.0.jar:2.7.0]
	at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138) ~[opensearch-cli-2.7.0.jar:2.7.0]
	at org.opensearch.cli.Command.main(Command.java:101) ~[opensearch-cli-2.7.0.jar:2.7.0]
	at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:137) ~[opensearch-2.7.0.jar:2.7.0]
	at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:103) ~[opensearch-2.7.0.jar:2.7.0]
Caused by: java.lang.IllegalArgumentException: unknown setting [cluster.indices.replication.strategy] please check that any required plugins are installed, or check the breaking changes documentation for removed settings
	at org.opensearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:608) ~[opensearch-2.7.0.jar:2.7.0]
	at org.opensearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:549) ~[opensearch-2.7.0.jar:2.7.0]
	at org.opensearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:519) ~[opensearch-2.7.0.jar:2.7.0]
	at org.opensearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:489) ~[opensearch-2.7.0.jar:2.7.0]
	at org.opensearch.common.settings.SettingsModule.<init>(SettingsModule.java:179) ~[opensearch-2.7.0.jar:2.7.0]
	at org.opensearch.node.Node.<init>(Node.java:514) ~[opensearch-2.7.0.jar:2.7.0]
	at org.opensearch.node.Node.<init>(Node.java:361) ~[opensearch-2.7.0.jar:2.7.0]
	at org.opensearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:242) ~[opensearch-2.7.0.jar:2.7.0]
	at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:242) ~[opensearch-2.7.0.jar:2.7.0]
	at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404) ~[opensearch-2.7.0.jar:2.7.0]
	at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:180) ~[opensearch-2.7.0.jar:2.7.0]
	... 6 more

Manifest with segRep:

---
schema-version: '1.0'
name: OpenSearch
ci:
  image:
    name: opensearchstaging/ci-runner:ci-runner-centos7-opensearch-build-v2
    args: -e JAVA_HOME=/opt/java/openjdk-17
components:
  - name: ml-commons
    integ-test:
      test-configs:
        - with-security
        - without-security
      additional-cluster-configs:
        opensearch.experimental.feature.replication_type.enabled: true
        cluster.indices.replication.strategy: "SEGMENT"

Steps to reproduce the issue:

Clone opensearch-build repository
Get the last successful build that you want to run integration test against from https://build.ci.opensearch.org/job/distribution-build-opensearch/
Run below command:

./test.sh integ-test manifests/2.7.0/opensearch-2.7.0-test-segRep.yml --component ml-commons -p opensearch=https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/2.7.0/7320/linux/x64/tar --keep

Make sure you pas --keep to keep the temp folders to check logs

Rishikesh1159 · 2023-03-24T05:40:36Z

Thanks @gaiksaya for detailed info on steps to reproduce. Looks like my changes in PR (new cluster setting) are not present in build 7320 build. My changes merged to 2.x branch after this build. So, inorder to set this cluster setting we need a new successful build. Currently the latest successful build for 2.7 is still 7320 build, so we need to wait until new successful build of 2.7 branch happens

gaiksaya · 2023-03-24T17:00:23Z

Triggered a new build: https://build.ci.opensearch.org/job/distribution-build-opensearch/7325/

gaiksaya · 2023-03-24T18:38:39Z

The build was successful and this build does contain the commit you mentioned. Details can be seen in distribution manifest: https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/2.7.0/7325/linux/x64/tar/dist/opensearch/manifest.yml

  - name: OpenSearch
    repository: https://github.com/opensearch-project/OpenSearch.git
    ref: 2.x
    commit_id: b33f7352f98971a6293a4f7e966f78745e9b2bb0

You can use above steps to reproduce with new build to test if cluster is able to start. Thanks!

Rishikesh1159 · 2023-03-24T19:00:29Z

Thanks @gaiksaya. I used latest successful build to run the test-manifest with segrep enabled as you mentioned above. I ran the test using the command you suggested above:
./test.sh integ-test manifests/2.7.0/opensearch-2.7.0-test-segRep.yml --component ml-commons -p opensearch=https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/2.7.0/7325/linux/x64/tar --keep
and my test manifest is :

---
schema-version: '1.0'
name: OpenSearch
ci:
  image:
    name: opensearchstaging/ci-runner:ci-runner-centos7-opensearch-build-v2
    args: -e JAVA_HOME=/opt/java/openjdk-17
components:
  - name: ml-commons
    integ-test:
      test-configs:
        - with-security
        - without-security
      additional-cluster-configs:
        opensearch.experimental.feature.replication_type.enabled: true
        cluster.indices.replication.strategy: "SEGMENT"

the test passed successfully

gaiksaya · 2023-03-24T22:29:00Z

Triggered a test with segment replication enabled https://build.ci.opensearch.org/blue/organizations/jenkins/integ-test/detail/integ-test/3979/pipeline

Rishikesh1159 added enhancement New Enhancement untriaged Issues that have not yet been triaged labels Mar 23, 2023

gaiksaya removed the untriaged Issues that have not yet been triaged label Mar 23, 2023

Rishikesh1159 changed the title ~~Opensearch: [Segment Replication] Test all plugin compatibility with Segment Replication enabled.~~ Opensearch: [Segment Replication] Add new Test manifest to test all plugin compatibility with Segment Replication enabled. Mar 23, 2023

gaiksaya mentioned this issue Mar 24, 2023

Add additional test manifest with segment replication enabled #3346

Merged

gaiksaya self-assigned this Mar 24, 2023

gaiksaya added this to OpenSearch Engineering Effectiveness Mar 24, 2023

github-project-automation bot moved this to Backlog in OpenSearch Engineering Effectiveness Mar 24, 2023

gaiksaya moved this from Backlog to In review in OpenSearch Engineering Effectiveness Mar 24, 2023

gaiksaya closed this as completed in #3346 Mar 24, 2023

github-project-automation bot moved this from In review to Done in OpenSearch Engineering Effectiveness Mar 24, 2023

Rishikesh1159 mentioned this issue Mar 27, 2023

[Segment Replication] Test all plugin compatibility opensearch-project/OpenSearch#6602

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Opensearch: [Segment Replication] Add new Test manifest to test all plugin compatibility with Segment Replication enabled. #3345

Opensearch: [Segment Replication] Add new Test manifest to test all plugin compatibility with Segment Replication enabled. #3345

Rishikesh1159 commented Mar 23, 2023 •

edited

Loading

gaiksaya commented Mar 23, 2023 •

edited

Loading

Rishikesh1159 commented Mar 23, 2023 •

edited

Loading

gaiksaya commented Mar 23, 2023 •

edited

Loading

Rishikesh1159 commented Mar 24, 2023 •

edited

Loading

gaiksaya commented Mar 24, 2023

gaiksaya commented Mar 24, 2023

Rishikesh1159 commented Mar 24, 2023

gaiksaya commented Mar 24, 2023

Opensearch: [Segment Replication] Add new Test manifest to test all plugin compatibility with Segment Replication enabled. #3345

Opensearch: [Segment Replication] Add new Test manifest to test all plugin compatibility with Segment Replication enabled. #3345

Comments

Rishikesh1159 commented Mar 23, 2023 • edited Loading

Is your feature request related to a problem? Please describe

Describe the solution you'd like

Describe alternatives you've considered

Additional context

gaiksaya commented Mar 23, 2023 • edited Loading

Rishikesh1159 commented Mar 23, 2023 • edited Loading

gaiksaya commented Mar 23, 2023 • edited Loading

Manifest with segRep:

Steps to reproduce the issue:

Rishikesh1159 commented Mar 24, 2023 • edited Loading

gaiksaya commented Mar 24, 2023

gaiksaya commented Mar 24, 2023

Rishikesh1159 commented Mar 24, 2023

gaiksaya commented Mar 24, 2023

Rishikesh1159 commented Mar 23, 2023 •

edited

Loading

gaiksaya commented Mar 23, 2023 •

edited

Loading

Rishikesh1159 commented Mar 23, 2023 •

edited

Loading

gaiksaya commented Mar 23, 2023 •

edited

Loading

Rishikesh1159 commented Mar 24, 2023 •

edited

Loading