Moving concurrent-search out of sandbox plugin to core behind feature flag

[sohami]

Description

Moving concurrent-search out of sandbox plugin to server module behind a feature flag

Issues Resolved

#6859

Check List

• New functionality includes testing.
  • All tests pass
• New functionality has been documented.
  • New functionality has javadoc added
• Commits are signed per the DCO using --signoff
• Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[sohami]

@reta @nknize Please help with review when you get chance

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/14174/
• CommitID: b775e88
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: UNSTABLE ❕
• TEST FAILURES:

      1 org.opensearch.indices.replication.SegmentReplicationRelocationIT.testRelocateWhileContinuouslyIndexingAndWaitingForRefresh
      1 org.opensearch.index.reindex.ReindexBasicTests.classMethod

• URL: https://build.ci.opensearch.org/job/gradle-check/14175/
• CommitID: b02f44c
  Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[sohami]

The test failures seems unrelated to this PR:

1. org.opensearch.index.reindex.ReindexBasicTests.classMethod - Looks like some race in threadpool getting shutdown and task being submitted for fail_stale_replica. Similar stack is seen here

 thg 4 18, 2023 3:10:50 SA org.apache.lucene.store.MemorySegmentIndexInputProvider <init>
INFO: Using MemorySegmentIndexInput with Java 19; to disable start with -Dorg.apache.lucene.store.MMapDirectory.enableMemorySegments=false
thg 4 18, 2023 3:11:27 SA com.carrotsearch.randomizedtesting.RandomizedRunner$QueueUncaughtExceptionsHandler uncaughtException
WARNING: Uncaught exception in thread: Thread[#196,opensearch[node_s1][generic][T#6],5,TGRP-ReindexBasicTests]
org.opensearch.core.concurrency.OpenSearchRejectedExecutionException: rejected execution of java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@1a874b8f[Not completed, task = java.util.concurrent.Executors$RunnableAdapter@1a8a24ec[Wrapped task = [threaded] fail_stale_replica]] on org.opensearch.threadpool.Scheduler$SafeScheduledThreadPoolExecutor@2dd992da[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 191]
	at __randomizedtesting.SeedInfo.seed([1060866F1479C3B8]:0)
	at org.opensearch.common.util.concurrent.OpenSearchAbortPolicy.rejectedExecution(OpenSearchAbortPolicy.java:67)
	at java.base/java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:841)
	at java.base/java.util.concurrent.ScheduledThreadPoolExecutor.delayedExecute(ScheduledThreadPoolExecutor.java:340)
	at java.base/java.util.concurrent.ScheduledThreadPoolExecutor.schedule(ScheduledThreadPoolExecutor.java:562)
	at org.opensearch.threadpool.ThreadPool.schedule(ThreadPool.java:424)
	at org.opensearch.common.util.concurrent.AbstractAsyncTask.rescheduleIfNecessary(AbstractAsyncTask.java:109)
	at org.opensearch.common.util.concurrent.AbstractAsyncTask.run(AbstractAsyncTask.java:174)
	at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:747)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
	at java.base/java.lang.Thread.run(Thread.java:1589)

REPRODUCE WITH: ./gradlew ':modules:reindex:test' --tests "org.opensearch.index.reindex.ReindexBasicTests" -Dtests.seed=1060866F1479C3B8 -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=en -Dtests.timezone=Etc/UTC -Druntime.java=19

2. org.opensearch.indices.replication.SegmentReplicationRelocationIT.testRelocateWhileContinuouslyIndexingAndWaitingForRefresh - The search failed as the index seems to be deleted as part of test cleanup. The second run was successful. There was an existing issue for this flaky test which was recently closed as it was not reproducible. I tried locally as well for 100 times and it is passing all the time.

Stacktrace
Failed to execute phase [query], all shards failed
	at __randomizedtesting.SeedInfo.seed([1060866F1479C3B8:9C145E582A6F6340]:0)
	at app//org.opensearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:664)
	at app//org.opensearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:372)
	at app//org.opensearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:699)
	at app//org.opensearch.action.search.AbstractSearchAsyncAction.onShardFailure(AbstractSearchAsyncAction.java:472)
	at app//org.opensearch.action.search.AbstractSearchAsyncAction.lambda$performPhaseOnShard$0(AbstractSearchAsyncAction.java:273)
	at app//org.opensearch.action.search.AbstractSearchAsyncAction$2.doRun(AbstractSearchAsyncAction.java:350)
	at app//org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52)
	at app//org.opensearch.threadpool.TaskAwareRunnable.doRun(TaskAwareRunnable.java:78)
	at app//org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52)
	at app//org.opensearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:59)
	at app//org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:806)
	at app//org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52)
	at java.base@19.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
	at java.base@19.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
	at java.base@19.0.2/java.lang.Thread.run(Thread.java:1589)

REPRODUCE WITH: ./gradlew ':server:internalClusterTest' --tests "org.opensearch.indices.replication.SegmentReplicationRelocationIT.testRelocateWhileContinuouslyIndexingAndWaitingForRefresh" -Dtests.seed=1060866F1479C3B8 -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=sr-Latn-ME -Dtests.timezone=Pacific/Fiji -Druntime.java=19

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: UNSTABLE ❕
• TEST FAILURES:

      1 org.opensearch.cluster.allocation.AwarenessAllocationIT.testThreeZoneOneReplicaWithForceZoneValueAndLoadAwareness

• URL: https://build.ci.opensearch.org/job/gradle-check/14206/
• CommitID: a7c8dfe
  Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[kotwanikunal]

@sohami LGTM. Can you please rebase and resolve the conflicts?
Also tagging @nknize

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/14822/
• CommitID: 155c977
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[kotwanikunal]

REPRODUCE WITH: ./gradlew ':server:internalClusterTest' --tests "org.opensearch.action.admin.cluster.stats.ClusterStatsIT.testNodeRolesWithDataNodeLegacySettings" -Dtests.seed=9C131CE66EBDF984 -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=ar-TN -Dtests.timezone=Africa/Kinshasa -Druntime.java=19

org.opensearch.action.admin.cluster.stats.ClusterStatsIT > testNodeRolesWithDataNodeLegacySettings FAILED
    java.lang.IllegalArgumentException: duplicate element: [cluster_manager]
        at __randomizedtesting.SeedInfo.seed([9C131CE66EBDF984:B56ED209C37D3942]:0)
        at java.****/java.util.ImmutableCollections$Set12.<init>(ImmutableCollections.java:791)
        at java.****/java.util.Set.of(Set.java:487)
        at org.opensearch.action.admin.cluster.stats.ClusterStatsIT.testNodeRolesWithDataNodeLegacySettings(ClusterStatsIT.java:415)

Strange! Seems like #7327 again.
@sohami Did you rebase with the latest main? I thought I fixed the issue with #7395

[sohami]

REPRODUCE WITH: ./gradlew ':server:internalClusterTest' --tests "org.opensearch.action.admin.cluster.stats.ClusterStatsIT.testNodeRolesWithDataNodeLegacySettings" -Dtests.seed=9C131CE66EBDF984 -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=ar-TN -Dtests.timezone=Africa/Kinshasa -Druntime.java=19

org.opensearch.action.admin.cluster.stats.ClusterStatsIT > testNodeRolesWithDataNodeLegacySettings FAILED
    java.lang.IllegalArgumentException: duplicate element: [cluster_manager]
        at __randomizedtesting.SeedInfo.seed([9C131CE66EBDF984:B56ED209C37D3942]:0)
        at java.****/java.util.ImmutableCollections$Set12.<init>(ImmutableCollections.java:791)
        at java.****/java.util.Set.of(Set.java:487)
        at org.opensearch.action.admin.cluster.stats.ClusterStatsIT.testNodeRolesWithDataNodeLegacySettings(ClusterStatsIT.java:415)

Strange! Seems like #7327 again. @sohami Did you rebase with the latest main? I thought I fixed the issue with #7395

I rebased 2 hours back and these commits were made an hour back so its not there. Rebased again

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: SUCCESS ✅
• URL: https://build.ci.opensearch.org/job/gradle-check/14826/
• CommitID: 7b6a722

[opensearch-trigger-bot]

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/backport-2.x 2.x
# Navigate to the new working tree
pushd ../.worktrees/backport-2.x
# Create a new branch
git switch --create backport/backport-7203-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 fde8fa47a14b4dbb17a670604ab0ff357eeb5b95
# Push it to GitHub
git push --set-upstream origin backport/backport-7203-to-2.x
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-7203-to-2.x.

[kotwanikunal]

Backport PR: #7408