[BUG] org.opensearch.search.SearchCancellationIT.testMSearchChildRequestCancellationWithClusterLevelTimeout

[dreamer-89]

Below test failure identified in #2238

**REPRODUCE WITH: ./gradlew ':server:internalClusterTest' --tests "org.opensearch.search.SearchCancellationIT.testMSearchChildRequestCancellationWithClusterLevelTimeout" -Dtests.seed=36921F5835FAE3A1 -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=en-IN -Dtests.timezone=Etc/GMT+5 -Druntime.java=17

org.opensearch.search.SearchCancellationIT > testMSearchChildRequestCancellationWithClusterLevelTimeout FAILED
    java.lang.AssertionError: Actual child request with cancellation failure is different that expected expected:<[0, 1]> but was:<[]>
        at __randomizedtesting.SeedInfo.seed([36921F5835FAE3A1:882FD4BDA4EC03D6]:0)
        at org.junit.Assert.fail(Assert.java:89)
        at org.junit.Assert.failNotEquals(Assert.java:835)
        at org.junit.Assert.assertEquals(Assert.java:120)
        at org.opensearch.search.SearchCancellationIT.ensureMSearchWasCancelled(SearchCancellationIT.java:194)
        at org.opensearch.search.SearchCancellationIT.testMSearchChildRequestCancellationWithClusterLevelTimeout(SearchCancellationIT.java:537)

[dreamer-89]

❌ Gradle Check failure 9e413b4
Log 2777

Reports 2777

[dreamer-89]

The test is not reproducible when tried with provided reproduce gradle command.

[saratvemulapalli]

#2541 (comment)

[owaiskazi19]

Ran this test in isolation for 400 times and wasn't able to reproduce it.

 ./gradlew ':server:internalClusterTest' --tests "org.opensearch.search.SearchCancellationIT.testMSearchChildRequestCancellationWithClusterLevelTimeout" -Dtests.seed=36921F5835FAE3A1 -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=en-IN -Dtests.timezone=Etc/GMT+5 -Druntime.java=17 -Dtests.iters=400
Starting a Gradle Daemon, 1 busy Daemon could not be reused, use --status for details

> Configure project :qa:os
Cannot add task 'destructiveDistroTest.docker' as a task with that name already exists.
=======================================
OpenSearch Build Hamster says Hello!
  Gradle Version        : 7.6
  OS Info               : Linux 5.11.0-1020-aws (amd64)
  Runtime JDK Version   : 17 (Oracle JDK)
  Runtime java.home     : /usr/lib/jvm/java-17/jdk-17.0.1
  Gradle JDK Version    : 17 (Oracle JDK)
  Gradle java.home      : /usr/lib/jvm/java-17/jdk-17.0.1
  Random Testing Seed   : 36921F5835FAE3A1
  In FIPS 140 mode      : false
=======================================

> Task :server:internalClusterTest
WARNING: A terminally deprecated method in java.lang.System has been called
WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.BootstrapForTesting (file:/home/ubuntu/OpenSearch/test/framework/build/distributions/framework-3.0.0-SNAPSHOT.jar)
WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.BootstrapForTesting
WARNING: System::setSecurityManager will be removed in a future release
WARNING: A terminally deprecated method in java.lang.System has been called
WARNING: System::setSecurityManager has been called by org.gradle.api.internal.tasks.testing.worker.TestWorker (file:/home/ubuntu/.gradle/wrapper/dists/gradle-7.6-all/9f832ih6bniajn45pbmqhk2cw/gradle-7.6/lib/plugins/gradle-testing-base-7.6.jar)
WARNING: Please consider reporting this to the maintainers of org.gradle.api.internal.tasks.testing.worker.TestWorker
WARNING: System::setSecurityManager will be removed in a future release

BUILD SUCCESSFUL in 20m 3s
43 actionable tasks: 1 executed, 42 up-to-date