[BUG] Test case org.opensearch.indices.IndicesRequestCacheIT.testDeleteAndCreateSameIndexShardOnSameNode {p0={"search.concurrent_segment_search.enabled":"false"}} is flaky #13600

reta · 2024-05-08T01:48:02Z

Describe the bug

The test case org.opensearch.indices.IndicesRequestCacheIT.testDeleteAndCreateSameIndexShardOnSameNode {p0={"search.concurrent_segment_search.enabled":"false"}} is flaky:

java.lang.AssertionError
	at __randomizedtesting.SeedInfo.seed([B565C22BDAF29AC0:8BD7562BB0AD6227]:0)
	at org.opensearch.indices.IndicesRequestCacheIT.shardDirectory(IndicesRequestCacheIT.java:1343)
	at org.opensearch.indices.IndicesRequestCacheIT.lambda$testDeleteAndCreateSameIndexShardOnSameNode$13(IndicesRequestCacheIT.java:1309)
	at org.opensearch.test.OpenSearchTestCase.assertBusy(OpenSearchTestCase.java:1128)
	at org.opensearch.test.OpenSearchTestCase.assertBusy(OpenSearchTestCase.java:1101)
	at org.opensearch.indices.IndicesRequestCacheIT.testDeleteAndCreateSameIndexShardOnSameNode(IndicesRequestCacheIT.java:1308)
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
	at java.base/java.lang.reflect.Method.invoke(Method.java:580)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
	at org.opensearch.test.OpenSearchTestClusterRule$1.evaluate(OpenSearchTestClusterRule.java:369)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
	at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
	at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
	at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
	at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.base/java.lang.Thread.run(Thread.java:1583)
	Suppressed: java.lang.AssertionError
		at org.opensearch.indices.IndicesRequestCacheIT.shardDirectory(IndicesRequestCacheIT.java:1343)
		at org.opensearch.indices.IndicesRequestCacheIT.lambda$testDeleteAndCreateSameIndexShardOnSameNode$13(IndicesRequestCacheIT.java:1309)
		at org.opensearch.test.OpenSearchTestCase.assertBusy(OpenSearchTestCase.java:1116)
		... 41 more
	Suppressed: java.lang.AssertionError
		at org.opensearch.indices.IndicesRequestCacheIT.shardDirectory(IndicesRequestCacheIT.java:1343)
		at org.opensearch.indices.IndicesRequestCacheIT.lambda$testDeleteAndCreateSameIndexShardOnSameNode$13(IndicesRequestCacheIT.java:1309)
		at org.opensearch.test.OpenSearchTestCase.assertBusy(OpenSearchTestCase.java:1116)
		... 41 more
	Suppressed: java.lang.AssertionError
		at org.opensearch.indices.IndicesRequestCacheIT.shardDirectory(IndicesRequestCacheIT.java:1343)
		at org.opensearch.indices.IndicesRequestCacheIT.lambda$testDeleteAndCreateSameIndexShardOnSameNode$13(IndicesRequestCacheIT.java:1309)
		at org.opensearch.test.OpenSearchTestCase.assertBusy(OpenSearchTestCase.java:1116)
		... 41 more
	Suppressed: java.lang.AssertionError
		at org.opensearch.indices.IndicesRequestCacheIT.shardDirectory(IndicesRequestCacheIT.java:1343)
		at org.opensearch.indices.IndicesRequestCacheIT.lambda$testDeleteAndCreateSameIndexShardOnSameNode$13(IndicesRequestCacheIT.java:1309)
		at org.opensearch.test.OpenSearchTestCase.assertBusy(OpenSearchTestCase.java:1116)
		... 41 more
	Suppressed: java.lang.AssertionError
		at org.opensearch.indices.IndicesRequestCacheIT.shardDirectory(IndicesRequestCacheIT.java:1343)
		at org.opensearch.indices.IndicesRequestCacheIT.lambda$testDeleteAndCreateSameIndexShardOnSameNode$13(IndicesRequestCacheIT.java:1309)
		at org.opensearch.test.OpenSearchTestCase.assertBusy(OpenSearchTestCase.java:1116)
		... 41 more
	Suppressed: java.lang.AssertionError
		at org.opensearch.indices.IndicesRequestCacheIT.shardDirectory(IndicesRequestCacheIT.java:1343)
		at org.opensearch.indices.IndicesRequestCacheIT.lambda$testDeleteAndCreateSameIndexShardOnSameNode$13(IndicesRequestCacheIT.java:1309)
		at org.opensearch.test.OpenSearchTestCase.assertBusy(OpenSearchTestCase.java:1116)
		... 41 more
	Suppressed: java.lang.AssertionError
		at org.opensearch.indices.IndicesRequestCacheIT.shardDirectory(IndicesRequestCacheIT.java:1343)
		at org.opensearch.indices.IndicesRequestCacheIT.lambda$testDeleteAndCreateSameIndexShardOnSameNode$13(IndicesRequestCacheIT.java:1309)
		at org.opensearch.test.OpenSearchTestCase.assertBusy(OpenSearchTestCase.java:1116)
		... 41 more
	Suppressed: java.lang.AssertionError
		at org.opensearch.indices.IndicesRequestCacheIT.shardDirectory(IndicesRequestCacheIT.java:1343)
		at org.opensearch.indices.IndicesRequestCacheIT.lambda$testDeleteAndCreateSameIndexShardOnSameNode$13(IndicesRequestCacheIT.java:1309)
		at org.opensearch.test.OpenSearchTestCase.assertBusy(OpenSearchTestCase.java:1116)
		... 41 more
	Suppressed: java.lang.AssertionError
		at org.opensearch.indices.IndicesRequestCacheIT.shardDirectory(IndicesRequestCacheIT.java:1343)
		at org.opensearch.indices.IndicesRequestCacheIT.lambda$testDeleteAndCreateSameIndexShardOnSameNode$13(IndicesRequestCacheIT.java:1309)
		at org.opensearch.test.OpenSearchTestCase.assertBusy(OpenSearchTestCase.java:1116)
		... 41 more
	Suppressed: java.lang.AssertionError
		at org.opensearch.indices.IndicesRequestCacheIT.shardDirectory(IndicesRequestCacheIT.java:1343)
		at org.opensearch.indices.IndicesRequestCacheIT.lambda$testDeleteAndCreateSameIndexShardOnSameNode$13(IndicesRequestCacheIT.java:1309)
		at org.opensearch.test.OpenSearchTestCase.assertBusy(OpenSearchTestCase.java:1116)
		... 41 more
	Suppressed: java.lang.AssertionError
		at org.opensearch.indices.IndicesRequestCacheIT.shardDirectory(IndicesRequestCacheIT.java:1343)
		at org.opensearch.indices.IndicesRequestCacheIT.lambda$testDeleteAndCreateSameIndexShardOnSameNode$13(IndicesRequestCacheIT.java:1309)
		at org.opensearch.test.OpenSearchTestCase.assertBusy(OpenSearchTestCase.java:1116)
		... 41 more
	Suppressed: java.lang.AssertionError
		at org.opensearch.indices.IndicesRequestCacheIT.shardDirectory(IndicesRequestCacheIT.java:1343)
		at org.opensearch.indices.IndicesRequestCacheIT.lambda$testDeleteAndCreateSameIndexShardOnSameNode$13(IndicesRequestCacheIT.java:1309)
		at org.opensearch.test.OpenSearchTestCase.assertBusy(OpenSearchTestCase.java:1116)
		... 41 more
	Suppressed: java.lang.AssertionError
		at org.opensearch.indices.IndicesRequestCacheIT.shardDirectory(IndicesRequestCacheIT.java:1343)
		at org.opensearch.indices.IndicesRequestCacheIT.lambda$testDeleteAndCreateSameIndexShardOnSameNode$13(IndicesRequestCacheIT.java:1309)
		at org.opensearch.test.OpenSearchTestCase.assertBusy(OpenSearchTestCase.java:1116)
		... 41 more

Related component

Search

To Reproduce

./gradlew ':server:internalClusterTest' --tests "org.opensearch.indices.IndicesRequestCacheIT" -Dtests.method="testDeleteAndCreateSameIndexShardOnSameNode {p0={"search.concurrent_segment_search.enabled":"false"}}" -Dtests.seed=B565C22BDAF29AC0

Expected behavior

The test must always pass

Additional Details

Plugins
Standard

Screenshots
If applicable, add screenshots to help explain your problem.

Host/Environment (please complete the following information):

CI

Additional context

https://build.ci.opensearch.org/job/gradle-check/38474/testReport/junit/org.opensearch.indices/IndicesRequestCacheIT/testDeleteAndCreateSameIndexShardOnSameNode__p0___search_concurrent_segment_search_enabled___false___/

The text was updated successfully, but these errors were encountered:

andrross · 2024-05-08T15:10:21Z

[Triage - attendees 1 2 3 4]
@reta Thanks for filing

kiranprakash154 · 2024-05-08T16:11:55Z

@sgup432 this came up in the Opensearch Community meeting, will this be fixed by your PR - #13600 ?

msfroh · 2024-05-13T21:32:19Z

Still failing for me: https://build.ci.opensearch.org/job/gradle-check/38635/testReport/junit/org.opensearch.indices/IndicesRequestCacheIT/testDeleteAndCreateSameIndexShardOnSameNode__p0___search_concurrent_segment_search_enabled___false___/

prudhvigodithi · 2024-05-14T17:04:22Z

FYI here are some metrics regarding the failing test Test case org.opensearch.indices.IndicesRequestCacheIT.testDeleteAndCreateSameIndexShardOnSameNode {p0={"search.concurrent_segment_search.enabled":"false"}}.

These are very frequent offenders and are causing a lot of pain in CI. Related issues: opensearch-project#11374 opensearch-project#12308 opensearch-project#13540 opensearch-project#13503 opensearch-project#13711 opensearch-project#13949 opensearch-project#13437 opensearch-project#13600 Signed-off-by: Andrew Ross <andrross@amazon.com>

These are very frequent offenders and are causing a lot of pain in CI. Related issues: - opensearch-project#11374 - opensearch-project#12308 - opensearch-project#13540 - opensearch-project#13503 - opensearch-project#13711 - opensearch-project#13949 - opensearch-project#13437 - opensearch-project#13600 Signed-off-by: Andrew Ross <andrross@amazon.com>

These are very frequent offenders and are causing a lot of pain in CI. Related issues: - #11374 - #12308 - #13540 - #13503 - #13711 - #13949 - #13437 - #13600 Signed-off-by: Andrew Ross <andrross@amazon.com>

These are very frequent offenders and are causing a lot of pain in CI. Related issues: - #11374 - #12308 - #13540 - #13503 - #13711 - #13949 - #13437 - #13600 Signed-off-by: Andrew Ross <andrross@amazon.com> (cherry picked from commit fbe048f) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

kkewwei · 2024-06-08T07:09:07Z

@sgup432 this came up in the Opensearch Community meeting, will this be fixed by your PR - #13600 ?

I encountered in the #14069, If you haven't started doing it yet, I will try to solve it.

These are very frequent offenders and are causing a lot of pain in CI. Related issues: - #11374 - #12308 - #13540 - #13503 - #13711 - #13949 - #13437 - #13600 (cherry picked from commit fbe048f) Signed-off-by: Andrew Ross <andrross@amazon.com> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

These are very frequent offenders and are causing a lot of pain in CI. Related issues: - opensearch-project#11374 - opensearch-project#12308 - opensearch-project#13540 - opensearch-project#13503 - opensearch-project#13711 - opensearch-project#13949 - opensearch-project#13437 - opensearch-project#13600 Signed-off-by: Andrew Ross <andrross@amazon.com>

…opensearch-project#14082) These are very frequent offenders and are causing a lot of pain in CI. Related issues: - opensearch-project#11374 - opensearch-project#12308 - opensearch-project#13540 - opensearch-project#13503 - opensearch-project#13711 - opensearch-project#13949 - opensearch-project#13437 - opensearch-project#13600 (cherry picked from commit fbe048f) Signed-off-by: Andrew Ross <andrross@amazon.com> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Signed-off-by: kkewwei <kkewwei@163.com>

These are very frequent offenders and are causing a lot of pain in CI. Related issues: - opensearch-project#11374 - opensearch-project#12308 - opensearch-project#13540 - opensearch-project#13503 - opensearch-project#13711 - opensearch-project#13949 - opensearch-project#13437 - opensearch-project#13600 Signed-off-by: Andrew Ross <andrross@amazon.com>

reta added bug Something isn't working untriaged flaky-test Random test failure that succeeds on second run labels May 8, 2024

github-actions bot added the Search Search query, autocomplete ...etc label May 8, 2024

github-project-automation bot added this to Search Project Board May 8, 2024

github-project-automation bot moved this to 🆕 New in Search Project Board May 8, 2024

reta mentioned this issue May 8, 2024

[AUTOCUT] Gradle Check Failure on push to 2.x #13599

Closed

andrross removed the untriaged label May 8, 2024

andrross added the Search:Performance label May 8, 2024

getsaurabh02 assigned kiranprakash154 May 8, 2024

sgup432 assigned sgup432 and unassigned kiranprakash154 May 8, 2024

rayshrey mentioned this issue May 9, 2024

[Writable Warm] Composite Directory implementation and integrating it with FileCache #12782

Merged

8 tasks

VachaShah mentioned this issue May 9, 2024

[Backport 2.x] Removing extra protobuf notice file #13617

Merged

8 tasks

reta mentioned this issue May 10, 2024

Fix get field mapping API returns 404 error in mixed cluster with multiple versions #13624

Merged

8 tasks

peternied mentioned this issue May 13, 2024

[AUTOCUT] Gradle Check Failure on push to main #13626

Closed

msfroh mentioned this issue May 13, 2024

Do not pass negative scores into function_score or script_score queries #13627

Closed

9 tasks

gaobinlong mentioned this issue May 14, 2024

Fix org.opensearch.rest.action.admin.cluster.RestNodesStatsActionTests.testIndexMetricsRequestWithoutIndicesMetric is flaky #13658

Merged

9 tasks

reta mentioned this issue May 14, 2024

Bump mockito from 5.11.0 to 5.12.0 #13665

Merged

9 tasks

finnegancarroll mentioned this issue May 14, 2024

Enhance tika document parsing tests #13618

Merged

8 tasks

reta mentioned this issue May 15, 2024

[AUTOCUT] Gradle Check Failure on push to 2.x #13685

Closed

himshikha mentioned this issue May 16, 2024

[Remote Routing Table] Initial commit for RemoteRoutingTableService setup #13304

Merged

7 tasks

reta mentioned this issue May 16, 2024

[AUTOCUT] Gradle Check Failure on push to main #13693

Closed

peternied mentioned this issue May 16, 2024

[AUTOCUT] Gradle Check Failure on push to main #13696

Closed

This was referenced May 17, 2024

Adds support to inject telemetry instances to plugins #13636

Merged

[AUTOCUT] Gradle Check Failure on push to main #13735

Closed

reta mentioned this issue Jun 4, 2024

[AUTOCUT] Gradle Check Failure on push to main #13959

Closed

akolarkunnu mentioned this issue Jun 4, 2024

COMPAT locale provider will be removed in a future release #13934

Closed

9 tasks

peteralfonsi mentioned this issue Jun 4, 2024

[Tiered Caching] Additional ITs for cache stats #13655

Merged

9 tasks

andrross mentioned this issue Jun 4, 2024

[Backport 2.x] Update gradle config for develocity rebranding (#13942) #13958

Merged

9 tasks

harishbhakuni mentioned this issue Jun 5, 2024

[Snapshot Interop][Bug Fix] Ensure lock file gets deleted if snapshot … #12016

Open

8 tasks

gaobinlong mentioned this issue Jun 5, 2024

Add fingerprint ingest processor #13724

Merged

9 tasks

akolarkunnu mentioned this issue Jun 5, 2024

COMPAT locale provider will be removed in a future release #13988

Merged

9 tasks

reta mentioned this issue Jun 5, 2024

[Backport 2.x] 13776: allow adding query parameters to RequestOptions #13778

Merged

9 tasks

shiv0408 mentioned this issue Jun 5, 2024

[AUTOCUT] Gradle Check Failure on push to 2.x #14005

Closed

harshavamsi mentioned this issue Jun 6, 2024

Set INDICES_MAX_CLAUSE_COUNT dynamically #13568

Merged

8 tasks

shourya035 mentioned this issue Jun 6, 2024

[Remote Store] Add checks to skip remote uploads after shard is closed #13998

Merged

9 tasks

akolarkunnu mentioned this issue Jun 6, 2024

A documentation issue, used CONTRIBUTING.md instead of DEVELOPER_GUIDE.md #14028

Merged

3 tasks

reta mentioned this issue Jun 6, 2024

[Backport 2.x] Bump HdrHistogram to 2.2.2 and move the dependency version to version.properties #14040

Merged

3 tasks

andrross mentioned this issue Jun 7, 2024

[Backport 2.x] [DerivedField] object type support in mappings and new settings for derived field #14050

Merged

andrross mentioned this issue Jun 7, 2024

Mute all flaky IndicesRequestCacheIT tests #14077

Merged

3 tasks

This was referenced Jun 8, 2024

Fix null values indexed as "null" strings in flat_object field #14069

Merged

Fix flaky test IndicesRequestCacheIT.testDeleteAndCreateSameIndexShardOnSameNode #14090

Merged

dblock closed this as completed in #14090 Jun 10, 2024

github-project-automation bot moved this from In Progress to Done in Performance Roadmap Jun 10, 2024

github-project-automation bot moved this from 🆕 New to ✅ Done in Search Project Board Jun 10, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[BUG] Test case org.opensearch.indices.IndicesRequestCacheIT.testDeleteAndCreateSameIndexShardOnSameNode {p0={"search.concurrent_segment_search.enabled":"false"}} is flaky #13600

[BUG] Test case org.opensearch.indices.IndicesRequestCacheIT.testDeleteAndCreateSameIndexShardOnSameNode {p0={"search.concurrent_segment_search.enabled":"false"}} is flaky #13600

reta commented May 8, 2024

andrross commented May 8, 2024

kiranprakash154 commented May 8, 2024

msfroh commented May 13, 2024

prudhvigodithi commented May 14, 2024

kkewwei commented Jun 8, 2024

[BUG] Test case org.opensearch.indices.IndicesRequestCacheIT.testDeleteAndCreateSameIndexShardOnSameNode {p0={"search.concurrent_segment_search.enabled":"false"}} is flaky #13600

[BUG] Test case org.opensearch.indices.IndicesRequestCacheIT.testDeleteAndCreateSameIndexShardOnSameNode {p0={"search.concurrent_segment_search.enabled":"false"}} is flaky #13600

Comments

reta commented May 8, 2024

Describe the bug

Related component

To Reproduce

Expected behavior

Additional Details

andrross commented May 8, 2024

kiranprakash154 commented May 8, 2024

msfroh commented May 13, 2024

prudhvigodithi commented May 14, 2024

kkewwei commented Jun 8, 2024