Fix flaky test cases for DiskThresholdDeciderIT #5952

RS146BIJAY · 2023-01-20T16:23:03Z

Signed-off-by: Rishav Sagar rissag@amazon.com

Description

Test cases added for guardrail for applying index create block when all nodes are breaching high disk watermark were flaky. Reallocation on index created during integ test run was causing some race conditions scenarios. This PR fixes these test cases by disabling relocation on the indices created during integ tests.

Issues Resolved

#5956

Check List

New functionality includes testing.
- All tests pass
New functionality has been documented.
- New functionality has javadoc added
Commits are signed per the DCO using --signoff
Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

CHANGELOG.md

github-actions · 2023-01-20T16:52:38Z

Gradle Check (Jenkins) Run Completed with:

RESULT: SUCCESS ✅
URL: https://build.ci.opensearch.org/job/gradle-check/9647/
CommitID: 561f670

github-actions · 2023-01-20T17:02:14Z

Gradle Check (Jenkins) Run Completed with:

RESULT: UNSTABLE ❕
TEST FAILURES:

      1 org.opensearch.indices.replication.SegmentReplicationRelocationIT.testMultipleShards
      1 org.opensearch.cluster.routing.allocation.decider.DiskThresholdDeciderIT.testIndexCreateBlockIsRemovedWhenAnyNodesNotExceedHighWatermark

URL: https://build.ci.opensearch.org/job/gradle-check/9648/
CommitID: a60d924
Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

github-actions · 2023-01-20T18:55:35Z

Gradle Check (Jenkins) Run Completed with:

RESULT: SUCCESS ✅
URL: https://build.ci.opensearch.org/job/gradle-check/9664/
CommitID: 62db94a

github-actions · 2023-01-20T19:43:54Z

Gradle Check (Jenkins) Run Completed with:

RESULT: UNSTABLE ❕
TEST FAILURES:

      1 org.opensearch.indices.replication.SegmentReplicationIT.testDeleteOperations
      1 org.opensearch.cluster.routing.allocation.decider.DiskThresholdDeciderIT.testIndexCreateBlockWhenAllNodesExceededHighWatermark

URL: https://build.ci.opensearch.org/job/gradle-check/9673/
CommitID: fdb31f7
Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

github-actions · 2023-01-21T18:38:43Z

Gradle Check (Jenkins) Run Completed with:

RESULT: UNSTABLE ❕
TEST FAILURES:

      1 org.opensearch.cluster.routing.allocation.decider.DiskThresholdDeciderIT.testIndexCreateBlockWhenAllNodesExceededHighWatermark

URL: https://build.ci.opensearch.org/job/gradle-check/9722/
CommitID: e14d9dc
Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

github-actions · 2023-01-21T20:00:50Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/9723/
CommitID: 03617e4
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-01-21T20:14:23Z

Gradle Check (Jenkins) Run Completed with:

RESULT: SUCCESS ✅
URL: https://build.ci.opensearch.org/job/gradle-check/9724/
CommitID: 34e47b6

github-actions · 2023-01-21T20:53:26Z

Gradle Check (Jenkins) Run Completed with:

RESULT: UNSTABLE ❕
TEST FAILURES:

      1 org.opensearch.cluster.routing.allocation.decider.DiskThresholdDeciderIT.testIndexCreateBlockIsRemovedWhenAnyNodesNotExceedHighWatermark
      1 org.opensearch.cluster.routing.allocation.decider.DiskThresholdDeciderIT.classMethod

URL: https://build.ci.opensearch.org/job/gradle-check/9725/
CommitID: 9172666
Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

github-actions · 2023-01-22T10:19:54Z

Gradle Check (Jenkins) Run Completed with:

RESULT: SUCCESS ✅
URL: https://build.ci.opensearch.org/job/gradle-check/9733/
CommitID: ca7c35a

github-actions · 2023-01-22T10:34:42Z

Gradle Check (Jenkins) Run Completed with:

RESULT: SUCCESS ✅
URL: https://build.ci.opensearch.org/job/gradle-check/9735/
CommitID: 0784c8d

github-actions · 2023-01-22T10:36:47Z

Gradle Check (Jenkins) Run Completed with:

RESULT: UNSTABLE ❕
TEST FAILURES:

      1 org.opensearch.indices.replication.SegmentReplicationIT.testDeleteOperations
      1 org.opensearch.cluster.routing.allocation.decider.DiskThresholdDeciderIT.testIndexCreateBlockWithAReadOnlyBlock
      1 org.opensearch.cluster.routing.allocation.decider.DiskThresholdDeciderIT.classMethod

URL: https://build.ci.opensearch.org/job/gradle-check/9734/
CommitID: 6fc213c
Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

github-actions · 2023-01-22T11:57:03Z

Gradle Check (Jenkins) Run Completed with:

RESULT: UNSTABLE ❕
TEST FAILURES:

      1 org.opensearch.cluster.routing.allocation.decider.DiskThresholdDeciderIT.testIndexCreateBlockWhenAllNodesExceededHighWatermark
      1 org.opensearch.cluster.routing.allocation.decider.DiskThresholdDeciderIT.classMethod

URL: https://build.ci.opensearch.org/job/gradle-check/9736/
CommitID: 58f5e15
Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

github-actions · 2023-01-22T13:27:26Z

Gradle Check (Jenkins) Run Completed with:

RESULT: SUCCESS ✅
URL: https://build.ci.opensearch.org/job/gradle-check/9738/
CommitID: 7a173e4

github-actions · 2023-01-25T19:00:07Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/9903/
CommitID: 80e2b4a
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-01-25T19:21:16Z

Gradle Check (Jenkins) Run Completed with:

RESULT: UNSTABLE ❕
TEST FAILURES:

      1 org.opensearch.indices.replication.SegmentReplicationRelocationIT.testCancellation
      1 org.opensearch.cluster.allocation.AwarenessAllocationIT.testThreeZoneOneReplicaWithForceZoneValueAndLoadAwareness

URL: https://build.ci.opensearch.org/job/gradle-check/9904/
CommitID: ce20ab5
Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

github-actions · 2023-01-25T21:24:12Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/9921/
CommitID: 4451ab7
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-01-25T21:43:06Z

Gradle Check (Jenkins) Run Completed with:

RESULT: UNSTABLE ❕
TEST FAILURES:

      1 org.opensearch.client.ReindexIT.testDeleteByQueryTask
      1 org.opensearch.action.admin.cluster.node.tasks.ResourceAwareTasksTests.testTaskResourceTrackingDuringTaskCancellation

URL: https://build.ci.opensearch.org/job/gradle-check/9922/
CommitID: 6cf33f0
Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

RS146BIJAY · 2023-01-25T21:49:10Z

Is there any way to make this deterministic? Reducing the refresh frequency may have made the race condition less likely, but has it actually been fixed?

Made few portion of test cases synchronous to make these test cases deterministic. Tested it out with 50 iterations run with no failure.

RS146BIJAY · 2023-01-25T21:50:21Z

@RS146BIJAY Thanks for contributing and fixing these flaky tests. Can you please try rerunning the test class for some more (20+ ideally) iterations? You can use the guidelines here. This will help you iteratively run the tests with the fixes with a single comand.

Ran the DiskThresholdDeciderIt successfully with 50 iterations on local.

andrross · 2023-01-25T23:59:51Z

test/framework/src/main/java/org/opensearch/cluster/MockInternalClusterInfoService.java

+    public void invokeListeners(final ClusterInfo clusterInfo) {
+        listeners.forEach(listener -> listener.accept(clusterInfo));
+    }


This doesn't quite seem right, architecturally. Being able to manually invoke listeners for an arbitrary ClusterInfo seems like it violates encapsulation. Also the fact that you're maintaining a parallel list of listeners will potentially make this brittle to changes in the behavior of InternalClusterInfoService. Is there no other way to accomplish what you need?

Looks like synchronising refresh function inside InternalClusterInfoService will not be required as it refreshes the ClusterInfo in a blocking fashion. Simply disabling auto refresh (auto refresh was making test cases flaky) of InternalClusterInfoService and explicitly calling refresh after index on node is populated fixed the issue for me (Ignore previous errors as I missed disabling auto refresh on cluster manager node). I have validated test cases with more than 50 iterations on local.

Let me know if you see anything else can cause issue.

github-actions · 2023-01-26T22:00:17Z

Gradle Check (Jenkins) Run Completed with:

RESULT: SUCCESS ✅
URL: https://build.ci.opensearch.org/job/gradle-check/9994/
CommitID: e288d9e

github-actions · 2023-01-27T11:22:00Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/10035/
CommitID: 01cd9c2
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-01-27T12:01:31Z

Gradle Check (Jenkins) Run Completed with:

RESULT: UNSTABLE ❕
TEST FAILURES:

      1 org.opensearch.cluster.routing.allocation.decider.DiskThresholdDeciderIT.testIndexCreateBlockWithAReadOnlyBlock

URL: https://build.ci.opensearch.org/job/gradle-check/10036/
CommitID: 4c6e3c4
Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

github-actions · 2023-01-27T12:14:57Z

Gradle Check (Jenkins) Run Completed with:

RESULT: UNSTABLE ❕
TEST FAILURES:

      1 org.opensearch.cluster.routing.allocation.decider.DiskThresholdDeciderIT.testIndexCreateBlockIsRemovedWhenAnyNodesNotExceedHighWatermark

URL: https://build.ci.opensearch.org/job/gradle-check/10039/
CommitID: 639b08e
Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

github-actions · 2023-01-27T17:30:13Z

Gradle Check (Jenkins) Run Completed with:

RESULT: UNSTABLE ❕
TEST FAILURES:

      1 org.opensearch.cluster.ClusterHealthIT.testHealthOnClusterManagerFailover

URL: https://build.ci.opensearch.org/job/gradle-check/10050/
CommitID: dc87309
Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

Signed-off-by: Rishav Sagar <rissag@amazon.com>

github-actions · 2023-01-27T19:15:59Z

Gradle Check (Jenkins) Run Completed with:

RESULT: SUCCESS ✅
URL: https://build.ci.opensearch.org/job/gradle-check/10064/
CommitID: a72eea4

andrross

Thanks for working through this @RS146BIJAY!

Signed-off-by: Rishav Sagar <rissag@amazon.com> Co-authored-by: Rishav Sagar <rissag@amazon.com> (cherry picked from commit f3ed0d6) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

(cherry picked from commit f3ed0d6) Signed-off-by: Rishav Sagar <rissag@amazon.com> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: Rishav Sagar <rissag@amazon.com>

RS146BIJAY force-pushed the index-block-guardrails branch from 561f670 to a60d924 Compare January 20, 2023 16:25

andrross reviewed Jan 20, 2023

View reviewed changes

CHANGELOG.md Outdated Show resolved Hide resolved

RS146BIJAY force-pushed the index-block-guardrails branch from a60d924 to 62db94a Compare January 20, 2023 18:25

andrross mentioned this pull request Jan 20, 2023

DiskThresholdDeciderIT.testIndexCreateBlockWithAReadOnlyBlock is flaky #5956

Closed

RS146BIJAY force-pushed the index-block-guardrails branch from 62db94a to fdb31f7 Compare January 20, 2023 19:11

kotwanikunal added the skip-changelog label Jan 20, 2023

RS146BIJAY force-pushed the index-block-guardrails branch from fdb31f7 to e14d9dc Compare January 21, 2023 18:09

RS146BIJAY force-pushed the index-block-guardrails branch 2 times, most recently from 03617e4 to 34e47b6 Compare January 21, 2023 19:44

RS146BIJAY force-pushed the index-block-guardrails branch from 34e47b6 to 9172666 Compare January 21, 2023 20:16

RS146BIJAY force-pushed the index-block-guardrails branch 3 times, most recently from 6fc213c to 0784c8d Compare January 22, 2023 10:04

RS146BIJAY force-pushed the index-block-guardrails branch from 0784c8d to 58f5e15 Compare January 22, 2023 11:18

RS146BIJAY force-pushed the index-block-guardrails branch from 58f5e15 to 7a173e4 Compare January 22, 2023 12:57

RS146BIJAY force-pushed the index-block-guardrails branch 2 times, most recently from f5d79b1 to 55a4ff4 Compare January 22, 2023 13:51

RS146BIJAY force-pushed the index-block-guardrails branch 2 times, most recently from 4451ab7 to 6cf33f0 Compare January 25, 2023 21:12

dblock approved these changes Jan 25, 2023

View reviewed changes

dblock requested a review from andrross January 25, 2023 23:37

andrross reviewed Jan 25, 2023

View reviewed changes

RS146BIJAY force-pushed the index-block-guardrails branch from 6cf33f0 to e288d9e Compare January 26, 2023 21:25

RS146BIJAY force-pushed the index-block-guardrails branch from e288d9e to 01cd9c2 Compare January 27, 2023 10:54

RS146BIJAY force-pushed the index-block-guardrails branch 2 times, most recently from 4c6e3c4 to 639b08e Compare January 27, 2023 11:45

RS146BIJAY force-pushed the index-block-guardrails branch from 639b08e to dc87309 Compare January 27, 2023 16:57

Fixing flaky test cases for DiskThresholdDeciderIT

a72eea4

Signed-off-by: Rishav Sagar <rissag@amazon.com>

RS146BIJAY force-pushed the index-block-guardrails branch from dc87309 to a72eea4 Compare January 27, 2023 18:46

andrross approved these changes Jan 27, 2023

View reviewed changes

andrross merged commit f3ed0d6 into opensearch-project:main Jan 27, 2023

andrross added the backport 2.x Backport to 2.x branch label Feb 3, 2023

opensearch-trigger-bot bot mentioned this pull request Feb 3, 2023

[Backport 2.x] Fix flaky test cases for DiskThresholdDeciderIT #6166

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix flaky test cases for DiskThresholdDeciderIT #5952

Fix flaky test cases for DiskThresholdDeciderIT #5952

RS146BIJAY commented Jan 20, 2023 •

edited

Loading

github-actions bot commented Jan 20, 2023

github-actions bot commented Jan 20, 2023

github-actions bot commented Jan 20, 2023

github-actions bot commented Jan 20, 2023

github-actions bot commented Jan 21, 2023

github-actions bot commented Jan 21, 2023

github-actions bot commented Jan 21, 2023

github-actions bot commented Jan 21, 2023

github-actions bot commented Jan 22, 2023

github-actions bot commented Jan 22, 2023

github-actions bot commented Jan 22, 2023

github-actions bot commented Jan 22, 2023

github-actions bot commented Jan 22, 2023

github-actions bot commented Jan 25, 2023

github-actions bot commented Jan 25, 2023

github-actions bot commented Jan 25, 2023

github-actions bot commented Jan 25, 2023

RS146BIJAY commented Jan 25, 2023

RS146BIJAY commented Jan 25, 2023

andrross Jan 25, 2023

RS146BIJAY Jan 27, 2023

github-actions bot commented Jan 26, 2023

github-actions bot commented Jan 27, 2023

github-actions bot commented Jan 27, 2023

github-actions bot commented Jan 27, 2023

github-actions bot commented Jan 27, 2023

github-actions bot commented Jan 27, 2023

andrross left a comment

Fix flaky test cases for DiskThresholdDeciderIT #5952

Fix flaky test cases for DiskThresholdDeciderIT #5952

Conversation

RS146BIJAY commented Jan 20, 2023 • edited Loading

Description

Issues Resolved

Check List

github-actions bot commented Jan 20, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Jan 20, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Jan 20, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Jan 20, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Jan 21, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Jan 21, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Jan 21, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Jan 21, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Jan 22, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Jan 22, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Jan 22, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Jan 22, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Jan 22, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Jan 25, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Jan 25, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Jan 25, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Jan 25, 2023

Gradle Check (Jenkins) Run Completed with:

RS146BIJAY commented Jan 25, 2023

RS146BIJAY commented Jan 25, 2023

andrross Jan 25, 2023

Choose a reason for hiding this comment

RS146BIJAY Jan 27, 2023

Choose a reason for hiding this comment

github-actions bot commented Jan 26, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Jan 27, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Jan 27, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Jan 27, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Jan 27, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Jan 27, 2023

Gradle Check (Jenkins) Run Completed with:

andrross left a comment

Choose a reason for hiding this comment

RS146BIJAY commented Jan 20, 2023 •

edited

Loading