Fix flaky Segment Replication test testStartReplicaAfterPrimaryIndexesDocs. #5722

mch2 · 2023-01-06T00:37:59Z

Signed-off-by: Marc Handalian handalm@amazon.com

Description

Fix flaky SegmentReplicationIT test testStartReplicaAfterPrimaryIndexesDocs.
This test was failing because we are validating post recovery if a shard is able to perform segrep while also performing validation of a passed in checkopint. In the post recovery scenario this checkpoint is passed as empty, yet the shard will be ahead of this empty checkpoint after docs are indexed and fail validation. This change differentiates shard validation from checkpoint validation and only performs the former post recovery.

This PR also introduces validation of the engine type before SR is invoked. This is to ensure NRTReplicationEngine is properly loaded on the replica. Without this SR would continue and blow up at a later stage with an index corruption error. This happens a lot when MockInternalEngine is randomly loaded in tests as this method by default returns true.

I've re-run this test 1k times in intellij without failure.

Issues Resolved

related #5669

Check List

New functionality includes testing.
- All tests pass
New functionality has been documented.
- New functionality has javadoc added
Commits are signed per the DCO using --signoff
Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

github-actions · 2023-01-06T00:39:50Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/8915/
CommitID: c2202ee
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-01-06T01:02:52Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/8916/
CommitID: 0d0a5da
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-01-06T01:37:36Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/8917/
CommitID: c87c641
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-01-06T02:59:32Z

Gradle Check (Jenkins) Run Completed with:

RESULT: UNSTABLE ❕
TEST FAILURES:

      1 org.opensearch.client.RestClientMultipleHostsIntegTests.testNodeSelector

URL: https://build.ci.opensearch.org/job/gradle-check/8919/
CommitID: d22f9ac
Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

codecov-commenter · 2023-01-06T03:01:39Z

Codecov Report

Merging #5722 (f8e001f) into main (b3e25bb) will increase coverage by 0.04%.
The diff coverage is 81.25%.

@@             Coverage Diff              @@
##               main    #5722      +/-   ##
============================================
+ Coverage     71.05%   71.10%   +0.04%     
+ Complexity    58744    58741       -3     
============================================
  Files          4766     4766              
  Lines        280030   280030              
  Branches      40434    40434              
============================================
+ Hits         198988   199117     +129     
+ Misses        64851    64679     -172     
- Partials      16191    16234      +43

Impacted Files	Coverage Δ
...ch/indices/cluster/IndicesClusterStateService.java	`69.66% <0.00%> (+1.02%)`	⬆️
...in/java/org/opensearch/index/shard/IndexShard.java	`71.16% <86.66%> (+0.96%)`	⬆️
...n/indices/forcemerge/ForceMergeRequestBuilder.java	`0.00% <0.00%> (-75.00%)`	⬇️
.../indices/forcemerge/TransportForceMergeAction.java	`25.00% <0.00%> (-75.00%)`	⬇️
.../java/org/opensearch/node/NodeClosedException.java	`50.00% <0.00%> (-50.00%)`	⬇️
...ava/org/opensearch/action/NoSuchNodeException.java	`0.00% <0.00%> (-50.00%)`	⬇️
...opensearch/persistent/PersistentTasksExecutor.java	`22.22% <0.00%> (-44.45%)`	⬇️
...adcast/BroadcastShardOperationFailedException.java	`55.55% <0.00%> (-44.45%)`	⬇️
.../admin/cluster/reroute/ClusterRerouteResponse.java	`60.00% <0.00%> (-40.00%)`	⬇️
...luster/routing/allocation/RoutingExplanations.java	`62.06% <0.00%> (-37.94%)`	⬇️
... and 493 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

github-actions · 2023-01-06T17:58:22Z

Gradle Check (Jenkins) Run Completed with:

RESULT: SUCCESS ✅
URL: https://build.ci.opensearch.org/job/gradle-check/8959/
CommitID: 05ad808

github-actions · 2023-01-06T18:03:22Z

Gradle Check (Jenkins) Run Completed with:

RESULT: SUCCESS ✅
URL: https://build.ci.opensearch.org/job/gradle-check/8960/
CommitID: f8e001f

CHANGELOG.md

dreamer-89 · 2023-01-06T18:05:54Z

server/src/main/java/org/opensearch/index/shard/IndexShard.java

-    public final boolean shouldProcessCheckpoint(ReplicationCheckpoint requestCheckpoint) {
-        if (state().equals(IndexShardState.STARTED) == false) {
-            logger.trace(() -> new ParameterizedMessage("Ignoring new replication checkpoint - shard is not started {}", state()));
+    public boolean isSegmentReplicationAllowed() {


nit: isSegmentReplicationAllowed does not sound that it is meant for target/replica. isSegRepSyncAllowed or isSegRepAllowedOnReplica ?

This method is not just invoked for replicas, even after primary is recovered it will be invoked from IndicesClusterStateService#forceSegmentReplication.

dreamer-89 · 2023-01-06T18:10:04Z

server/src/main/java/org/opensearch/index/shard/IndexShard.java

-     *
-     * @param requestCheckpoint       received checkpoint that is checked for processing
-     * @return true if checkpoint should be processed
+     * Checks if this shard is able to perform segment replication.


nit:

Checks if this shard is able to perform segment replication.

Checks if this target shard should start round of segment replication with primary ?

Updated - I'm intentionally not mentioning primary here as these paths will be reused for remote store replication.

server/src/main/java/org/opensearch/indices/cluster/IndicesClusterStateService.java

github-actions · 2023-01-06T18:59:52Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/8964/
CommitID: de44084
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

dreamer-89 · 2023-01-06T19:17:15Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌

URL: https://build.ci.opensearch.org/job/gradle-check/8964/

CommitID: de44084
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

Does not seem related, refiring the gradle check.

REPRODUCE WITH: ./gradlew ':server:test' --tests "org.opensearch.action.support.replication.TransportReplicationActionTests.testClosedIndexOnReroute" -Dtests.seed=353F0B5144D33354 -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=fr-LU -Dtests.timezone=America/Argentina/San_Juan -Druntime.java=19

org.opensearch.action.support.replication.TransportReplicationActionTests > testClosedIndexOnReroute FAILED
    java.lang.IllegalStateException: No local node found. Is the node started?
        at __randomizedtesting.SeedInfo.seed([353F0B5144D33354:59C1A9D22741BA1F]:0)
        at org.opensearch.cluster.service.ClusterService.localNode(ClusterService.java:156)
        at org.opensearch.action.support.replication.TransportReplicationAction$ReroutePhase.<init>(TransportReplicationAction.java:890)
        at org.opensearch.action.support.replication.TransportReplicationAction$ReroutePhase.<init>(TransportReplicationAction.java:883)
        at org.opensearch.action.support.replication.TransportReplicationActionTests.testClosedIndexOnReroute(TransportReplicationActionTests.java:640)

github-actions · 2023-01-06T19:42:51Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/8967/
CommitID: de44084
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-01-06T20:35:31Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/8969/
CommitID: 0cdb895
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

dreamer-89 · 2023-01-06T21:38:26Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌

URL: https://build.ci.opensearch.org/job/gradle-check/8969/

CommitID: 0cdb895
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

This is fixed in #5737. @mch2 : Can you please rebase your changes against main ?

This test was failing because we are validating post recovery if a shard is able to perform segrep while also performing validation if a passed in checkopint. In the post recovery test this checkpoint is always empty, yet the shard will be ahead of this checkpoint after docs are indexed. This change differentiates shard validation from checkpoint validation. Signed-off-by: Marc Handalian <handalm@amazon.com> Fix spotless. Signed-off-by: Marc Handalian <handalm@amazon.com> Fix testIsSegmentReplicationAllowed_WrongEngineType. Signed-off-by: Marc Handalian <handalm@amazon.com> Update warn logs in isSegmentReplicationAllowed. Signed-off-by: Marc Handalian <handalm@amazon.com>

Signed-off-by: Marc Handalian <handalm@amazon.com>

github-actions · 2023-01-06T23:31:40Z

Gradle Check (Jenkins) Run Completed with:

RESULT: UNSTABLE ❕
TEST FAILURES:

      1 org.opensearch.indices.replication.SegmentReplicationIT.testRestartPrimary

URL: https://build.ci.opensearch.org/job/gradle-check/8983/
CommitID: 7b2bc53
Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

…sDocs. (opensearch-project#5722) * Fix flaky SR test testStartReplicaAfterPrimaryIndexesDocs. This test was failing because we are validating post recovery if a shard is able to perform segrep while also performing validation if a passed in checkopint. In the post recovery test this checkpoint is always empty, yet the shard will be ahead of this checkpoint after docs are indexed. This change differentiates shard validation from checkpoint validation. Signed-off-by: Marc Handalian <handalm@amazon.com> Fix spotless. Signed-off-by: Marc Handalian <handalm@amazon.com> Fix testIsSegmentReplicationAllowed_WrongEngineType. Signed-off-by: Marc Handalian <handalm@amazon.com> Update warn logs in isSegmentReplicationAllowed. Signed-off-by: Marc Handalian <handalm@amazon.com> * PR feedback. Signed-off-by: Marc Handalian <handalm@amazon.com> Signed-off-by: Marc Handalian <handalm@amazon.com>

….x. (#5945) * [Segment Replication] Add snapshot and restore tests for segment replication feature (#3993) * [Segment Replication] Add snapshots tests with segment replication enabled Signed-off-by: Suraj Singh <surajrider@gmail.com> * Fix spotless failures Signed-off-by: Suraj Singh <surajrider@gmail.com> * Add changelog entry, address review comments, add failover test Signed-off-by: Suraj Singh <surajrider@gmail.com> * Fix spotless failures Signed-off-by: Suraj Singh <surajrider@gmail.com> * Address review comments 2 Signed-off-by: Suraj Singh <surajrider@gmail.com> Signed-off-by: Suraj Singh <surajrider@gmail.com> * Remove changelog update. Signed-off-by: Marc Handalian <handalm@amazon.com> * Mute flaky test testStartReplicaAfterPrimaryIndexesDocs. (#5714) Signed-off-by: Marc Handalian <handalm@amazon.com> Signed-off-by: Marc Handalian <handalm@amazon.com> * Fix flaky Segment Replication test testStartReplicaAfterPrimaryIndexesDocs. (#5722) * Fix flaky SR test testStartReplicaAfterPrimaryIndexesDocs. This test was failing because we are validating post recovery if a shard is able to perform segrep while also performing validation if a passed in checkopint. In the post recovery test this checkpoint is always empty, yet the shard will be ahead of this checkpoint after docs are indexed. This change differentiates shard validation from checkpoint validation. Signed-off-by: Marc Handalian <handalm@amazon.com> Fix spotless. Signed-off-by: Marc Handalian <handalm@amazon.com> Fix testIsSegmentReplicationAllowed_WrongEngineType. Signed-off-by: Marc Handalian <handalm@amazon.com> Update warn logs in isSegmentReplicationAllowed. Signed-off-by: Marc Handalian <handalm@amazon.com> * PR feedback. Signed-off-by: Marc Handalian <handalm@amazon.com> Signed-off-by: Marc Handalian <handalm@amazon.com> * [Segment Replication] Mute flaky tests (#5739) Signed-off-by: Suraj Singh <surajrider@gmail.com> Signed-off-by: Suraj Singh <surajrider@gmail.com> * [Segment Replication] Mute flaky tests (#5742) Signed-off-by: Suraj Singh <surajrider@gmail.com> Signed-off-by: Suraj Singh <surajrider@gmail.com> * Fix spotless. Signed-off-by: Marc Handalian <handalm@amazon.com> * Muting flaky SegmentReplication ITs. (#5700) Signed-off-by: Marc Handalian <handalm@amazon.com> Signed-off-by: Marc Handalian <handalm@amazon.com> Signed-off-by: Suraj Singh <surajrider@gmail.com> Signed-off-by: Marc Handalian <handalm@amazon.com> Co-authored-by: Suraj Singh <surajrider@gmail.com>

mch2 requested review from reta, anasalkouz, andrross, Bukhtawar, CEHENKLE, dblock, setiah, kartg, kotwanikunal, nknize, owaiskazi19, adnapibar, Rishikesh1159, ryanbogan, saratvemulapalli, shwetathareja, dreamer-89, tlfeng and xuezhou25 as code owners January 6, 2023 00:37

mch2 force-pushed the sr-validation branch from c2202ee to 0d0a5da Compare January 6, 2023 00:38

mch2 force-pushed the sr-validation branch from 05ad808 to f8e001f Compare January 6, 2023 17:33

mch2 mentioned this pull request Jan 6, 2023

[Flaky Test Failure] RestClientMultipleHostsIntegTests.testNodeSelector #5734

Open

mch2 force-pushed the sr-validation branch from f8e001f to de44084 Compare January 6, 2023 18:28

mch2 requested review from gbbafna and VachaShah as code owners January 6, 2023 18:28

dreamer-89 reviewed Jan 6, 2023

View reviewed changes

mch2 added the skip-changelog label Jan 6, 2023

dreamer-89 mentioned this pull request Jan 6, 2023

Fix TransportReplicationActionTests #5737

Merged

6 tasks

mch2 added 2 commits January 6, 2023 15:03

PR feedback.

7b2bc53

Signed-off-by: Marc Handalian <handalm@amazon.com>

mch2 force-pushed the sr-validation branch from 0cdb895 to 7b2bc53 Compare January 6, 2023 23:03

Rishikesh1159 approved these changes Jan 6, 2023

View reviewed changes

Rishikesh1159 mentioned this pull request Jan 6, 2023

[Meta] Segment Replication flaky test failures #5669

Closed

8 tasks

dreamer-89 merged commit 85f4149 into opensearch-project:main Jan 7, 2023

mch2 deleted the sr-validation branch January 7, 2023 00:09

This was referenced Jan 7, 2023

[Segment Replication] Mute flaky tests #5742

Merged

[Segment Replication] Update replica recovery to perform replication from source #5743

Closed

gbbafna mentioned this pull request Jan 9, 2023

[Flaky Test Failure] TransportReplicationActionTests.testClosedIndexOnReroute #5765

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix flaky Segment Replication test testStartReplicaAfterPrimaryIndexesDocs. #5722

Fix flaky Segment Replication test testStartReplicaAfterPrimaryIndexesDocs. #5722

mch2 commented Jan 6, 2023 •

edited

Loading

github-actions bot commented Jan 6, 2023

github-actions bot commented Jan 6, 2023

github-actions bot commented Jan 6, 2023

github-actions bot commented Jan 6, 2023

codecov-commenter commented Jan 6, 2023 •

edited

Loading

github-actions bot commented Jan 6, 2023

github-actions bot commented Jan 6, 2023

dreamer-89 Jan 6, 2023

mch2 Jan 6, 2023

dreamer-89 Jan 6, 2023

mch2 Jan 6, 2023

github-actions bot commented Jan 6, 2023

dreamer-89 commented Jan 6, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Jan 6, 2023

github-actions bot commented Jan 6, 2023

dreamer-89 commented Jan 6, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Jan 6, 2023

Fix flaky Segment Replication test testStartReplicaAfterPrimaryIndexesDocs. #5722

Fix flaky Segment Replication test testStartReplicaAfterPrimaryIndexesDocs. #5722

Conversation

mch2 commented Jan 6, 2023 • edited Loading

Description

Issues Resolved

Check List

github-actions bot commented Jan 6, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Jan 6, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Jan 6, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Jan 6, 2023

Gradle Check (Jenkins) Run Completed with:

codecov-commenter commented Jan 6, 2023 • edited Loading

Codecov Report

github-actions bot commented Jan 6, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Jan 6, 2023

Gradle Check (Jenkins) Run Completed with:

dreamer-89 Jan 6, 2023

Choose a reason for hiding this comment

mch2 Jan 6, 2023

Choose a reason for hiding this comment

dreamer-89 Jan 6, 2023

Choose a reason for hiding this comment

mch2 Jan 6, 2023

Choose a reason for hiding this comment

github-actions bot commented Jan 6, 2023

Gradle Check (Jenkins) Run Completed with:

dreamer-89 commented Jan 6, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Jan 6, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Jan 6, 2023

Gradle Check (Jenkins) Run Completed with:

dreamer-89 commented Jan 6, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Jan 6, 2023

Gradle Check (Jenkins) Run Completed with:

mch2 commented Jan 6, 2023 •

edited

Loading

codecov-commenter commented Jan 6, 2023 •

edited

Loading