[Backport 2.x] [Segment Replication] Add check to cancel ongoing replication with old primary on onNewCheckpoint on replica #4396

dreamer-89 · 2022-09-02T20:04:10Z

Manul backport of #4363 to 2.x

…d primary on onNewCheckpoint on replica (opensearch-project#4363) * [Segment Replication] Add check to cancel ongoing replication with old primary on onNewCheckpoint on replica Signed-off-by: Suraj Singh <surajrider@gmail.com> * Add changelog entry Signed-off-by: Suraj Singh <surajrider@gmail.com> * Address review comments Signed-off-by: Suraj Singh <surajrider@gmail.com> * Address review comments 2 Signed-off-by: Suraj Singh <surajrider@gmail.com> * Test failures Signed-off-by: Suraj Singh <surajrider@gmail.com> Signed-off-by: Suraj Singh <surajrider@gmail.com>

github-actions · 2022-09-02T20:40:53Z

Gradle Check (Jenkins) Run Completed with:

RESULT: SUCCESS ✅
URL: https://build.ci.opensearch.org/job/gradle-check/2575/
CommitID: 5fde990

codecov-commenter · 2022-09-02T20:43:01Z

Codecov Report

Merging #4396 (495d8e6) into 2.x (b202fd1) will increase coverage by 0.06%.
The diff coverage is 75.23%.

@@             Coverage Diff              @@
##                2.x    #4396      +/-   ##
============================================
+ Coverage     70.54%   70.61%   +0.06%     
- Complexity    56942    57126     +184     
============================================
  Files          4572     4584      +12     
  Lines        273816   274497     +681     
  Branches      40152    40226      +74     
============================================
+ Hits         193170   193833     +663     
+ Misses        64455    64431      -24     
- Partials      16191    16233      +42

Impacted Files	Coverage Δ
.../org/opensearch/client/support/AbstractClient.java	`34.70% <0.00%> (-0.44%)`	⬇️
...pensearch/common/settings/IndexScopedSettings.java	`100.00% <ø> (ø)`
.../java/org/opensearch/common/util/FeatureFlags.java	`50.00% <ø> (ø)`
...rc/main/java/org/opensearch/index/IndexModule.java	`82.03% <ø> (ø)`
.../java/org/opensearch/plugins/IndexStorePlugin.java	`100.00% <ø> (ø)`
.../java/org/opensearch/snapshots/RestoreService.java	`55.55% <0.00%> (-6.55%)`	⬇️
...opensearch/cluster/routing/ShardRoutingHelper.java	`95.65% <ø> (ø)`
...ore/restore/TransportRestoreRemoteStoreAction.java	`25.00% <25.00%> (ø)`
...c/main/java/org/opensearch/index/IndexService.java	`73.57% <28.57%> (-0.56%)`	⬇️
...on/admin/cluster/RestRestoreRemoteStoreAction.java	`33.33% <33.33%> (ø)`
... and 519 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

github-actions · 2022-09-03T00:52:57Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/2589/
CommitID: 495d8e6

dreamer-89 · 2022-09-03T01:02:02Z

503 error during ping on github adoptium binary release. Will retry gradle check after some time.

> Task :distribution:bwc:maintenance:buildBwcLinuxTar
 [1.3.5] > Task :distribution:archives:buildLinux FAILED
 [1.3.5] build complete, generating: /var/jenkins/workspace/gradle-check/search/distribution/bwc/maintenance/build/bwc/checkout-1.3/build/2589.tar.bz2
 [1.3.5] 
 [1.3.5] 148 actionable tasks: 148 executed
 [1.3.5] FAILURE: Build failed with an exception.
 [1.3.5] 
 [1.3.5] * What went wrong:
 [1.3.5] Execution failed for task ':distribution:archives:buildLinux'.
 [1.3.5] > Could not resolve all files for configuration ':distribution:archives:jdk_bundled_linux_x64'.
 [1.3.5]    > Could not resolve adoptium_11:linux:11.0.16.
 [1.3.5]      Required by:
 [1.3.5]          project :distribution:archives
 [1.3.5]       > Could not resolve adoptium_11:linux:11.0.16.
 [1.3.5]          > Could not get resource 'https://github.com/adoptium/temurin11-binaries/releases/download/jdk-11.0.16+8/OpenJDK11U-jdk_x64_linux_hotspot_11.0.16_8.tar.gz'.
 [1.3.5]             > Could not HEAD 'https://github.com/adoptium/temurin11-binaries/releases/download/jdk-11.0.16+8/OpenJDK11U-jdk_x64_linux_hotspot_11.0.16_8.tar.gz'. Received status code 503 from server: Backend fetch failed
 [1.3.5] 
 [1.3.5] * Try:
 [1.3.5] Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.
 [1.3.5] 
 [1.3.5] * Get more help at https://help.gradle.org/
 [1.3.5] 
 [1.3.5] BUILD FAILED in 4m 18s

github-actions · 2022-09-03T02:20:40Z

Gradle Check (Jenkins) Run Completed with:

RESULT: UNSTABLE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/2591/
CommitID: 495d8e6

dreamer-89 · 2022-09-03T05:29:07Z

This test was added as part of PR #4363. Looking into it.

REPRODUCE WITH: ./gradlew ':server:test' --tests "org.opensearch.indices.replication.SegmentReplicationTargetServiceTests.testOnNewCheckpointFromNewPrimaryCancelOngoingReplication" -Dtests.seed=6B983CC6B30CF463 -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=nb-NO -Dtests.timezone=Africa/Malabo -Druntime.java=17

org.opensearch.indices.replication.SegmentReplicationTargetServiceTests > testOnNewCheckpointFromNewPrimaryCancelOngoingReplication FAILED
    org.mockito.exceptions.misusing.UnfinishedStubbingException: 
    Unfinished stubbing detected here:
    -> at org.opensearch.indices.replication.SegmentReplicationTargetServiceTests.testOnNewCheckpointFromNewPrimaryCancelOngoingReplication(SegmentReplicationTargetServiceTests.java:232)

    E.g. thenReturn() may be missing.
    Examples of correct stubbing:
        when(mock.isOk()).thenReturn(true);
        when(mock.isOk()).thenThrow(exception);
        doThrow(exception).when(mock).someVoidMethod();
    Hints:
     1. missing thenReturn()
     2. you are trying to stub a final method, which is not supported
     3. you are stubbing the behaviour of another mock inside before 'thenReturn' instruction is completed
        at __randomizedtesting.SeedInfo.seed([6B983CC6B30CF463:B2A5F5E8335F5752]:0)
        at app//org.opensearch.indices.replication.SegmentReplicationTargetServiceTests.testOnNewCheckpointFromNewPrimaryCancelOngoingReplication(SegmentReplicationTargetServiceTests.java:232)

github-actions · 2022-09-03T06:02:40Z

Gradle Check (Jenkins) Run Completed with:

RESULT: SUCCESS ✅
URL: https://build.ci.opensearch.org/job/gradle-check/2594/
CommitID: 495d8e6

mch2 · 2022-09-05T03:18:20Z

This test was added as part of PR #4363. Looking into it.

REPRODUCE WITH: ./gradlew ':server:test' --tests "org.opensearch.indices.replication.SegmentReplicationTargetServiceTests.testOnNewCheckpointFromNewPrimaryCancelOngoingReplication" -Dtests.seed=6B983CC6B30CF463 -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=nb-NO -Dtests.timezone=Africa/Malabo -Druntime.java=17

org.opensearch.indices.replication.SegmentReplicationTargetServiceTests > testOnNewCheckpointFromNewPrimaryCancelOngoingReplication FAILED
    org.mockito.exceptions.misusing.UnfinishedStubbingException: 
    Unfinished stubbing detected here:
    -> at org.opensearch.indices.replication.SegmentReplicationTargetServiceTests.testOnNewCheckpointFromNewPrimaryCancelOngoingReplication(SegmentReplicationTargetServiceTests.java:232)

    E.g. thenReturn() may be missing.
    Examples of correct stubbing:
        when(mock.isOk()).thenReturn(true);
        when(mock.isOk()).thenThrow(exception);
        doThrow(exception).when(mock).someVoidMethod();
    Hints:
     1. missing thenReturn()
     2. you are trying to stub a final method, which is not supported
     3. you are stubbing the behaviour of another mock inside before 'thenReturn' instruction is completed
        at __randomizedtesting.SeedInfo.seed([6B983CC6B30CF463:B2A5F5E8335F5752]:0)
        at app//org.opensearch.indices.replication.SegmentReplicationTargetServiceTests.testOnNewCheckpointFromNewPrimaryCancelOngoingReplication(SegmentReplicationTargetServiceTests.java:232)

For another seed - #4366 (comment)

dreamer-89 · 2022-09-05T19:10:37Z

This test was added as part of PR #4363. Looking into it.

REPRODUCE WITH: ./gradlew ':server:test' --tests "org.opensearch.indices.replication.SegmentReplicationTargetServiceTests.testOnNewCheckpointFromNewPrimaryCancelOngoingReplication" -Dtests.seed=6B983CC6B30CF463 -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=nb-NO -Dtests.timezone=Africa/Malabo -Druntime.java=17

org.opensearch.indices.replication.SegmentReplicationTargetServiceTests > testOnNewCheckpointFromNewPrimaryCancelOngoingReplication FAILED
    org.mockito.exceptions.misusing.UnfinishedStubbingException: 
    Unfinished stubbing detected here:
    -> at org.opensearch.indices.replication.SegmentReplicationTargetServiceTests.testOnNewCheckpointFromNewPrimaryCancelOngoingReplication(SegmentReplicationTargetServiceTests.java:232)

    E.g. thenReturn() may be missing.
    Examples of correct stubbing:
        when(mock.isOk()).thenReturn(true);
        when(mock.isOk()).thenThrow(exception);
        doThrow(exception).when(mock).someVoidMethod();
    Hints:
     1. missing thenReturn()
     2. you are trying to stub a final method, which is not supported
     3. you are stubbing the behaviour of another mock inside before 'thenReturn' instruction is completed
        at __randomizedtesting.SeedInfo.seed([6B983CC6B30CF463:B2A5F5E8335F5752]:0)
        at app//org.opensearch.indices.replication.SegmentReplicationTargetServiceTests.testOnNewCheckpointFromNewPrimaryCancelOngoingReplication(SegmentReplicationTargetServiceTests.java:232)

For another seed - #4366 (comment)

Raised #4414

dreamer-89 requested review from a team and reta as code owners September 2, 2022 20:04

mch2 approved these changes Sep 3, 2022

View reviewed changes

dreamer-89 changed the title ~~[Segment Replication] Add check to cancel ongoing replication with old primary on onNewCheckpoint on replica~~ [Backport to 2.x] [Segment Replication] Add check to cancel ongoing replication with old primary on onNewCheckpoint on replica Sep 3, 2022

Merge branch '2.x' into mch2_updateTargetService_2x

495d8e6

dreamer-89 changed the title ~~[Backport to 2.x] [Segment Replication] Add check to cancel ongoing replication with old primary on onNewCheckpoint on replica~~ [Backport 2.x] [Segment Replication] Add check to cancel ongoing replication with old primary on onNewCheckpoint on replica Sep 3, 2022

dreamer-89 merged commit fc97324 into opensearch-project:2.x Sep 5, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Backport 2.x] [Segment Replication] Add check to cancel ongoing replication with old primary on onNewCheckpoint on replica #4396

[Backport 2.x] [Segment Replication] Add check to cancel ongoing replication with old primary on onNewCheckpoint on replica #4396

dreamer-89 commented Sep 2, 2022

github-actions bot commented Sep 2, 2022

codecov-commenter commented Sep 2, 2022 •

edited

Loading

github-actions bot commented Sep 3, 2022

dreamer-89 commented Sep 3, 2022 •

edited

Loading

github-actions bot commented Sep 3, 2022

dreamer-89 commented Sep 3, 2022

github-actions bot commented Sep 3, 2022

mch2 commented Sep 5, 2022

dreamer-89 commented Sep 5, 2022

[Backport 2.x] [Segment Replication] Add check to cancel ongoing replication with old primary on onNewCheckpoint on replica #4396

[Backport 2.x] [Segment Replication] Add check to cancel ongoing replication with old primary on onNewCheckpoint on replica #4396

Conversation

dreamer-89 commented Sep 2, 2022

github-actions bot commented Sep 2, 2022

Gradle Check (Jenkins) Run Completed with:

codecov-commenter commented Sep 2, 2022 • edited Loading

Codecov Report

github-actions bot commented Sep 3, 2022

Gradle Check (Jenkins) Run Completed with:

dreamer-89 commented Sep 3, 2022 • edited Loading

github-actions bot commented Sep 3, 2022

Gradle Check (Jenkins) Run Completed with:

dreamer-89 commented Sep 3, 2022

github-actions bot commented Sep 3, 2022

Gradle Check (Jenkins) Run Completed with:

mch2 commented Sep 5, 2022

dreamer-89 commented Sep 5, 2022

codecov-commenter commented Sep 2, 2022 •

edited

Loading

dreamer-89 commented Sep 3, 2022 •

edited

Loading