Segment Replication - Fix ShardLockObtained error during corruption cases

[mch2]

Description

This change fixes a bug where shards could not be recreated locally after corruption. This occured because the store was not decref'd to 0 if the commit on close would fail with a corruption exception.

This change handles failures during the commit on close and marks the store corrupt if it occurs.
This change also handles failures while fetching local metadata by wrapping them as a corruption exception - without this shards would indefinitely retry and fail replication events. Rethrowing the exception as a corruption exception ensures the shard is properly failed and re-recovered. For this to happen a file would have to be wiped directly from the store, but would require a node reboot to correct it.

Added some ITs that will test these corruption cases - primary sending faulty data to the replica and a file being wiped underneath the shard.

Related Issues

Resolves #8286
Resolves #9966

Check List

• New functionality includes testing.
  • All tests pass
• New functionality has been documented.
  • New functionality has javadoc added
• Commits are signed per the DCO using --signoff
• Commit changes are listed out in CHANGELOG.md file (See: Changelog)
• GitHub issue/PR created in OpenSearch documentation repo for the required public documentation changes (#[Issue/PR number])

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/27075/
• CommitID: 9dfc85d
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: UNSTABLE ❕
• TEST FAILURES:

      1 org.opensearch.remotestore.SegmentReplicationUsingRemoteStoreIT.testWipeSegmentBetweenSyncs

• URL: https://build.ci.opensearch.org/job/gradle-check/27078/
• CommitID: 9dfc85d
  Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[codecov]

Codecov Report

Merging #10370 (2b80149) into main (a0cb344) will decrease coverage by 0.11%.
Report is 14 commits behind head on main.
The diff coverage is 50.00%.

@@             Coverage Diff              @@
##               main   #10370      +/-   ##
============================================
- Coverage     71.21%   71.10%   -0.11%     
+ Complexity    58337    58265      -72     
============================================
  Files          4832     4832              
  Lines        274828   274850      +22     
  Branches      40043    40045       +2     
============================================
- Hits         195708   195422     -286     
- Misses        62728    63011     +283     
- Partials      16392    16417      +25     

Files	Coverage Δ
.../indices/replication/SegmentReplicationTarget.java	84.87% <33.33%> (ø)
...rc/main/java/org/opensearch/index/store/Store.java	81.10% <40.00%> (+1.10%)	⬆️
...s/replication/SegmentReplicationTargetService.java	53.87% <50.00%> (-3.08%)	⬇️
.../opensearch/index/engine/NRTReplicationEngine.java	78.19% <57.14%> (-0.58%)	⬇️

... and 482 files with indirect coverage changes

[mch2]

Gradle Check (Jenkins) Run Completed with:

• RESULT: UNSTABLE ❕
• TEST FAILURES:

      1 org.opensearch.remotestore.SegmentReplicationUsingRemoteStoreIT.testWipeSegmentBetweenSyncs

• URL: https://build.ci.opensearch.org/job/gradle-check/27078/
• CommitID: 9dfc85d
  Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

grr this IT is still showing flaky, yanking it.

[mch2]

Gradle Check (Jenkins) Run Completed with:

• RESULT: UNSTABLE ❕
• TEST FAILURES:

      1 org.opensearch.remotestore.SegmentReplicationUsingRemoteStoreIT.testWipeSegmentBetweenSyncs

• URL: https://build.ci.opensearch.org/job/gradle-check/27078/
• CommitID: 9dfc85d
  Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

grr this IT is still showing flaky, yanking it.

From this trace it looks like multiple nodes are spinning up node_t0-node_t4. I haven't seen this locally but looks like the test is spawning additional data nodes instead of recovering on the original node/shard path.

[mch2]

Gradle Check (Jenkins) Run Completed with:

• RESULT: UNSTABLE ❕
• TEST FAILURES:

      1 org.opensearch.remotestore.SegmentReplicationUsingRemoteStoreIT.testWipeSegmentBetweenSyncs

• URL: https://build.ci.opensearch.org/job/gradle-check/27078/
• CommitID: 9dfc85d
  Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

grr this IT is still showing flaky, yanking it.

From this trace it looks like multiple nodes are spinning up node_t0-node_t4. I haven't seen this locally but looks like the test is spawning additional data nodes instead of recovering on the original node/shard path.

Ok found the cause here and fixing it in next commit - The issue appears to occur when a flush fails after we cause corruption - if during SR this will throw a FlushFailedException which is not treated as corruption case and assume the ReplicationTarget threads have already been cancelled. This case needs to be treated as corruption so the shard is failed.

[mch2]

Gradle Check (Jenkins) Run Completed with:

• RESULT: UNSTABLE ❕
• TEST FAILURES:

      1 org.opensearch.remotestore.SegmentReplicationUsingRemoteStoreIT.testWipeSegmentBetweenSyncs

• URL: https://build.ci.opensearch.org/job/gradle-check/27078/
• CommitID: 9dfc85d
  Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

grr this IT is still showing flaky, yanking it.

From this trace it looks like multiple nodes are spinning up node_t0-node_t4. I haven't seen this locally but looks like the test is spawning additional data nodes instead of recovering on the original node/shard path.

Ok found the cause here and fixing it in next commit - The issue appears to occur when a flush fails after we cause corruption - if during SR this will throw a FlushFailedException which is not treated as corruption case and assume the ReplicationTarget threads have already been cancelled. This case needs to be treated as corruption so the shard is failed.

Fixed this by ensuring we fail engines on corruption if a flush fails in NRTReplicationEngine. Also cleaned up exception handling in finalizeReplication method.

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: UNSTABLE ❕
• TEST FAILURES:

      1 org.opensearch.remotestore.SegmentReplicationUsingRemoteStoreIT.testMultipleShards

• URL: https://build.ci.opensearch.org/job/gradle-check/27171/
• CommitID: 2b80149
  Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[mch2]

Gradle Check (Jenkins) Run Completed with:

• RESULT: UNSTABLE ❕
• TEST FAILURES:

      1 org.opensearch.remotestore.SegmentReplicationUsingRemoteStoreIT.testMultipleShards

• URL: https://build.ci.opensearch.org/job/gradle-check/27171/
• CommitID: 2b80149
  Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

#9712

[andrross]

Just some annoying test nitpicks that can be followed up (or ignored) in a subsequent PR

[opensearch-trigger-bot]

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Navigate to the root of your repository
cd $(git rev-parse --show-toplevel)
# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/OpenSearch/backport-2.x 2.x
# Navigate to the new working tree
pushd ../.worktrees/OpenSearch/backport-2.x
# Create a new branch
git switch --create backport/backport-10370-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 651a9aa2ca05244e74bcaabcc89d878977efe259
# Push it to GitHub
git push --set-upstream origin backport/backport-10370-to-2.x
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/OpenSearch/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-10370-to-2.x.