[Backport 2.x] [Remote Store] Add index specific setting for remote repository (#4253)

[sachinpkale]

Signed-off-by: Sachin Kale kalsac@amazon.com

Description

PR link to main: #4253

Bug #1

• Currently, remote store implementation assumes repository with cluster UUID to be created before creating an index and uses this repository to store/retrieve translog and segments.
• Accessing cluster UUID in IndexService fails one of the assertions which prevent accessing cluster UUID in the same flow where we change the cluster state. More details here.
• In this change, we pass repository ID while creating the index.
• With this change, following things are achieved:
  • No conflicts. With current implementation, there is a possibility of existence of repository with same name as cluster UUID. With new approach, there will not be such conflicts.
  • User can create different repositories for different indices.
  • No assertion failure.

Bug #2

• When remote store is enabled for an index, on fail-over, expected behavior is, new primary will start uploading new segments (which are not already uploaded by old primary) to the remote segment store. Currently, all the segments from new primary are getting uploaded.
• This happens when RemoteSegmentStoreDirectory is having stale state of uploaded segments.
  Solution: Call RemoteSegmentStoreDirectory.init() on failover.

Issues Resolved

• [BUG] [Remote Store] AssertionError while creating an index with index.remote_store.enabled=true #4233
• [BUG][Remote Segment Store] On failover, new primary uploads all the segments to remote store, it should be incremental #4398

Check List

• New functionality includes testing.
  • All tests pass
• New functionality has been documented.
  • New functionality has javadoc added
• Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: SUCCESS ✅
• URL: https://build.ci.opensearch.org/job/gradle-check/2697/
• CommitID: eceb260

[codecov-commenter]

Codecov Report

Merging #4418 (eceb260) into 2.x (fc97324) will decrease coverage by 0.02%.
The diff coverage is 72.41%.

@@             Coverage Diff              @@
##                2.x    #4418      +/-   ##
============================================
- Coverage     70.68%   70.65%   -0.03%     
+ Complexity    57182    57149      -33     
============================================
  Files          4584     4584              
  Lines        274497   274522      +25     
  Branches      40226    40229       +3     
============================================
- Hits         194018   193960      -58     
- Misses        64218    64385     +167     
+ Partials      16261    16177      -84     

Impacted Files	Coverage Δ
...c/main/java/org/opensearch/index/IndexService.java	74.25% <0.00%> (+0.22%)	⬆️
...search/index/shard/RemoteStoreRefreshListener.java	78.31% <60.00%> (-1.18%)	⬇️
...org/opensearch/cluster/metadata/IndexMetadata.java	83.51% <68.75%> (-0.80%)	⬇️
...pensearch/common/settings/IndexScopedSettings.java	100.00% <100.00%> (ø)
...org/opensearch/common/settings/SettingsModule.java	86.17% <100.00%> (ø)
.../main/java/org/opensearch/index/IndexSettings.java	86.76% <100.00%> (+0.07%)	⬆️
...n/indices/forcemerge/ForceMergeRequestBuilder.java	0.00% <0.00%> (-75.00%)	⬇️
.../indices/forcemerge/TransportForceMergeAction.java	25.00% <0.00%> (-75.00%)	⬇️
...adonly/AddIndexBlockClusterStateUpdateRequest.java	0.00% <0.00%> (-75.00%)	⬇️
...readonly/TransportVerifyShardIndexBlockAction.java	9.75% <0.00%> (-73.18%)	⬇️
... and 496 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.