[Segment Replication] Update getLatestReplicationCheckpoint for closed indices #4532
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…ndices Signed-off-by: Suraj Singh <surajrider@gmail.com>
dreamer-89
changed the title
Signed-off-by: Suraj Singh <surajrider@gmail.com>
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
mch2
reviewed
Sep 16, 2022
| @@ -1405,7 +1405,8 @@ public ReplicationCheckpoint getLatestReplicationCheckpoint() { | |||
| if (indexSettings.isSegRepEnabled() == false) { | |||
| return null; | |||
| } | |||
| if (getEngineOrNull() == null) { | |||
| final IndexMetadata indexMetadata = indexSettings.getIndexMetadata(); | |||
| if (getEngineOrNull() == null || (indexMetadata != null && indexMetadata.getState() == IndexMetadata.State.CLOSE)) { | |||
| return ReplicationCheckpoint.empty(shardId); | |||
There was a problem hiding this comment.
I'm not sure we have a use case for actually using these checkpoints when the shard is closed, but even so I think we may want to change this logic to actually read and return the current checkpoint instead of returning empty. A shard could be closed but have a local commit. We can check for the presence of a segments_n and compute the checkpoint from the store with store.readLastCommittedSegmentsInfo. Where the processed seqNo can be read from user data.
There was a problem hiding this comment.
Thanks @mch2 for the review and calling this out. Updating.
server/src/main/java/org/opensearch/index/shard/IndexShard.java
Outdated
Show resolved
Hide resolved
dreamer-89
force-pushed
the
update_getCheckpoint
branch
2 times, most recently
from
9ab5caf
to
2759e47
Compare
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
Signed-off-by: Suraj Singh <surajrider@gmail.com>
dreamer-89
force-pushed
the
update_getCheckpoint
branch
from
2759e47
to
703427f
Compare
This comment was marked as outdated.
This comment was marked as outdated.
Gradle Check (Jenkins) Run Completed with:
|
Codecov Report
@@ Coverage Diff @@
## main #4532 +/- ##
============================================
+ Coverage 70.62% 70.74% +0.12%
- Complexity 57263 57368 +105
============================================
Files 4617 4617
Lines 275499 275509 +10
Branches 40331 40332 +1
============================================
+ Hits 194562 194909 +347
+ Misses 64528 64259 -269
+ Partials 16409 16341 -68
Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here. |
dreamer-89
changed the title
|
@mch2 : Addressed the review comments, can you please have a look ? |
Gradle Check (Jenkins) Run Completed with:
|
|
Flaky test tracked in #4162 |
Gradle Check (Jenkins) Run Completed with:
|
mch2
reviewed
Sep 21, 2022
| // If index is closed, read replication checkpoint from last committed segments info on local store | ||
| if (indexSettings.getIndexMetadata().getState() == IndexMetadata.State.CLOSE) { | ||
| try { | ||
| final SegmentInfos segmentInfos = store.readLastCommittedSegmentsInfo(); |
There was a problem hiding this comment.
How will this behave if the engine is not yet open? I think we need to check for the existence of a commit first. This method will throw if one is not present.
|
@dreamer-89 Is this still required? it's pending for review for a while |
@anasalkouz : Yes, this is needed and appears while running ClosedIndices* integration tests. This was initially needed for CCR (as mentioned in description) but now appears to be a valid use case related to closed indices. I will work on this PR. |
|
Closing this stale PR, tracking fix in #6828 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Signed-off-by: Suraj Singh surajrider@gmail.com
Description
This change updates the IndexShard.getLatestReplicationCheckpoint to return empty checkpoint for closed indices. This change is needed as when indices are closed, shard info is still shared among nodes via transport actions as mentioned in #3823 (comment)
Issues Related
#3823
Check List
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.