Use delCount of SegmentInfos to calculate numDocs #36323
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Today, we iterate the bitset of hardLiveDocs to calculate the number of live docs. This calculation might be expensive if we enable soft-deletes (by default) for old indices whose soft-deletes was disabled previously and had hard-deletes. Once soft-deletes is enabled, we no longer hard-update or hard-delete documents directly. We have hard-deletes in two scenarios: (1) from old segments where soft-deletes was disabled, (2) when IndexWriter hits non-aborted exceptions. These two cases, IW flushes SegmentInfos before exposing the hard-deletes; thus we can use the hard-delete count of SegmentInfos.
dnhatn
added
>enhancement
v7.0.0
:Distributed Indexing/Engine
|
Pinging @elastic/es-distributed |
jpountz
approved these changes
Dec 7, 2018
dnhatn
added a commit
that referenced
this pull request
Dec 9, 2018
Today, we iterate the bitset of hardLiveDocs to calculate the number of live docs. This calculation might be expensive if we enable soft-deletes (by default) for old indices whose soft-deletes was disabled previously and had hard-deletes. Once soft-deletes is enabled, we no longer hard-update or hard-delete documents directly. We have hard-deletes in two scenarios: (1) from old segments where soft-deletes was disabled, (2) when IndexWriter hits non-aborted exceptions. These two cases, IW flushes SegmentInfos before exposing the hard-deletes; thus we can use the hard-delete count of SegmentInfos.
jasontedor
added a commit
to liketic/elasticsearch
that referenced
this pull request
Dec 9, 2018
* elastic/6.x: (37 commits) [HLRC] Added support for Follow Stats API (elastic#36253) Exposed engine must have all ops below gcp during rollback (elastic#36159) TEST: Always enable soft-deletes in ShardChangesTests Use delCount of SegmentInfos to calculate numDocs (elastic#36323) Add soft-deletes upgrade tests (elastic#36286) Remove LocalCheckpointTracker#resetCheckpoint (elastic#34667) Option to use endpoints starting with _security (elastic#36379) [CCR] Restructured QA modules (elastic#36404) RestClient: on retry timeout add root exception (elastic#25576) [HLRC] Add support for put privileges API (elastic#35679) HLRC: Add rollup search (elastic#36334) Explicitly recommend to forceMerge before freezing (elastic#36376) Rename internal repository actions to be internal (elastic#36377) Core: Remove parseDefaulting from DateFormatter (elastic#36386) [ML] Prevent stack overflow while copying ML jobs and datafeeds (elastic#36370) Docs: Fix Jackson reference (elastic#36366) [ILM] Fix issue where index may not yet be in 'hot' phase (elastic#35716) Undeprecate /_watcher endpoints (elastic#36269) Docs: Fix typo in bool query (elastic#36350) HLRC: Add delete template API (elastic#36320) ...
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Today, we iterate the bitset of hardLiveDocs to calculate the number of
live docs. This calculation might be expensive if we enable soft-deletes
(by default) for old indices whose soft-deletes was disabled previously
and had hard-deletes.
Once soft-deletes is enabled, we no longer hard-update or hard-delete
documents directly. We have hard-deletes in two scenarios: (1) from old
segments where soft-deletes was disabled, (2) when IndexWriter hits
non-aborted exceptions. These two cases, IW flushes SegmentInfos before
exposing the hard-deletes; thus we can use the hard-delete count of
SegmentInfos.