-
Notifications
You must be signed in to change notification settings - Fork 328
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat(mito): enable inverted index #3158
Merged
zhongzc
merged 11 commits into
GreptimeTeam:main
from
zhongzc:zhongzc/inverted-index-enable
Jan 15, 2024
Merged
feat(mito): enable inverted index #3158
zhongzc
merged 11 commits into
GreptimeTeam:main
from
zhongzc:zhongzc/inverted-index-enable
Jan 15, 2024
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>
7 tasks
… Engine Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>
Codecov ReportAttention:
Additional details and impacted files@@ Coverage Diff @@
## main #3158 +/- ##
==========================================
- Coverage 85.43% 85.09% -0.34%
==========================================
Files 823 829 +6
Lines 134922 135714 +792
==========================================
+ Hits 115268 115492 +224
- Misses 19654 20222 +568 |
zhongzc
commented
Jan 15, 2024
evenyag
reviewed
Jan 15, 2024
Co-authored-by: Yingwen <realevenyag@gmail.com>
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>
zhongzc
force-pushed
the
zhongzc/inverted-index-enable
branch
from
January 15, 2024 06:18
82322c2
to
6403d01
Compare
…to field of WriteCache Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>
fengjiachun
reviewed
Jan 15, 2024
evenyag
approved these changes
Jan 15, 2024
fengjiachun
approved these changes
Jan 15, 2024
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
I hereby agree to the terms of the GreptimeDB CLA
What's changed and what's your intention?
This PR has enabled the inverted index for the mito engine.
Main changes:
IntermediateManager
with the objectives:IntermediateManager
be a singleton to prevent repetitive deletions that could cause errors.IntermediateLocation
. Previously, the path was placed near the data files, but with the introduction of theIntermediateManager
, data files and intermediate files were completely isolated, hence a more customized path was adopted.Indexer
, embedded withinParquetWriter
.Indexer
is used to create the index and hides error handling internally, exposing three methods toParquetWriter
that do not return errors:update
,finish
, andabort
.InvertedIndexConfig
toMitoConfig
, which includes the following parameters:create_on_flush
,create_on_compaction
,apply_on_query
intermediate_path
: The file system path for intermediatesmem_threshold_on_create
: Memory control when creating the indexMitoConfig::sanitize
, takingdata_home
as an input because bothintermediate_path
andexperimental_write_cache_path
depend ondata_home
for setting default paths.ScanRegion
disables index apply during queries based on theapply_on_query
parameter.SstWriteRequest
introducescreate_inverted_index
andmem_threshold_index_create
. The decision to create an index during flush and compaction is controlled by passing the configs fromMitoConfig
to these two parameters.Checklist
Refer to a related PR or issue link (optional)
#2705