-
Notifications
You must be signed in to change notification settings - Fork 133
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Provide hybrid scan setting for consistency requirement #1819
Provide hybrid scan setting for consistency requirement #1819
Conversation
Signed-off-by: Chen Dai <daichen@amazon.com>
Signed-off-by: Chen Dai <daichen@amazon.com>
Signed-off-by: Chen Dai <daichen@amazon.com>
Codecov Report
@@ Coverage Diff @@
## feature/flint opensearch-project/sql#1819 +/- ##
================================================
Coverage 97.19% 97.19%
Complexity 4107 4107
================================================
Files 371 371
Lines 10464 10464
Branches 706 706
================================================
Hits 10170 10170
Misses 287 287
Partials 7 7
Flags with carried forward coverage won't be shown. Click here to find out more. |
partitions | ||
.flatMap(_.files.map(f => f.getPath.toUri.toString)) | ||
.toDF(FILE_PATH_COLUMN) | ||
.join(indexScan, Seq(FILE_PATH_COLUMN), "left") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
i assume left join is expensive, is there an magic number to avoid left join, which mean scan without index? if it make sense, we can add an issue to track it at perf test stage.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sure, will cover this in our benchmark after merged. Thanks!
Description
Add hybrid scan mode which covers the latest source files that haven't refreshed to Flint index yet:
Issues Resolved
opensearch-project/opensearch-spark#2
Check List
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.