[HUDI-3760] Adding capability to fetch Metadata Records by prefix #5208

Merged

nsivabalan merged 73 commits intoapache:masterfrom onehouseinc:ak/mtd-kpfx

Apr 6, 2022

+1,356-667

Commits on Apr 6, 2022

Added getRecordsByKeyPrefix to HoodieHFileReader
Alexey Kudinkin
committed
Tidying up
Alexey Kudinkin
committed
Added getRecordIteratorByKeyPrefix to HoodieHFileReader;
Alexey Kudinkin
committed
Made HoodieDataBlock accept keys specified as prefixes;
Alexey Kudinkin
committed
Tidying up
Alexey Kudinkin
committed
Modified AbstractHoodieLogRecordReader to be able to scan records based on key-prefix rather than the full-key;
Alexey Kudinkin
committed
Adjusted HoodieFileReader interface
Alexey Kudinkin
committed
Add HoodieBackedTableMetadata::getRecordsByKeyPrefixes
Alexey Kudinkin
committed
Added test for HFileReader seq fetching records by key-prefixes
Alexey Kudinkin
committed
Fixing compilation
Alexey Kudinkin
committed
Fixing read from Metadata table for col stats partition

nsivabalan
authored and
Alexey Kudinkin
committed
Fixing prefix lookup in HFile reader and enhancing tests for the same

nsivabalan
authored and
Alexey Kudinkin
committed
Adding test to TestHoodieBackedMetadata to test prefix look up in col stats partition in Metadata table

nsivabalan
authored and
Alexey Kudinkin
committed
Fixing determining col stats partition availability in dataskipping code path

nsivabalan
authored and
Alexey Kudinkin
committed
Fixing compilation error in test with rebase

nsivabalan
authored and
Alexey Kudinkin
committed
Fixing HFileScanner usage and HoodieData for prefix based look up with metadata table

nsivabalan
authored and
Alexey Kudinkin
committed
fixing arg name for useCachedReaders

nsivabalan
authored and
Alexey Kudinkin
committed
Fixing test failures related to write schema not added to Hfile on the write path

nsivabalan
authored and
Alexey Kudinkin
committed
Extracted common utils
Alexey Kudinkin
committed
Rebased ColumnStatsIndexSupport to fetch CSI records by key-prefix instead of reading the whole table (if available)
Alexey Kudinkin
committed
Tidying up
Alexey Kudinkin
committed
Rebased HoodieFileIndex onto updated API
Alexey Kudinkin
committed
Fixed HoodieDataBlock to properly handle when records are fetched by key-prefixes
Alexey Kudinkin
committed
Fixed HoodieBackedTableMetadata to avoid doing full-scans
Alexey Kudinkin
committed
Fixed tests
Alexey Kudinkin
committed
Fixing compilation
Alexey Kudinkin
committed
Fixed iterator to properly loop
Alexey Kudinkin
committed
Fixed incorrect key-prefix access gen
Alexey Kudinkin
committed
Tidying up
Alexey Kudinkin
committed
Removed references to HFile from HoodieFileReader;
Alexey Kudinkin
committed
Cleaned up HoodieFileReader APIs;
Alexey Kudinkin
committed
Cleaned up HoodieHFileReader APIs;
Alexey Kudinkin
committed
Fixing compilation
Alexey Kudinkin
committed
lint
Alexey Kudinkin
committed
Removed APIs which dereference iterators from HoodieFileReader, instead inlining them where used
Alexey Kudinkin
committed
Fixing compilation
Alexey Kudinkin
committed
Fixed records iterators
Alexey Kudinkin
committed
Fixed tests
Alexey Kudinkin
committed
Fixing some more
Alexey Kudinkin
committed
Fixing a little more
Alexey Kudinkin
committed
Fixing task being non-serializable
Alexey Kudinkin
committed
Tidying up
Alexey Kudinkin
committed
Consolidated configuration around forceFullScan property for LogReaders
Alexey Kudinkin
committed
Adding assertions
Alexey Kudinkin
committed
Test with all log-files scan modes
Alexey Kudinkin
committed
Killed dead-code
Alexey Kudinkin
committed
Fixing incorrect sorting of the keys
Alexey Kudinkin
committed
Fixing tests
Alexey Kudinkin
committed
Tidying up
Alexey Kudinkin
committed
Cleaned up ColumnStatsIndexSupport removing duplication
Alexey Kudinkin
committed
Fixed schemas used to read from MT, to be fetched from HFile
Alexey Kudinkin
committed
Fixed TestHoodieFileIndex
Alexey Kudinkin
committed
Tidying up
Alexey Kudinkin
committed
Disallow full-scans for "column_stats", "bloom_filters" partitions
Alexey Kudinkin
committed
Killed fsDataInputStream
Alexey Kudinkin
committed
Rebased schema field to become final, thread-safe
Alexey Kudinkin
committed
Tidying up
Alexey Kudinkin
committed
Tidying up
Alexey Kudinkin
committed
Added sharedScanner instance to be re-used by point-lookup queries
Alexey Kudinkin
committed
Fixed RecordIterator, RecordByKeysIterator
Alexey Kudinkin
committed
Fixed RecordByKeyPrefixIterator
Alexey Kudinkin
committed
Tidying up
Alexey Kudinkin
committed
Missing license
Alexey Kudinkin
committed
Fixing tests
Alexey Kudinkin
committed
Fixed RecordByKeyPrefixIterator to properly handle EOF
Alexey Kudinkin
committed
Rebased HoodieMergeOnReadRDD to use Metadata Config from the File Index
Alexey Kudinkin
committed
Fixed opts for MT reads
Alexey Kudinkin
committed
Force MT full-scan when reading it from Spark DS;
Alexey Kudinkin
committed
Fixed query-type missing from config props in Spark Streaming
Alexey Kudinkin
committed
Added fallback to default val for QUERY_TYPE config in Spark File Index
Alexey Kudinkin
committed
Fixed TestLayoutOptimization missing required MT configs
Alexey Kudinkin
committed
Cleaned up duplicated methods
Alexey Kudinkin
committed
Close reader and remove public API from metadata payload
codope
committed