Add FieldExpansionIterator that replaces the ANYFIELD scanner #2660
Conversation
warehouse/query-core/src/main/java/datawave/query/data/parsers/ShardIndexKey.java
Outdated
Show resolved
Hide resolved
warehouse/query-core/src/main/java/datawave/query/jexl/lookups/FieldExpansionIndexLookup.java
Show resolved
Hide resolved
warehouse/query-core/src/main/java/datawave/core/iterators/FieldExpansionIterator.java
Show resolved
Hide resolved
| if (date.compareTo(startDate) < 0) { | ||
| // advance to start date | ||
| log.trace("Key before start date: {} < {}", date, startDate); | ||
| return new FilterResult(false, AdvanceResult.NEXT); |
There was a problem hiding this comment.
Instead of NEXT_CQ maybe this should be a custom hint that triggers a seek to the start date..
warehouse/query-core/src/main/java/datawave/query/data/parsers/ShardIndexKey.java
Show resolved
Hide resolved
warehouse/query-core/src/main/java/datawave/query/data/parsers/ShardIndexKey.java
Show resolved
Hide resolved
| public FieldExpansionIndexLookup(ShardQueryConfiguration config, ScannerFactory scannerFactory, String term, Set<String> fields, | ||
| ExecutorService execService) { | ||
| super(config, scannerFactory, true, execService); | ||
| this.term = term.toLowerCase(); |
There was a problem hiding this comment.
why lower case here? normalizers do not necessarily lower case values.
There was a problem hiding this comment.
Right you are...I need to revisit this case. The previous implementation was doing a just-in-time expansion for normalization which implies the ExpandMultiNormalizedTerms doesn't handle unfielded terms well..
| } | ||
|
|
||
| if (datatypes != null && !datatypes.contains(parser.getDatatype())) { | ||
| // figure out if we are before the first datatype -- seek to first datatype |
There was a problem hiding this comment.
This comment does not seem relevant here
There was a problem hiding this comment.
can we seek to the next datatype?
There was a problem hiding this comment.
Due to the structure of the shard index keys a seek based on the datatype is never practical. I'll correct the comment.
Add iterator that expands fields for literal values.