Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Do NOT allow termvectors on nested fields #32728

Merged
merged 2 commits into from
Aug 23, 2018
Merged

Do NOT allow termvectors on nested fields #32728

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
1387c18
Do not allow termvectors on nested fields
mayya-sharipova Aug 8, 2018
bb9bd7c
Address feedback
mayya-sharipova Aug 10, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions docs/reference/docs/termvectors.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,10 @@ in similar way to the <<query-dsl-multi-match-query,multi match query>>
[WARNING]
Note that the usage of `/_termvector` is deprecated in 2.0, and replaced by `/_termvectors`.

[WARNING]
Term Vectors API doesn't work on nested fields. `/_termvectors` on a nested
field and any sub-fields of a nested field returns empty results.

[float]
=== Return values

Expand Down
49 changes: 49 additions & 0 deletions rest-api-spec/src/main/resources/rest-api-spec/test/termvectors/50_nested.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,49 @@
setup:
- do:
indices.create:
index: testidx
body:
mappings:
_doc:
properties:
nested1:
type : nested
properties:
nested1-text:
type: text
object1:
properties:
object1-text:
type: text
object1-nested1:
type: nested
properties:
object1-nested1-text:
type: text
- do:
index:
index: testidx
type: _doc
id: 1
body:
"nested1" : [{ "nested1-text": "text1" }]
"object1" : [{ "object1-text": "text2" }, "object1-nested1" : [{"object1-nested1-text" : "text3"}]]

- do:
indices.refresh: {}

---
"Termvectors on nested fields should return empty results":

- do:
termvectors:
index: testidx
type: _doc
id: 1
fields: ["nested1", "nested1.nested1-text", "object1.object1-nested1", "object1.object1-nested1.object1-nested1-text", "object1.object1-text"]

- is_false: term_vectors.nested1
- is_false: term_vectors.nested1\.nested1-text # escaping as the field name contains dot
- is_false: term_vectors.object1\.object1-nested1
- is_false: term_vectors.object1\.object1-nested1\.object1-nested1-text
- is_true: term_vectors.object1\.object1-text
17 changes: 14 additions & 3 deletions server/src/main/java/org/elasticsearch/index/termvectors/TermVectorsService.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,7 @@
import org.elasticsearch.index.mapper.KeywordFieldMapper;
import org.elasticsearch.index.mapper.MappedFieldType;
import org.elasticsearch.index.mapper.MapperService;
import org.elasticsearch.index.mapper.ObjectMapper;
import org.elasticsearch.index.mapper.ParseContext;
import org.elasticsearch.index.mapper.ParsedDocument;
import org.elasticsearch.index.mapper.SourceFieldMapper;
Expand Down Expand Up @@ -160,7 +161,7 @@ private static void handleFieldWildcards(IndexShard indexShard, TermVectorsReque
request.selectedFields(fieldNames.toArray(Strings.EMPTY_ARRAY));
}

private static boolean isValidField(MappedFieldType fieldType) {
private static boolean isValidField(MappedFieldType fieldType, IndexShard indexShard) {
// must be a string
if (fieldType instanceof StringFieldType == false) {
return false;
Expand All @@ -169,6 +170,16 @@ private static boolean isValidField(MappedFieldType fieldType) {
if (fieldType.indexOptions() == IndexOptions.NONE) {
return false;
}
// and must not be under nested field
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should check all levels in the path, for instance in foo.bar.baz, foo can be a simple object field and foo.bar a nested field.

int dotIndex = fieldType.name().indexOf('.');
while (dotIndex > -1) {
String parentField = fieldType.name().substring(0, dotIndex);
ObjectMapper mapper = indexShard.mapperService().getObjectMapper(parentField);
if (mapper != null && mapper.nested().isNested()) {
return false;
}
dotIndex = fieldType.name().indexOf('.', dotIndex + 1);
}
return true;
}

Expand All @@ -177,7 +188,7 @@ private static Fields addGeneratedTermVectors(IndexShard indexShard, Engine.GetR
Set<String> validFields = new HashSet<>();
for (String field : selectedFields) {
MappedFieldType fieldType = indexShard.mapperService().fullName(field);
if (!isValidField(fieldType)) {
if (isValidField(fieldType, indexShard) == false) {
continue;
}
// already retrieved, only if the analyzer hasn't been overridden at the field
Expand Down Expand Up @@ -284,7 +295,7 @@ private static Fields generateTermVectorsFromDoc(IndexShard indexShard, TermVect
Collection<DocumentField> documentFields = new HashSet<>();
for (IndexableField field : doc.getFields()) {
MappedFieldType fieldType = indexShard.mapperService().fullName(field.name());
if (!isValidField(fieldType)) {
if (isValidField(fieldType, indexShard) == false) {
continue;
}
if (request.selectedFields() != null && !request.selectedFields().contains(field.name())) {
Expand Down