Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Search: Support of wildcard on docvalue_fields #32980

Merged
merged 2 commits into from
Aug 23, 2018
Merged

Search: Support of wildcard on docvalue_fields #32980

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
e307323
Search: Support of wildcard on docvalue_fields
iverase Aug 20, 2018
da57451
Addresses review comments and update documentation with the new
iverase Aug 20, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 21 additions & 0 deletions docs/reference/search/request/docvalue-fields.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,27 @@ GET /_search

Doc value fields can work on fields that are not stored.

`*` can be used as a wild card, for example:

[source,js]
--------------------------------------------------
GET /_search
{
"query" : {
"match_all": {}
},
"docvalue_fields" : [
{
"field": "*field", <1>
"format": "use_field_mapping" <2>
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This option is deprecated in 7.x so I suspect this will make the docs tests fail. We could do just docvalue_fields: [ "*field" ].

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shall I update the docs or this should be addressed in a different issue?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry this is my bad, I assumed that #30831 had been merged, which is not true. There is nothing to fix in your PR.

}
]
}
--------------------------------------------------
// CONSOLE
<1> Match all fields ending with `field`
<2> Format to be applied to all matching fields.

Note that if the fields parameter specifies fields without docvalues it will try to load the value from the fielddata cache
causing the terms for that field to be loaded to memory (cached), which will result in more memory consumption.

Expand Down
19 changes: 14 additions & 5 deletions server/src/main/java/org/elasticsearch/search/SearchService.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -98,6 +98,8 @@
import org.elasticsearch.transport.TransportRequest;

import java.io.IOException;
import java.util.ArrayList;
import java.util.Collection;
import java.util.Collections;
import java.util.HashMap;
import java.util.List;
Expand Down Expand Up @@ -789,14 +791,21 @@ private void parseSource(DefaultSearchContext context, SearchSourceBuilder sourc
context.fetchSourceContext(source.fetchSource());
}
if (source.docValueFields() != null) {
List<DocValueFieldsContext.FieldAndFormat> docValueFields = new ArrayList<>();
for (DocValueFieldsContext.FieldAndFormat format : source.docValueFields()) {
Collection<String> fieldNames = context.mapperService().simpleMatchToFullName(format.field);
for (String fieldName: fieldNames) {
docValueFields.add(new DocValueFieldsContext.FieldAndFormat(fieldName, format.format));
}
}
int maxAllowedDocvalueFields = context.mapperService().getIndexSettings().getMaxDocvalueFields();
if (source.docValueFields().size() > maxAllowedDocvalueFields) {
if (docValueFields.size() > maxAllowedDocvalueFields) {
throw new IllegalArgumentException(
"Trying to retrieve too many docvalue_fields. Must be less than or equal to: [" + maxAllowedDocvalueFields
+ "] but was [" + source.docValueFields().size() + "]. This limit can be set by changing the ["
+ IndexSettings.MAX_DOCVALUE_FIELDS_SEARCH_SETTING.getKey() + "] index level setting.");
"Trying to retrieve too many docvalue_fields. Must be less than or equal to: [" + maxAllowedDocvalueFields
+ "] but was [" + docValueFields.size() + "]. This limit can be set by changing the ["
+ IndexSettings.MAX_DOCVALUE_FIELDS_SEARCH_SETTING.getKey() + "] index level setting.");
}
context.docValueFieldsContext(new DocValueFieldsContext(source.docValueFields()));
context.docValueFieldsContext(new DocValueFieldsContext(docValueFields));
}
if (source.highlighter() != null) {
HighlightBuilder highlightBuilder = source.highlighter();
Expand Down
90 changes: 90 additions & 0 deletions server/src/test/java/org/elasticsearch/search/fields/SearchFieldsIT.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -810,6 +810,32 @@ public void testDocValueFields() throws Exception {
equalTo(new BytesRef(new byte[] {42, 100})));
assertThat(searchResponse.getHits().getAt(0).getFields().get("ip_field").getValue(), equalTo("::1"));

builder = client().prepareSearch().setQuery(matchAllQuery())
.addDocValueField("*field");
searchResponse = builder.execute().actionGet();

assertThat(searchResponse.getHits().getTotalHits(), equalTo(1L));
assertThat(searchResponse.getHits().getHits().length, equalTo(1));
fields = new HashSet<>(searchResponse.getHits().getAt(0).getFields().keySet());
assertThat(fields, equalTo(newHashSet("byte_field", "short_field", "integer_field", "long_field",
"float_field", "double_field", "date_field", "boolean_field", "text_field", "keyword_field",
"binary_field", "ip_field")));

assertThat(searchResponse.getHits().getAt(0).getFields().get("byte_field").getValue().toString(), equalTo("1"));
assertThat(searchResponse.getHits().getAt(0).getFields().get("short_field").getValue().toString(), equalTo("2"));
assertThat(searchResponse.getHits().getAt(0).getFields().get("integer_field").getValue(), equalTo((Object) 3L));
assertThat(searchResponse.getHits().getAt(0).getFields().get("long_field").getValue(), equalTo((Object) 4L));
assertThat(searchResponse.getHits().getAt(0).getFields().get("float_field").getValue(), equalTo((Object) 5.0));
assertThat(searchResponse.getHits().getAt(0).getFields().get("double_field").getValue(), equalTo((Object) 6.0d));
dateField = searchResponse.getHits().getAt(0).getFields().get("date_field").getValue();
assertThat(dateField.toInstant().toEpochMilli(), equalTo(date.toInstant().toEpochMilli()));
assertThat(searchResponse.getHits().getAt(0).getFields().get("boolean_field").getValue(), equalTo((Object) true));
assertThat(searchResponse.getHits().getAt(0).getFields().get("text_field").getValue(), equalTo("foo"));
assertThat(searchResponse.getHits().getAt(0).getFields().get("keyword_field").getValue(), equalTo("foo"));
assertThat(searchResponse.getHits().getAt(0).getFields().get("binary_field").getValue(),
equalTo(new BytesRef(new byte[] {42, 100})));
assertThat(searchResponse.getHits().getAt(0).getFields().get("ip_field").getValue(), equalTo("::1"));

builder = client().prepareSearch().setQuery(matchAllQuery())
.addDocValueField("text_field", "use_field_mapping")
.addDocValueField("keyword_field", "use_field_mapping")
Expand Down Expand Up @@ -977,6 +1003,70 @@ public void testDocValueFieldsWithFieldAlias() throws Exception {
assertThat(fetchedDate, equalTo(date));
}

public void testWildcardDocValueFieldsWithFieldAlias() throws Exception {
XContentBuilder mapping = XContentFactory.jsonBuilder()
.startObject()
.startObject("type")
.startObject("_source")
.field("enabled", false)
.endObject()
.startObject("properties")
.startObject("text_field")
.field("type", "text")
.field("fielddata", true)
.endObject()
.startObject("date_field")
.field("type", "date")
.field("format", "yyyy-MM-dd")
.endObject()
.startObject("text_field_alias")
.field("type", "alias")
.field("path", "text_field")
.endObject()
.startObject("date_field_alias")
.field("type", "alias")
.field("path", "date_field")
.endObject()
.endObject()
.endObject()
.endObject();
assertAcked(prepareCreate("test").addMapping("type", mapping));
ensureGreen("test");

ZonedDateTime date = ZonedDateTime.of(1990, 12, 29, 0, 0, 0, 0, ZoneOffset.UTC);
DateTimeFormatter formatter = DateTimeFormatter.ofPattern("yyyy-MM-dd", Locale.ROOT);

index("test", "type", "1", "text_field", "foo", "date_field", formatter.format(date));
refresh("test");

SearchRequestBuilder builder = client().prepareSearch().setQuery(matchAllQuery())
.addDocValueField("*alias", "use_field_mapping")
.addDocValueField("date_field");
SearchResponse searchResponse = builder.execute().actionGet();

assertNoFailures(searchResponse);
assertHitCount(searchResponse, 1);
SearchHit hit = searchResponse.getHits().getAt(0);

Map<String, DocumentField> fields = hit.getFields();
assertThat(fields.keySet(), equalTo(newHashSet("text_field_alias", "date_field_alias", "date_field")));

DocumentField textFieldAlias = fields.get("text_field_alias");
assertThat(textFieldAlias.getName(), equalTo("text_field_alias"));
assertThat(textFieldAlias.getValue(), equalTo("foo"));

DocumentField dateFieldAlias = fields.get("date_field_alias");
assertThat(dateFieldAlias.getName(), equalTo("date_field_alias"));
assertThat(dateFieldAlias.getValue(),
equalTo("1990-12-29"));

DocumentField dateField = fields.get("date_field");
assertThat(dateField.getName(), equalTo("date_field"));

ZonedDateTime fetchedDate = dateField.getValue();
assertThat(fetchedDate, equalTo(date));
}


public void testStoredFieldsWithFieldAlias() throws Exception {
XContentBuilder mapping = XContentFactory.jsonBuilder()
Expand Down