Error sorting optional field

[antoine777]

Linux RedHat 64bits

• Kibana 4 RC1
• ELS 1.4.3

After i added a new field (request_time) on existing type i'm getting this error message when i search for my events.

I think the problem is that my older indexes does not contain this new field. By inspecting the request, the error is returned by ELS because the new field is not considered optional for sorting the events

SearchParseException[[logstash-casext-tomcat-access-2015.02.16][0]: from[-1],size[500]: Parse Failure [Failed to parse source [{\"size\":500,\"sort\":{\"request_time\":\"asc\"},\"query\":{\"filtered\":{\"query\":{\"query_string\":{\"analyze_wildcard\":true,\"query\":\"*\"}},\"filter\":{\"bool\":{\"must\":[{\"range\":{\"@timestamp\":{\"gte\":1424643508461,\"lte\":1424686708461}}}],\"must_not\":[]}}}},\"highlight\":{\"pre_tags\":[\"@kibana-highlighted-field@\"],\"post_tags\":[\"@/kibana-highlighted-field@\"],\"fields\":{\"*\":{}}},\"aggs\":{\"2\":{\"date_histogram\":{\"field\":\"@timestamp\",\"interval\":\"10m\",\"pre_zone\":\"+01:00\",\"pre_zone_adjust_large_interval\":true,\"min_doc_count\":0,\"extended_bounds\":{\"min\":1424643508461,\"max\":1424686708461}}}},\"fields\":[\"*\",\"_source\"],\"script_fields\":{},\"fielddata_fields\":[\"@timestamp\"]}]]]; nested: SearchParseException[[logstash-casext-tomcat-access-2015.02.16][0]: from[-1],size[500]: Parse Failure [No mapping found for [request_time] in order to sort on]]; 

ELS stacktrace :

Caused by: org.elasticsearch.search.SearchParseException: [logstash-casext-tomcat-access-2015.02.18][4]: from[-1],size[500]: Parse Failure [No mapping found for [request_time] in order to sort on]
at org.elasticsearch.search.sort.SortParseElement.addSortField(SortParseElement.java:210)
at org.elasticsearch.search.sort.SortParseElement.addCompoundSortField(SortParseElement.java:141)
at org.elasticsearch.search.sort.SortParseElement.parse(SortParseElement.java:96)
at org.elasticsearch.search.SearchService.parseSource(SearchService.java:671)

[rashidkpc]

I'd recommend opening a ticket on the elasticsearch/elasticsearch repo for this one. Unfortunately there's not much kibana can do about this. When we glob the indices elasticsearch will scatter the request to all of the shards for all of them without regard for if the sort is possible, thus causing failures. Especially with something-* patterns, we have no way of telling elasticsearch not to try to sort data in those indices.

[antoine777]

I'm not sure but would it possible for kibana to use the missing parameter or the unmapped_type parameter as mentioned in ELS doc
http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-request-sort.html

[jdve]

Yeah, kibana4 needs about the same thing as #1153 was for kibana3. It would be cool if this was in the advanced section, much like query:queryString:options, so that these parameters could be tweaked. Maybe an option named query:sort:options?