Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for field aliases. #32172

Merged
merged 9 commits into from
Jul 18, 2018
3 changes: 2 additions & 1 deletion docs/reference/indices/clearcache.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,8 @@ explicitly by setting `query`, `fielddata` or `request`.

All caches relating to a specific field(s) can also be cleared by
specifying `fields` parameter with a comma delimited list of the
relevant fields.
relevant fields. Note that the provided names must refer to concrete
fields -- objects and field aliases are not supported.

[float]
=== Multi Index
Expand Down
6 changes: 4 additions & 2 deletions docs/reference/mapping.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -124,8 +124,10 @@ fields to an existing index with the <<indices-put-mapping,PUT mapping API>>.

Other than where documented, *existing field mappings cannot be
updated*. Changing the mapping would mean invalidating already indexed
documents. Instead, you should create a new index with the correct mappings
and <<docs-reindex,reindex>> your data into that index.
documents. Instead, you should create a new index with the correct mappings
and <<docs-reindex,reindex>> your data into that index. If you only wish
to rename a field and not change its mappings, it may make sense to introduce
an <<alias, `alias`>> field.

[float]
== Example mapping
Expand Down
4 changes: 4 additions & 0 deletions docs/reference/mapping/types.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,8 @@ string:: <<text,`text`>> and <<keyword,`keyword`>>

<<parent-join>>:: Defines parent/child relation for documents within the same index

<<alias>>:: Defines an alias to an existing field.

<<feature>>:: Record numeric features to boost hits at query time.

<<feature-vector>>:: Record numeric feature vectors to boost hits at query time.
Expand All @@ -58,6 +60,8 @@ the <<analysis-standard-analyzer,`standard` analyzer>>, the
This is the purpose of _multi-fields_. Most datatypes support multi-fields
via the <<multi-fields>> parameter.

include::types/alias.asciidoc[]

include::types/array.asciidoc[]

include::types/binary.asciidoc[]
Expand Down
101 changes: 101 additions & 0 deletions docs/reference/mapping/types/alias.asciidoc
Original file line number Diff line number Diff line change
@@ -0,0 +1,101 @@
[[alias]]
=== Alias datatype

An `alias` mapping defines an alternate name for a field in the index.
The alias can be used in place of the target field in <<search, search>> requests,
and selected other APIs like <<search-field-caps, field capabilities>>.

[source,js]
--------------------------------
PUT trips
{
"mappings": {
"_doc": {
"properties": {
"distance": {
"type": "long"
},
"route_length_miles": {
"type": "alias",
"path": "distance" // <1>
},
"transit_mode": {
"type": "keyword"
}
}
}
}
}

GET _search
{
"query": {
"range" : {
"route_length_miles" : {
"gte" : 39
}
}
}
}
--------------------------------
// CONSOLE

<1> The path to the target field. Note that this must be the full path, including any parent
objects (e.g. `object1.object2.field`).

Almost all components of the search request accept field aliases. In particular, aliases can be
used in queries, aggregations, and sort fields, as well as when requesting `docvalue_fields`,
`stored_fields`, suggestions, and highlights. Scripts also support aliases when accessing
field values. Please see the section on <<unsupported-apis, unsupported APIs>> for exceptions.

In some parts of the search request and when requesting field capabilities, field wildcard patterns can be
provided. In these cases, the wildcard pattern will match field aliases in addition to concrete fields:

[source,js]
--------------------------------
GET trips/_field_caps?fields=route_*,transit_mode
--------------------------------
// CONSOLE
// TEST[continued]

[[alias-targets]]
==== Alias targets

There are a few restrictions on the target of an alias:

* The target must be a concrete field, and not an object or another field alias.
* The target field must exist at the time the alias is created.
* If nested objects are defined, a field alias must have the same nested scope as its target.

Additionally, a field alias can only have one target. This means that it is not possible to use a
field alias to query over multiple target fields in a single clause.

[[unsupported-apis]]
==== Unsupported APIs

Writes to field aliases are not supported: attempting to use an alias in an index or update request
will result in a failure. Likewise, aliases cannot be used as the target of `copy_to`.

Because alias names are not present in the document source, aliases cannot be used when performing
source filtering. For example, the following request will return an empty result for `_source`:

[source,js]
--------------------------------
GET /_search
{
"query" : {
"match_all": {}
},
"_source": "route_length_miles"
}
--------------------------------
// CONSOLE
// TEST[continued]

Currently only the search and field capabilities APIs will accept and resolve field aliases.
Other APIs that accept field names, such as <<docs-termvectors, term vectors>>, cannot be used
with field aliases.

Finally, some queries, such as `terms`, `geo_shape`, and `more_like_this`, allow for fetching query
information from an indexed document. Because field aliases aren't supported when fetching documents,
the part of the query that specifies the lookup path cannot refer to a field by its alias.
Original file line number Diff line number Diff line change
Expand Up @@ -20,27 +20,52 @@
package org.elasticsearch.script.expression;

import org.elasticsearch.common.settings.Settings;
import org.elasticsearch.index.IndexService;
import org.elasticsearch.index.query.QueryShardContext;
import org.elasticsearch.index.fielddata.AtomicNumericFieldData;
import org.elasticsearch.index.fielddata.IndexNumericFieldData;
import org.elasticsearch.index.fielddata.SortedNumericDoubleValues;
import org.elasticsearch.index.mapper.MapperService;
import org.elasticsearch.index.mapper.NumberFieldMapper.NumberFieldType;
import org.elasticsearch.index.mapper.NumberFieldMapper.NumberType;
import org.elasticsearch.script.ScriptException;
import org.elasticsearch.script.SearchScript;
import org.elasticsearch.search.lookup.SearchLookup;
import org.elasticsearch.test.ESSingleNodeTestCase;
import org.elasticsearch.test.ESTestCase;

import java.io.IOException;
import java.text.ParseException;
import java.util.Collections;

public class ExpressionTests extends ESSingleNodeTestCase {
ExpressionScriptEngine service;
SearchLookup lookup;
import static org.mockito.Matchers.anyInt;
import static org.mockito.Matchers.anyObject;
import static org.mockito.Mockito.mock;
import static org.mockito.Mockito.when;

public class ExpressionTests extends ESTestCase {
private ExpressionScriptEngine service;
private SearchLookup lookup;

@Override
public void setUp() throws Exception {
super.setUp();
IndexService index = createIndex("test", Settings.EMPTY, "type", "d", "type=double");

NumberFieldType fieldType = new NumberFieldType(NumberType.DOUBLE);
MapperService mapperService = mock(MapperService.class);
when(mapperService.fullName("field")).thenReturn(fieldType);
when(mapperService.fullName("alias")).thenReturn(fieldType);

SortedNumericDoubleValues doubleValues = mock(SortedNumericDoubleValues.class);
when(doubleValues.advanceExact(anyInt())).thenReturn(true);
when(doubleValues.nextValue()).thenReturn(2.718);

AtomicNumericFieldData atomicFieldData = mock(AtomicNumericFieldData.class);
when(atomicFieldData.getDoubleValues()).thenReturn(doubleValues);

IndexNumericFieldData fieldData = mock(IndexNumericFieldData.class);
when(fieldData.getFieldName()).thenReturn("field");
when(fieldData.load(anyObject())).thenReturn(atomicFieldData);

service = new ExpressionScriptEngine(Settings.EMPTY);
QueryShardContext shardContext = index.newQueryShardContext(0, null, () -> 0, null);
lookup = new SearchLookup(index.mapperService(), shardContext::getForField, null);
lookup = new SearchLookup(mapperService, ignored -> fieldData, null);
}

private SearchScript.LeafFactory compile(String expression) {
Expand All @@ -50,22 +75,38 @@ private SearchScript.LeafFactory compile(String expression) {

public void testNeedsScores() {
assertFalse(compile("1.2").needs_score());
assertFalse(compile("doc['d'].value").needs_score());
assertFalse(compile("doc['field'].value").needs_score());
assertTrue(compile("1/_score").needs_score());
assertTrue(compile("doc['d'].value * _score").needs_score());
assertTrue(compile("doc['field'].value * _score").needs_score());
}

public void testCompileError() {
ScriptException e = expectThrows(ScriptException.class, () -> {
compile("doc['d'].value * *@#)(@$*@#$ + 4");
compile("doc['field'].value * *@#)(@$*@#$ + 4");
});
assertTrue(e.getCause() instanceof ParseException);
}

public void testLinkError() {
ScriptException e = expectThrows(ScriptException.class, () -> {
compile("doc['e'].value * 5");
compile("doc['nonexistent'].value * 5");
});
assertTrue(e.getCause() instanceof ParseException);
}

public void testFieldAccess() throws IOException {
SearchScript script = compile("doc['field'].value").newInstance(null);
script.setDocument(1);

double result = script.runAsDouble();
assertEquals(2.718, result, 0.0);
}

public void testFieldAccessWithFieldAlias() throws IOException {
SearchScript script = compile("doc['alias'].value").newInstance(null);
script.setDocument(1);

double result = script.runAsDouble();
assertEquals(2.718, result, 0.0);
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -194,6 +194,10 @@ Query getCandidateMatchesQuery() {
return candidateMatchesQuery;
}

Query getVerifiedMatchesQuery() {
return verifiedMatchesQuery;
}

// Comparing identity here to avoid being cached
// Note that in theory if the same instance gets used multiple times it could still get cached,
// however since we create a new query instance each time we this query this shouldn't happen and thus
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -618,13 +618,13 @@ protected Analyzer getWrappedAnalyzer(String fieldName) {
docSearcher.setQueryCache(null);
}

PercolatorFieldMapper percolatorFieldMapper = (PercolatorFieldMapper) docMapper.mappers().getMapper(field);
boolean mapUnmappedFieldsAsString = percolatorFieldMapper.isMapUnmappedFieldAsText();
PercolatorFieldMapper.FieldType pft = (PercolatorFieldMapper.FieldType) fieldType;
String name = this.name != null ? this.name : pft.name();
QueryShardContext percolateShardContext = wrap(context);
PercolateQuery.QueryStore queryStore = createStore(pft.queryBuilderField,
percolateShardContext,
pft.mapUnmappedFieldsAsText);

String name = this.name != null ? this.name : field;
PercolatorFieldMapper.FieldType pft = (PercolatorFieldMapper.FieldType) fieldType;
PercolateQuery.QueryStore queryStore = createStore(pft.queryBuilderField, percolateShardContext, mapUnmappedFieldsAsString);
return pft.percolateQuery(name, queryStore, documents, docSearcher, context.indexVersionCreated());
}

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -136,13 +136,19 @@ public PercolatorFieldMapper build(BuilderContext context) {
fieldType.rangeField = rangeFieldMapper.fieldType();
NumberFieldMapper minimumShouldMatchFieldMapper = createMinimumShouldMatchField(context);
fieldType.minimumShouldMatchField = minimumShouldMatchFieldMapper.fieldType();
fieldType.mapUnmappedFieldsAsText = getMapUnmappedFieldAsText(context.indexSettings());

context.path().remove();
setupFieldType(context);
return new PercolatorFieldMapper(name(), fieldType, defaultFieldType, context.indexSettings(),
multiFieldsBuilder.build(this, context), copyTo, queryShardContext, extractedTermsField,
extractionResultField, queryBuilderField, rangeFieldMapper, minimumShouldMatchFieldMapper);
}

private static boolean getMapUnmappedFieldAsText(Settings indexSettings) {
return INDEX_MAP_UNMAPPED_FIELDS_AS_TEXT_SETTING.get(indexSettings);
}

static KeywordFieldMapper createExtractQueryFieldBuilder(String name, BuilderContext context) {
KeywordFieldMapper.Builder queryMetaDataFieldBuilder = new KeywordFieldMapper.Builder(name);
queryMetaDataFieldBuilder.docValues(false);
Expand Down Expand Up @@ -195,6 +201,7 @@ static class FieldType extends MappedFieldType {
MappedFieldType minimumShouldMatchField;

RangeFieldMapper.RangeFieldType rangeField;
boolean mapUnmappedFieldsAsText;

FieldType() {
setIndexOptions(IndexOptions.NONE);
Expand All @@ -209,6 +216,7 @@ static class FieldType extends MappedFieldType {
queryBuilderField = ref.queryBuilderField;
rangeField = ref.rangeField;
minimumShouldMatchField = ref.minimumShouldMatchField;
mapUnmappedFieldsAsText = ref.mapUnmappedFieldsAsText;
}

@Override
Expand Down Expand Up @@ -327,7 +335,6 @@ Tuple<List<BytesRef>, Map<String, List<byte[]>>> extractTermsAndRanges(IndexRead

}

private final boolean mapUnmappedFieldAsText;
private final Supplier<QueryShardContext> queryShardContext;
private KeywordFieldMapper queryTermsField;
private KeywordFieldMapper extractionResultField;
Expand All @@ -348,14 +355,9 @@ Tuple<List<BytesRef>, Map<String, List<byte[]>>> extractTermsAndRanges(IndexRead
this.extractionResultField = extractionResultField;
this.queryBuilderField = queryBuilderField;
this.minimumShouldMatchFieldMapper = minimumShouldMatchFieldMapper;
this.mapUnmappedFieldAsText = getMapUnmappedFieldAsText(indexSettings);
this.rangeFieldMapper = rangeFieldMapper;
}

private static boolean getMapUnmappedFieldAsText(Settings indexSettings) {
return INDEX_MAP_UNMAPPED_FIELDS_AS_TEXT_SETTING.get(indexSettings);
}

@Override
public FieldMapper updateFieldType(Map<String, MappedFieldType> fullNameToFieldType) {
PercolatorFieldMapper updated = (PercolatorFieldMapper) super.updateFieldType(fullNameToFieldType);
Expand Down Expand Up @@ -402,7 +404,7 @@ public Mapper parse(ParseContext context) throws IOException {

Version indexVersion = context.mapperService().getIndexSettings().getIndexVersionCreated();
createQueryBuilderField(indexVersion, queryBuilderField, queryBuilder, context);
Query query = toQuery(queryShardContext, mapUnmappedFieldAsText, queryBuilder);
Query query = toQuery(queryShardContext, isMapUnmappedFieldAsText(), queryBuilder);
processQuery(query, context);
return null;
}
Expand Down Expand Up @@ -522,7 +524,7 @@ protected String contentType() {
}

boolean isMapUnmappedFieldAsText() {
return mapUnmappedFieldAsText;
return ((FieldType) fieldType).mapUnmappedFieldsAsText;
}

/**
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -194,8 +194,7 @@ public void testDuel() throws Exception {
}
Collections.sort(intValues);

MappedFieldType intFieldType = mapperService.documentMapper("type").mappers()
.getMapper("int_field").fieldType();
MappedFieldType intFieldType = mapperService.fullName("int_field");

List<Supplier<Query>> queryFunctions = new ArrayList<>();
queryFunctions.add(MatchNoDocsQuery::new);
Expand Down Expand Up @@ -327,8 +326,7 @@ public void testDuel2() throws Exception {
stringValues.add("value2");
stringValues.add("value3");

MappedFieldType intFieldType = mapperService.documentMapper("type").mappers()
.getMapper("int_field").fieldType();
MappedFieldType intFieldType = mapperService.fullName("int_field");
List<int[]> ranges = new ArrayList<>();
ranges.add(new int[]{-5, 5});
ranges.add(new int[]{0, 10});
Expand Down
Loading