SQL: Resolve attributes recursively for improved subquery support #69765

palesz · 2021-03-02T01:31:37Z

Previously we did not resolve the attributes recursively which meant that if a field or expression was re-aliased multiple times (through multiple levels of subqueries), the aliases were only resolved one level down. This led to failed query translation because ReferenceAttributes were pointing to non-existing attributes during query translation.

For example the query

SELECT i AS j FROM ( SELECT int AS i FROM test) ORDER BY j

failed during translation because the OrderBy resolved the j ReferenceAttribute to another i ReferenceAttribute that was later removed by an Optimization:

OrderBy[[Order[j{r}#4,ASC,LAST]]]                                             ! OrderBy[[Order[i{r}#2,ASC,LAST]]]
\_Project[[j]]                                                                = \_Project[[j]]
  \_Project[[i]]                                                              !   \_EsRelation[test][date{f}#6, some{f}#7, some.string{f}#8, some.string..]
    \_EsRelation[test][date{f}#6, some{f}#7, some.string{f}#8, some.string..] !

By resolving the Attributes recursively both j{r} and i{r} will resolve to test.int{f} above:

OrderBy[[Order[test.int{f}#22,ASC,LAST]]]                                     = OrderBy[[Order[test.int{f}#22,ASC,LAST]]]
\_Project[[j]]                                                                = \_Project[[j]]
  \_Project[[i]]                                                              !   \_EsRelation[test][date{f}#6, some{f}#7, some.string{f}#8, some.string..]
    \_EsRelation[test][date{f}#6, some{f}#7, some.string{f}#8, some.string..] !

The scope of recursive resolution depends on how the AttributeMap is constructed and populated.

Fixes #67237

Code simplification in the AttributeMap

elasticmachine · 2021-03-02T23:07:38Z

Pinging @elastic/es-ql (Team:QL)

costin

Glad to see progress on this however the breakage to the map contract isn't the way forward (see my comments).

costin · 2021-03-03T18:38:23Z

x-pack/plugin/ql/src/main/java/org/elasticsearch/xpack/ql/expression/AttributeMap.java

+    /**
+     * @param key {@link NamedExpression} to look up
+     * @return Looks up the key in the AttributeMap recursively, meaning if the value for this key
+     * is another key in the map, it will follow it. It will return the final value or null if the key
+     * does not exist in the map.
+     */
    @Override
    public E get(Object key) {
-        if (key instanceof NamedExpression) {
-            return delegate.get(new AttributeWrapper(((NamedExpression) key).toAttribute()));
-        }
-        return null;
+        return getOrDefault(key, null);
    }

+    /**
+     * @param key {@link NamedExpression} to look up
+     * @return Looks up the key in the AttributeMap recursively, meaning if the value for this key
+     * is another key in the map, it will follow it. It will return the final value or the
+     * specified default value if the key does not exist in the map.
+     */
    @Override
    public E getOrDefault(Object key, E defaultValue) {
-        E e;
-        return (((e = get(key)) != null) || containsKey(key))
-            ? e
-            : defaultValue;
+        E candidate = defaultValue;
+        E value = null;
+        while (((value = lookup(key)) != null || containsKey(key)) && value != key) {
+            key = candidate = value;
+        }
+        return candidate;
+    }


This is backwards.
getOrDefault should delegate to get not vice-versa. The former is an extension of the latter historically and conceptually.

Moreover by making get and getOrDefault recursive the Map contract has been fundamentally broken:

m.put(a, b); assertTrue(b, m.get(a)); m.put(b,c); assertTrue(b, m.get(a)); // fails

Instead of renaming get to lookup and modifying the contract, simply introduce a separate method (e.g. traverse(key), getResolvedValue(key), getRecursive(key) or something else) to the original map that does the recursive lookup and use that instead.

I think I agree as well on this, I'd prefer to have a dedicated method that does this more complex logic rather than overriding the default get.

Good points, but maybe the solution here is not just a simple rename of the method (resolve(key)), but to not implement the Map<Attribute, E> interface anymore. The Map contract was broken since the beginning because instead of equals() the semanticEquals() is used for key equality checks (which for example violates "containsKey(key) returns true if and only if this map contains a mapping for a key k such that (key==null ? k==null : key.equals(k)).")
After noget()s are required anymore, unless you can think of a place where recursive key resolution would actually cause harm (if that is the case, I'd say the AttributeMap building should be fixed instead to limit the scope from where the attributes are picked up from).

AttributeMap is a Map for AttributeWrappers; the get, contains rely on equals. For convenience the underlying class is not exposed instead Attribute is used with the wrapping happening automatically.
It's similar to IdentityHashMap for example.

resolve(key) is a compound method that relies on get, the latter might not be used publicly now however I see too many downsides in removing it (and the Map contract) instead of keeping it.
For example get/put do per key read/write but resolve, which does multi-key read has not write counter-party; removing/replacing get breaks too many assumptions.

I am not sure about the downsides or the breaking assumptions, since we barely utilize anything from the Map interface, practically only some read methods: isEmpty, size and get/getOrDefault (which can be renamed to resolve). The building of the map (puts) happen through the Builder methods today, here is how a full minimalization of the public methods would look like (at this point the AttributeMap is rather an AttributeResolver): palesz@fe814ec

Nonetheless, I can keep the current Map implementation and create a new resolve method for the recursive resolution, switch out all the get/getOrDefault calls to resolve and leave the gets there between the other unused methods.

costin · 2021-03-03T18:40:31Z

x-pack/plugin/sql/src/main/java/org/elasticsearch/xpack/sql/analysis/analyzer/Verifier.java

@@ -316,10 +316,11 @@ private static boolean checkGroupByOrder(LogicalPlan p, Set<Failure> localFailur
                Map<Expression, Node<?>> missing = new LinkedHashMap<>();

                o.order().forEach(oe -> {
-                    Expression e = oe.child();
+                    final Expression e = oe.child();
+                    final Expression resolvedE = attributeRefs.getOrDefault(e, e);


See my comment above - it's misleading for a map lookup to do resolution.

costin · 2021-03-03T18:49:07Z

x-pack/plugin/sql/src/main/java/org/elasticsearch/xpack/sql/optimizer/Optimizer.java

        Batch refs = new Batch("Replace References", Limiter.ONCE,
-                new ReplaceReferenceAttributeWithSource()
-                );
+            new ReplaceReferenceAttributeWithSource()
+        );

        Batch operators = new Batch("Operator Optimization",
                // combining


costin · 2021-03-03T18:49:15Z

x-pack/plugin/sql/src/main/java/org/elasticsearch/xpack/sql/optimizer/Optimizer.java

@@ -222,7 +222,7 @@ public LogicalPlan apply(LogicalPlan plan) {
            AttributeMap.Builder<Expression> builder = AttributeMap.builder();
            // collect aliases
            plan.forEachExpressionUp(Alias.class, a -> builder.put(a.toAttribute(), a.child()));
-            final Map<Attribute, Expression> collectRefs = builder.build();
+            final AttributeMap<Expression> collectRefs = builder.build();


costin · 2021-03-03T18:53:16Z

x-pack/plugin/sql/src/main/java/org/elasticsearch/xpack/sql/planner/QueryFolder.java

@@ -600,10 +600,12 @@ else if (target.foldable()) {
                else {
                    GroupByKey matchingGroup = null;
                    if (groupingContext != null) {
-                        matchingGroup = groupingContext.groupFor(target);
+                        final Expression resolvedTarget = queryC.aliases().getOrDefault(target, target);


This bit of code is confusing due to the left overs.
Target doesn't seem to be used anywhere or at least its initial value is no good - instead of adding a resolvedTarget just reinitialize it:
target = queryC.aliases().getOrDefault(target, target)

Same for id.

costin · 2021-03-03T18:54:35Z

x-pack/plugin/sql/src/test/java/org/elasticsearch/xpack/sql/planner/QueryTranslatorTests.java

@@ -2658,4 +2657,29 @@ public void testSubqueryWithAliasOrderByAlias() throws Exception {
            "( SELECT int AS i FROM test ) AS s " +
            "ORDER BY s.i > 10");
    }
+
+    public void testReferenceResolutionInSubqueries() {


Please break this method into multiple tests that indicate the difference between them in their name.

Recursive get/getDefault() to resolve() in AttributeMap Removed noise Test breakdown

costin · 2021-03-09T07:28:41Z

x-pack/plugin/ql/src/main/java/org/elasticsearch/xpack/ql/expression/AttributeMap.java

+    @SuppressWarnings({ "unchecked" })
+    public E resolve(Object key, E defaultValue) {
+        AttributeMap<Object> map = (AttributeMap<Object>) this;
+        Object candidate = defaultValue;
+        Object value = null;
+        while ((value = map.getOrDefault(key, NOT_FOUND)) != NOT_FOUND && value != key) {
+            key = candidate = value;
+        }
+        return (E) candidate;
    }


As this method builds on other types methods in this class, it shouldn't have unchecked warnings nor used NOT_FOUND. Also not sure why the candidate is returned instead of value, should be the other way around.

Further more this implementation looks buggy: if the map contains (e,e) calling resolve(e, default) would incorrectly return default and not e.
Further more there's no handling of cycles which leads to an infinite loop: (a, b), (b, a).

Below a suggestion to fix both issues:

Suggested change

@SuppressWarnings({ "unchecked" })

public E resolve(Object key, E defaultValue) {

AttributeMap<Object> map = (AttributeMap<Object>) this;

Object candidate = defaultValue;

Object value = null;

while ((value = map.getOrDefault(key, NOT_FOUND)) != NOT_FOUND && value != key) {

key = candidate = value;

}

return (E) candidate;

}

public E resolveOrDefault(Object key, E defaultValue) {

E value = defaultValue;

E candidate;

int allowedLookups = 10;

while (key != value && ((candidate = get(key)) != null || containsKey(key))) {

if (--allowedLookups == 0) {

throw new QlIllegalArgumentException("Potential cycle detected");

}

key = candidate;

value = candidate;

}

return value;

}

Good catch with the default value. I was thinking of adding the limitation on the number of lookups, but seemed we would need quite a huge bug in our AttributeMap population for it to happen with more than one hops in the cycle. Anyways, I added this defensive check.

The suggested code won't really work, because:

1.resolveOrDefault(e, e) will always return e even if the map has e -> f mapping.
2. The loop will always terminate after the first lookup (key == value == candidate)

Anyways, the updated code fixes the bug you found.

costin · 2021-03-09T07:31:08Z

x-pack/plugin/ql/src/test/java/org/elasticsearch/xpack/ql/expression/AttributeMapTests.java

@@ -61,6 +63,30 @@ public void testAttributeMapWithSameAliasesCanResolveAttributes() {
        assertTrue(newAttributeMap.get(param2.toAttribute()) == param2.child());
    }

+    public void testResolve() {


I'd like to see more tests for dealing with:
a. same key, value association - (e,e)
b. handling of default value across association: resolveOrDefault(k, default) for a map with (k, default), (default, value) and also resolveOrDefault(k, k)
c. handling of cycles: (a, b), (b, c), (c,a ) with a mixture of default value in place.

costin · 2021-03-09T07:33:57Z

x-pack/plugin/sql/src/main/java/org/elasticsearch/xpack/sql/analysis/analyzer/Verifier.java

@@ -406,7 +411,7 @@ private static boolean checkGroupByHavingHasOnlyAggs(Expression e, Set<Expressio

        // resolve FunctionAttribute to backing functions
        if (e instanceof ReferenceAttribute) {
-            e = attributeRefs.get(e);
+            e = attributeRefs.resolve(e, null);


The null variant is used enough times to make a resolve method, in addition to resolveOrDefault on the main class similar to get and getOrDefault (even if it's just syntactic sugar).

costin

LGTM

costin · 2021-03-11T13:46:23Z

x-pack/plugin/ql/src/main/java/org/elasticsearch/xpack/ql/expression/AttributeMap.java

+            if (--allowedLookups == 0) {
+                throw new QlIllegalArgumentException("Potential cycle detected");
+            }
+            key = value = candidate;


Please break the assignment in two pieces.

…astic#69765) Previously we did not resolve the attributes recursively which meant that if a field or expression was re-aliased multiple times (through multiple levels of subqueries), the aliases were only resolved one level down. This led to failed query translation because `ReferenceAttribute`s were pointing to non-existing attributes during query translation. For example the query ```sql SELECT i AS j FROM ( SELECT int AS i FROM test) ORDER BY j ``` failed during translation because the `OrderBy` resolved the `j` ReferenceAttribute to another `i` ReferenceAttribute that was later removed by an Optimization: ``` OrderBy[[Order[j{r}elastic#4,ASC,LAST]]] ! OrderBy[[Order[i{r}elastic#2,ASC,LAST]]] \_Project[[j]] = \_Project[[j]] \_Project[[i]] ! \_EsRelation[test][date{f}elastic#6, some{f}elastic#7, some.string{f}elastic#8, some.string..] \_EsRelation[test][date{f}elastic#6, some{f}elastic#7, some.string{f}elastic#8, some.string..] ! ``` By resolving the `Attributes` recursively both `j{r}` and `i{r}` will resolve to `test.int{f}` above: ``` OrderBy[[Order[test.int{f}elastic#22,ASC,LAST]]] = OrderBy[[Order[test.int{f}elastic#22,ASC,LAST]]] \_Project[[j]] = \_Project[[j]] \_Project[[i]] ! \_EsRelation[test][date{f}elastic#6, some{f}elastic#7, some.string{f}elastic#8, some.string..] \_EsRelation[test][date{f}elastic#6, some{f}elastic#7, some.string{f}elastic#8, some.string..] ! ``` The scope of recursive resolution depends on how the `AttributeMap` is constructed and populated. Fixes elastic#67237

…9765) (#70325) Previously we did not resolve the attributes recursively which meant that if a field or expression was re-aliased multiple times (through multiple levels of subqueries), the aliases were only resolved one level down. This led to failed query translation because `ReferenceAttribute`s were pointing to non-existing attributes during query translation. For example the query ```sql SELECT i AS j FROM ( SELECT int AS i FROM test) ORDER BY j ``` failed during translation because the `OrderBy` resolved the `j` ReferenceAttribute to another `i` ReferenceAttribute that was later removed by an Optimization: ``` OrderBy[[Order[j{r}#4,ASC,LAST]]] ! OrderBy[[Order[i{r}#2,ASC,LAST]]] \_Project[[j]] = \_Project[[j]] \_Project[[i]] ! \_EsRelation[test][date{f}#6, some{f}#7, some.string{f}#8, some.string..] \_EsRelation[test][date{f}#6, some{f}#7, some.string{f}#8, some.string..] ! ``` By resolving the `Attributes` recursively both `j{r}` and `i{r}` will resolve to `test.int{f}` above: ``` OrderBy[[Order[test.int{f}#22,ASC,LAST]]] = OrderBy[[Order[test.int{f}#22,ASC,LAST]]] \_Project[[j]] = \_Project[[j]] \_Project[[i]] ! \_EsRelation[test][date{f}#6, some{f}#7, some.string{f}#8, some.string..] \_EsRelation[test][date{f}#6, some{f}#7, some.string{f}#8, some.string..] ! ``` The scope of recursive resolution depends on how the `AttributeMap` is constructed and populated. Fixes #67237

…9765) (#70322) Previously we did not resolve the attributes recursively which meant that if a field or expression was re-aliased multiple times (through multiple levels of subqueries), the aliases were only resolved one level down. This led to failed query translation because `ReferenceAttribute`s were pointing to non-existing attributes during query translation. For example the query ```sql SELECT i AS j FROM ( SELECT int AS i FROM test) ORDER BY j ``` failed during translation because the `OrderBy` resolved the `j` ReferenceAttribute to another `i` ReferenceAttribute that was later removed by an Optimization: ``` OrderBy[[Order[j{r}#4,ASC,LAST]]] ! OrderBy[[Order[i{r}#2,ASC,LAST]]] \_Project[[j]] = \_Project[[j]] \_Project[[i]] ! \_EsRelation[test][date{f}#6, some{f}#7, some.string{f}#8, some.string..] \_EsRelation[test][date{f}#6, some{f}#7, some.string{f}#8, some.string..] ! ``` By resolving the `Attributes` recursively both `j{r}` and `i{r}` will resolve to `test.int{f}` above: ``` OrderBy[[Order[test.int{f}#22,ASC,LAST]]] = OrderBy[[Order[test.int{f}#22,ASC,LAST]]] \_Project[[j]] = \_Project[[j]] \_Project[[i]] ! \_EsRelation[test][date{f}#6, some{f}#7, some.string{f}#8, some.string..] \_EsRelation[test][date{f}#6, some{f}#7, some.string{f}#8, some.string..] ! ``` The scope of recursive resolution depends on how the `AttributeMap` is constructed and populated. Fixes #67237

Resolve references recursively

aba2f63

Code simplification in the AttributeMap

palesz force-pushed the subquery-npe-fix branch from f80b9de to aba2f63 Compare March 2, 2021 17:26

palesz changed the title ~~SQL: Resolve attributes recursively to improve subquery support~~ SQL: Resolve attributes recursively for improved subquery support Mar 2, 2021

palesz added >bug :Analytics/SQL SQL querying Team:QL (Deprecated) Meta label for query languages team labels Mar 2, 2021

palesz added :Analytics/EQL EQL querying v7.12.1 v7.13.0 v8.0.0 labels Mar 2, 2021

Andras Palinkas added 2 commits March 2, 2021 18:19

Merge remote-tracking branch 'origin/master' into subquery-npe-fix

39e4cf1

Verifier error message fix

015acb3

palesz requested review from astefan, costin, matriv and bpintea March 3, 2021 14:23

costin requested changes Mar 3, 2021

View reviewed changes

Andras Palinkas added 2 commits March 5, 2021 19:32

PR suggestions

3b755be

Recursive get/getDefault() to resolve() in AttributeMap Removed noise Test breakdown

Merge remote-tracking branch 'origin/master' into subquery-npe-fix

f739e39

palesz requested a review from costin March 6, 2021 00:38

costin requested changes Mar 9, 2021

View reviewed changes

Andras Palinkas added 2 commits March 10, 2021 18:07

Bugfix and PR suggestions

932c01c

Merge remote-tracking branch 'origin/master' into subquery-npe-fix

fa49615

palesz requested a review from costin March 10, 2021 23:32

costin approved these changes Mar 11, 2021

View reviewed changes

Andras Palinkas added 2 commits March 11, 2021 09:51

PR suggestions

1cb642d

Merge remote-tracking branch 'origin/master' into subquery-npe-fix

50ab4d1

palesz merged commit 6e1eb44 into elastic:master Mar 11, 2021

palesz added the backport pending label Mar 11, 2021

palesz removed the backport pending label Mar 11, 2021

gwbrown removed the :Analytics/EQL EQL querying label Apr 14, 2021

jakelandis added v8.0.0-alpha1 and removed v8.0.0 labels Jul 26, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

SQL: Resolve attributes recursively for improved subquery support #69765

SQL: Resolve attributes recursively for improved subquery support #69765

palesz commented Mar 2, 2021 •

edited

Loading

elasticmachine commented Mar 2, 2021

costin left a comment

costin Mar 3, 2021

matriv Mar 3, 2021

palesz Mar 4, 2021

costin Mar 5, 2021

palesz Mar 5, 2021

palesz Mar 6, 2021

costin Mar 3, 2021

palesz Mar 6, 2021

costin Mar 3, 2021

palesz Mar 6, 2021

costin Mar 3, 2021

palesz Mar 6, 2021

costin Mar 3, 2021

palesz Mar 6, 2021

costin Mar 3, 2021

palesz Mar 6, 2021

costin Mar 9, 2021

palesz Mar 10, 2021

costin Mar 9, 2021

palesz Mar 10, 2021

costin Mar 9, 2021

palesz Mar 10, 2021

costin left a comment

costin Mar 11, 2021

SQL: Resolve attributes recursively for improved subquery support #69765

SQL: Resolve attributes recursively for improved subquery support #69765

Conversation

palesz commented Mar 2, 2021 • edited Loading

elasticmachine commented Mar 2, 2021

costin left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

costin left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

palesz commented Mar 2, 2021 •

edited

Loading