Support uncorrelated scalar subquery after comparison operator in WHERE clause #14148

lancelly · 2024-11-20T06:41:57Z

As titled.

…d code cleaning

Cpaulyz · 2024-11-20T07:02:57Z

What a spectacular work 🦾! Fabulous design and implementation, LGTM! ⚡️⚡️🔥

THUMarkLau · 2024-11-27T07:40:30Z

I am utterly astounded and profoundly impressed by your recent Pull Request, "Support uncorrelated scalar subquery after comparison operator in WHERE clause," for the IoTDB project. Your exceptional work in enabling SQL subqueries within the WHERE clause is nothing short of groundbreaking and represents a monumental leap forward for the IoTDB community.

Your meticulous attention to detail and unparalleled expertise have culminated in a feature that not only enhances the functionality of IoTDB but also elevates its capabilities to new heights. The ability to support uncorrelated scalar subqueries after comparison operators is a testament to your deep understanding of complex database systems and your unwavering commitment to advancing open-source technology.

This contribution is poised to revolutionize the way users interact with IoTDB, providing them with unprecedented flexibility and power in their data queries. Your innovative approach and the elegance with which you have integrated this feature are truly commendable. It is evident that you have poured your heart and soul into this endeavor, and the results speak volumes about your dedication and brilliance.

In a world where technological advancements are often incremental, your work stands out as a beacon of ingenuity and excellence. You have not only addressed a significant gap in IoTDB's capabilities but have also set a new standard for what can be achieved through collaborative open-source development.

I am in awe of your talent and am incredibly proud to witness your contributions making such a profound impact. Your work will undoubtedly inspire others in the community to strive for greatness and push the boundaries of what is possible.

Congratulations on this remarkable achievement, and thank you for your invaluable contribution to the IoTDB project. The future of IoTDB is brighter because of your extraordinary efforts.

codecov · 2024-11-27T08:15:48Z

Codecov Report

Attention: Patch coverage is 25.80645% with 782 lines in your changes missing coverage. Please review.

Project coverage is 39.65%. Comparing base (05bc4fd) to head (951b50d).
Report is 4 commits behind head on master.

Files with missing lines	Patch %	Lines
...ngine/plan/relational/planner/SubqueryPlanner.java	20.52%	120 Missing ⚠️
...rocess/join/SimpleNestedLoopCrossJoinOperator.java	0.00%	97 Missing ⚠️
...engine/plan/relational/planner/node/ApplyNode.java	0.00%	62 Missing ⚠️
...al/planner/optimizations/QueryCardinalityUtil.java	16.66%	50 Missing ⚠️
.../source/relational/TableFullOuterJoinOperator.java	0.00%	48 Missing ⚠️
...planner/optimizations/UnaliasSymbolReferences.java	39.13%	42 Missing ⚠️
...ator/source/relational/TableInnerJoinOperator.java	0.00%	37 Missing ⚠️
...tive/rule/TransformUncorrelatedSubqueryToJoin.java	37.73%	33 Missing ⚠️
...eryengine/plan/planner/TableOperatorGenerator.java	0.00%	32 Missing ⚠️
...ion/operator/process/EnforceSingleRowOperator.java	0.00%	31 Missing ⚠️
... and 30 more

Additional details and impacted files

@@             Coverage Diff              @@
##             master   #14148      +/-   ##
============================================
+ Coverage     39.63%   39.65%   +0.01%     
  Complexity       71       71              
============================================
  Files          4242     4268      +26     
  Lines        270488   271435     +947     
  Branches      32806    32941     +135     
============================================
+ Hits         107215   107632     +417     
- Misses       163273   163803     +530

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Beyyes · 2024-11-28T07:32:04Z

...iotdb/db/queryengine/execution/operator/process/join/merge/comparator/JoinKeyComparator.java

+
+  /*Use primitive types to avoid boxing and unboxing.*/
+
+  boolean lessThan(int left, int right);


Also need boolean type?

Beyyes · 2024-11-28T08:25:33Z

...apache/iotdb/db/queryengine/execution/operator/source/relational/TableInnerJoinOperator.java

+      case BLOB:
+      case TEXT:
+      case BOOLEAN:
+      default:


BLOB,TEXT,BOOLEAN types are also needed to be support

JackieTien97

SimpleNestedLoopCrossJoinOperator, TableInnerJoinOperator and TableOperatorGenerator are not reviewed, should be reviewed by @Beyyes after the JoinKeyComaprator refactoring

JackieTien97 · 2024-11-28T09:23:54Z

...c/main/java/org/apache/iotdb/db/queryengine/plan/relational/planner/ExpressionExtractor.java

    }

-    @Override
+    /*@Override


why need to delete the visitAggregation?

And what exactly the ExpressionExtractor used for?

visitAggregation is not removed, its already been commented out previously. For now, ExpressionExtractor is used by SymbolExtractor to extractor symbols for optimizing phase.

JackieTien97 · 2024-11-28T09:36:32Z

...e/iotdb/db/queryengine/plan/relational/planner/distribute/TableDistributedPlanGenerator.java

+  public List<PlanNode> visitEnforceSingleRow(EnforceSingleRowNode node, PlanContext context) {
+    List<PlanNode> childrenNodes = node.getChild().accept(this, context);
+    OrderingScheme childOrdering = nodeOrderingMap.get(childrenNodes.get(0).getPlanNodeId());
+    if (childOrdering != null) {
+      nodeOrderingMap.put(node.getPlanNodeId(), childOrdering);
+    }
+
+    node.setChild(mergeChildrenViaCollectOrMergeSort(childOrdering, childrenNodes));
+    return Collections.singletonList(node);
+  }


add EnforceSingleRowNode for each children and then call mergeChildrenViaCollectOrMergeSort, and then followed by a EnforceSingleRowNode may be better.

JackieTien97 · 2024-11-28T09:51:38Z

...in/java/org/apache/iotdb/db/queryengine/plan/relational/planner/node/CorrelatedJoinNode.java

+ * <p>INNER - does not return any row for input row when subquery relation is empty LEFT - does
+ * return input completed with NULL values when subquery relation is empty
+ */
+public class CorrelatedJoinNode extends TwoChildProcessNode {


may need to explain what's the difference between CorrelatedJoinNode and ApplyNode, and how an unCorrelated Join be transformed from CorrelatedJoinNode or ApplyNode to a JoinNode

JackieTien97 · 2024-11-28T09:59:39Z

...tdb/db/queryengine/plan/relational/planner/optimizations/CheckSubqueryNodesAreRewritten.java

+import static org.apache.iotdb.db.queryengine.plan.relational.planner.PlanNodeSearcher.searchFrom;
+import static org.apache.iotdb.rpc.TSStatusCode.SEMANTIC_ERROR;
+
+public class CheckSubqueryNodesAreRewritten implements PlanOptimizer {


a little bit confused here, what plan nodes will correlated subqueries use? not ApplyNode and CorrelatedJoinNode?

I think Join and SemiJoin are often used by correlated subqueries.

JackieTien97 · 2024-11-28T10:05:23Z

...pache/iotdb/db/queryengine/plan/relational/planner/optimizations/LogicalOptimizeFactory.java

+            plannerContext,
+            ruleStats,
+            ImmutableSet.of(
+                new RemoveRedundantEnforceSingleRowNode(),


may need to add it previous IterativeOptimizer redundantly

I checked this, will add it in an preprevious IterativeOptimizer redundantly.

JackieTien97 · 2024-11-28T10:08:21Z

...pache/iotdb/db/queryengine/plan/relational/planner/optimizations/LogicalOptimizeFactory.java

+            ImmutableSet.of(
+                new RemoveRedundantEnforceSingleRowNode(),
+                new TransformUncorrelatedSubqueryToJoin())),
+        new CheckSubqueryNodesAreRewritten(),


In trino, it seems that we also need to add another simplifyOptimizer after it.

Yes, but I'm not certain its related to subquery. I will add simplifyOptimizer after this rule.

JackieTien97 · 2024-11-28T10:11:25Z

.../apache/iotdb/db/queryengine/plan/relational/planner/optimizations/QueryCardinalityUtil.java

+    //        public Range<Long> visitTopN(TopNNode node, Void context)
+    //        {
+    //            return applyLimit(node.getSource(), node.getCount());
+    //        }


we also have TopKNode

JackieTien97 · 2024-11-28T10:19:19Z

...db/db/queryengine/execution/operator/process/join/merge/comparator/AscJoinKeyComparator.java

+
+import org.apache.tsfile.utils.Binary;
+
+public class AscJoinKeyComparator implements JoinKeyComparator {


The Comparator shouldn't be this way, we should only have

lessThan(TsBlock left, int leftRowIndex, TsBlock right, int rightRowIndex)

equalsTo(TsBlock left, int leftRowIndex, TsBlock right, int rightRowIndex)

lessThanOrEqual(TsBlock left, int leftRowIndex, TsBlock right, int rightRowIndex)

in interface, and asc/desc implementation class for each type.

JackieTien97 · 2024-11-28T12:07:49Z

...e/iotdb/db/queryengine/plan/relational/planner/distribute/TableDistributedPlanGenerator.java

+    node.setLeftChild(mergeChildrenViaCollectOrMergeSort(null, leftChildrenNodes));
+    node.setRightChild(mergeChildrenViaCollectOrMergeSort(null, rightChildrenNodes));


directly passing null as OrderingScheme? should use the left and right child's orderingschema

Nice catch. Fixed.

sonarcloud · 2024-11-29T11:24:31Z

Quality Gate failed

Failed conditions
C Reliability Rating on New Code (required ≥ A)

See analysis details on SonarQube Cloud

Catch issues before they fail your Quality Gate with our IDE extension SonarQube for IDE

lancelly added 4 commits November 17, 2024 16:17

Support uncorrelated scalar subquery in where clause without UT/IT an…

4ad3b87

…d code cleaning

remove accidentally added files

3a548e8

Merge branch 'master' into support_uncorrelated_subquery_in_where

d869028

add some ITs

9418525

lancelly added 4 commits November 20, 2024 18:16

add ser/deser methods of some PlanNodes

7808a2a

support other data types in tableInnerJoinOperator

adbbf8c

merge master and resolve conflicts

8468e57

self check and fix some typos

6ba9c6d

lancelly marked this pull request as ready for review November 27, 2024 07:22

lancelly added 2 commits November 27, 2024 17:57

fix ITs

d312d96

remove temporary workaround in pushAggregationIntoTableScan

ed5f5e2

Beyyes reviewed Nov 28, 2024

View reviewed changes

JackieTien97 requested changes Nov 28, 2024

View reviewed changes

lancelly added 4 commits November 29, 2024 11:46

fix some of the comments

99565fe

resolve conflicts with master and refactor JoinKeyComparator

05add9c

add boolean joinKeyComparator

6297374

fix typo

951b50d

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support uncorrelated scalar subquery after comparison operator in WHERE clause #14148

Support uncorrelated scalar subquery after comparison operator in WHERE clause #14148

lancelly commented Nov 20, 2024 •

edited

Loading

Cpaulyz commented Nov 20, 2024 •

edited

Loading

THUMarkLau commented Nov 27, 2024

codecov bot commented Nov 27, 2024 •

edited

Loading

Beyyes Nov 28, 2024

Beyyes Nov 28, 2024 •

edited

Loading

JackieTien97 left a comment

JackieTien97 Nov 28, 2024

JackieTien97 Nov 28, 2024

lancelly Nov 29, 2024 •

edited

Loading

JackieTien97 Nov 28, 2024

JackieTien97 Nov 28, 2024

JackieTien97 Nov 28, 2024

lancelly Nov 29, 2024

JackieTien97 Nov 28, 2024

lancelly Nov 29, 2024

JackieTien97 Nov 28, 2024

lancelly Nov 29, 2024

JackieTien97 Nov 28, 2024

lancelly Nov 29, 2024

JackieTien97 Nov 28, 2024

JackieTien97 Nov 28, 2024

lancelly Nov 29, 2024

sonarcloud bot commented Nov 29, 2024


		/Use primitive types to avoid boxing and unboxing./

		boolean lessThan(int left, int right);


		import org.apache.tsfile.utils.Binary;

		public class AscJoinKeyComparator implements JoinKeyComparator {

		node.setLeftChild(mergeChildrenViaCollectOrMergeSort(null, leftChildrenNodes));
		node.setRightChild(mergeChildrenViaCollectOrMergeSort(null, rightChildrenNodes));

Support uncorrelated scalar subquery after comparison operator in WHERE clause #14148

Are you sure you want to change the base?

Support uncorrelated scalar subquery after comparison operator in WHERE clause #14148

Conversation

lancelly commented Nov 20, 2024 • edited Loading

Cpaulyz commented Nov 20, 2024 • edited Loading

THUMarkLau commented Nov 27, 2024

codecov bot commented Nov 27, 2024 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Beyyes Nov 28, 2024 • edited Loading

Choose a reason for hiding this comment

JackieTien97 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lancelly Nov 29, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sonarcloud bot commented Nov 29, 2024

Quality Gate failed

lancelly commented Nov 20, 2024 •

edited

Loading

Cpaulyz commented Nov 20, 2024 •

edited

Loading

codecov bot commented Nov 27, 2024 •

edited

Loading

Beyyes Nov 28, 2024 •

edited

Loading

lancelly Nov 29, 2024 •

edited

Loading