Window function on msq #15470

somu-imply · 2023-12-01T20:06:11Z

This PR aims to introduce Window functions on MSQ by doing the following:

Introduce a Window querykit for handling window queries along with its factory and a processor for window queries
If a window operator is present with a partition by clause, pushes the partition as a shuffle spec of the previous stage
In presence of empty OVER() clause lets all operators loose on a single rac
In presence of no empty OVER() clause, breaks down each window into individual stages
Associated machinery to handle window functions in MSQ
Introduced a separate hidden engine feature WINDOW_LEAF_OPERATOR which is set only for MSQ engine. In presence of this feature, the planner plans without the leaf operators by creating a window query over an inner scan query. In case of native this is set to false and the planner generates the leafOperators
Guardrails around materialization
Comprehensive UTs

Release notes

Add support in MSQE to run window functions using a context flag enableWindowing:true. In the native engine, we need a group by clause to enable window functions. In the MSQE the requirement of providing a mandatory group by clause to enable window functions is removed.

This PR has:

...age-query/src/main/java/org/apache/druid/msq/querykit/WindowOperatorQueryFrameProcessor.java

processing/src/main/java/org/apache/druid/query/rowsandcols/LazilyDecoratedRowsAndColumns.java

somu-imply · 2023-12-14T19:14:22Z

...age-query/src/main/java/org/apache/druid/msq/querykit/WindowOperatorQueryFrameProcessor.java

+          catch (IOException e) {
+            throw new RuntimeException(e);
+          }
+          return Operator.Signal.GO;


Instead of returning GO check if the frames can be paused. In such a case return that. Also need to test pausing frames through the MSQ framework correctly

somu-imply · 2023-12-14T19:16:14Z

...re/multi-stage-query/src/main/java/org/apache/druid/msq/querykit/WindowOperatorQueryKit.java

+                       .inputs(new StageInputSpec(firstStageNumber - 1))
+                       .signature(rowSignature)
+                       .maxWorkerCount(maxWorkerCount)
+                       .shuffleSpec(null)


Currently the shuffle spec is null. Tell the previous stage to shuffle by the appropriate partition here so that the data comes correctly. For example if previous stage is a groupByPostShuffle, find a way to tell it to set a shuffle spec for the next stage. Since the inner query has no idea of the outer operators, we can use the context to pass the information

The shuffle spec for a stage tells it how to partition the data for the next stage. Therefore it should use a combination of the resultShuffleSpecFactory to construct the final shuffleSpec.
If you want the data in a particular format inside a stage, its input should always be a stage, and the shuffle spec of that stage should be set accordingly. Hash Shuffle uses similar logic.

...age-query/src/main/java/org/apache/druid/msq/querykit/WindowOperatorQueryFrameProcessor.java

extensions-core/multi-stage-query/src/main/java/org/apache/druid/msq/exec/Limits.java

...e/multi-stage-query/src/main/java/org/apache/druid/msq/querykit/groupby/GroupByQueryKit.java

...age-query/src/main/java/org/apache/druid/msq/querykit/WindowOperatorQueryFrameProcessor.java

github-advanced-security

CodeQL found more than 20 potential problems in the proposed changes. Check the Files changed tab for more details.

...age-query/src/main/java/org/apache/druid/msq/querykit/WindowOperatorQueryFrameProcessor.java

...stage-query/src/main/java/org/apache/druid/msq/indexing/error/TooManyRowsInAWindowFault.java

sql/src/test/java/org/apache/druid/sql/calcite/CalciteSysQueryTest.java

...s-core/multi-stage-query/src/main/java/org/apache/druid/msq/util/MultiStageQueryContext.java

...e/multi-stage-query/src/main/java/org/apache/druid/msq/querykit/groupby/GroupByQueryKit.java

...age-query/src/main/java/org/apache/druid/msq/querykit/WindowOperatorQueryFrameProcessor.java

docs/multi-stage-query/known-issues.md

cryptoe · 2024-03-28T02:13:59Z

...stage-query/src/main/java/org/apache/druid/msq/indexing/error/TooManyRowsInAWindowFault.java

+  {
+    super(
+        CODE,
+        "Too many rows in a window (requested = %d, max = %d). Try creating a window with a higher cardinality column or change the query shape.",


We should also mention the user can set MAX_ROWS_MATERIALIZED_IN_WINDOW config in the query context. We should also tell the user that setting this config can lead to OOM errors so use with caution.

cryptoe · 2024-03-28T02:32:45Z

...e/multi-stage-query/src/main/java/org/apache/druid/msq/querykit/groupby/GroupByQueryKit.java

+                         .inputs(new StageInputSpec(firstStageNumber))
+                         .signature(stageSignature)
+                         .maxWorkerCount(maxWorkerCount)
+                         .shuffleSpec(nextShuffleWindowSpec)


In case of limit, this should not be nextShuffeWIndowSpec no ?

In case of a limit on the inner query, the window is going to operate on the result of the limit, so I think it should be the nextShuffleSpec as it contains the partition by for the next window

Lets add a UT for this if its already not there

I think the limit and offset should be applied on the grouping key. So it should be shuffleSpecFactoryPostAggregation != null ? : null
Also we can actually short circuit the shuffle spec of the OffsetLimitProcessor to null since limit always gets applied on 1 worker and 1 partition. So we would be okay in case a window processor is the next stage since the data would already be sorted :)

cryptoe

I feel the PR is almost there. Left some comments.
Thanks for working on this.

cryptoe · 2024-03-28T02:36:50Z

...age-query/src/main/java/org/apache/druid/msq/querykit/WindowOperatorQueryFrameProcessor.java

+
+  private final List<OperatorFactory> operatorFactoryList;
+  private final ObjectMapper jsonMapper;
+  private final ArrayList<RowsAndColumns> frameRowsAndCols;


frameRowsAndCols who clears this array list, I was expecting after we add stuff to the result, the frameRowsAndCols can be cleared no ?

Yes it is being cleared once the result is written

It seems to be cleared after we are done writing results to the frames which seems suspect.
Shouldn't it be cleared once we have added stuff to resultRowsAndCols ?

cryptoe · 2024-03-28T02:38:26Z

...e/multi-stage-query/src/main/java/org/apache/druid/msq/querykit/groupby/GroupByQueryKit.java

+                         .inputs(new StageInputSpec(firstStageNumber))
+                         .signature(stageSignature)
+                         .maxWorkerCount(maxWorkerCount)
+                         .shuffleSpec(nextShuffleWindowSpec)


Lets add a UT for this if its already not there

...ry/src/main/java/org/apache/druid/msq/querykit/WindowOperatorQueryFrameProcessorFactory.java

cryptoe

Changes LGTM. Thanks for the patience @somu-imply !!.

cryptoe · 2024-05-22T05:36:56Z

Updated the release notes taking into account the follow up PR as well #16229

somu-imply added 4 commits November 14, 2023 08:36

Initial code

4ef900d

Merge remote-tracking branch 'upstream/master' into windowFunctionOnMSQ

5e8dab1

Hacky way of atleast getting things to work

9ea01fc

Temp unfinished changes

9c4ac74

github-actions bot added Area - Batch Ingestion Area - Querying Area - MSQ For multi stage queries - https://github.com/apache/druid/issues/12262 labels Dec 1, 2023

soumyava added the WIP label Dec 1, 2023

github-advanced-security bot found potential problems Dec 1, 2023

View reviewed changes

...age-query/src/main/java/org/apache/druid/msq/querykit/WindowOperatorQueryFrameProcessor.java Fixed Show fixed Hide fixed

somu-imply added 3 commits December 6, 2023 20:18

Converting rac back to frames

54f9ac3

Working UTs

d6cef47

Fixing running window function in console

40a18f1

somu-imply commented Dec 14, 2023

View reviewed changes

...age-query/src/main/java/org/apache/druid/msq/querykit/WindowOperatorQueryFrameProcessor.java Outdated Show resolved Hide resolved

somu-imply commented Dec 14, 2023

View reviewed changes

processing/src/main/java/org/apache/druid/query/rowsandcols/LazilyDecoratedRowsAndColumns.java Outdated Show resolved Hide resolved

somu-imply commented Dec 14, 2023

View reviewed changes

somu-imply added 2 commits December 31, 2023 05:30

Merge remote-tracking branch 'upstream/master' into windowFunctionOnMSQ

3ecc96a

Adding shuffle spec and separating out stages for each window

7e34aa8

github-advanced-security bot found potential problems Jan 5, 2024

View reviewed changes

...age-query/src/main/java/org/apache/druid/msq/querykit/WindowOperatorQueryFrameProcessor.java Fixed Show fixed Hide fixed

somu-imply added 2 commits January 8, 2024 14:19

serde stuff by adding ops to proc factory

f5a1f59

Updating for first set of reviews

ab6e317

somu-imply marked this pull request as ready for review January 9, 2024 05:54

github-advanced-security bot found potential problems Jan 9, 2024

View reviewed changes

...age-query/src/main/java/org/apache/druid/msq/querykit/WindowOperatorQueryFrameProcessor.java Fixed Show fixed Hide fixed

somu-imply marked this pull request as draft January 15, 2024 14:02

somu-imply added 5 commits January 19, 2024 12:59

Changes for partition boundary detection

f1efec3

cleaning up some code, adding some tests

1dae450

Fixing up shuffle in group by if window afterwards

ccfe473

Merge remote-tracking branch 'upstream/master' into windowFunctionOnMSQ

500f54f

fix after merge

98f4ba5

github-advanced-security bot found potential problems Jan 20, 2024

View reviewed changes

...age-query/src/main/java/org/apache/druid/msq/querykit/WindowOperatorQueryFrameProcessor.java Fixed Show fixed Hide fixed

LakshSingla reviewed Mar 21, 2024

View reviewed changes

extensions-core/multi-stage-query/src/main/java/org/apache/druid/msq/exec/Limits.java Outdated Show resolved Hide resolved

LakshSingla reviewed Mar 21, 2024

View reviewed changes

extensions-core/multi-stage-query/src/main/java/org/apache/druid/msq/exec/Limits.java Show resolved Hide resolved

LakshSingla reviewed Mar 21, 2024

View reviewed changes

...e/multi-stage-query/src/main/java/org/apache/druid/msq/querykit/groupby/GroupByQueryKit.java Outdated Show resolved Hide resolved

LakshSingla reviewed Mar 21, 2024

View reviewed changes

...age-query/src/main/java/org/apache/druid/msq/querykit/WindowOperatorQueryFrameProcessor.java Show resolved Hide resolved

github-advanced-security bot found potential problems Mar 21, 2024

View reviewed changes

LakshSingla reviewed Mar 21, 2024

View reviewed changes

...age-query/src/main/java/org/apache/druid/msq/querykit/WindowOperatorQueryFrameProcessor.java Outdated Show resolved Hide resolved

Changes to one test case

04424d7

cryptoe reviewed Mar 26, 2024

View reviewed changes

...age-query/src/main/java/org/apache/druid/msq/querykit/WindowOperatorQueryFrameProcessor.java Outdated Show resolved Hide resolved

More fixes around guardrails and addressing last set of review comments

f0946b1

github-actions bot added the Area - Documentation label Mar 27, 2024

somu-imply added 3 commits March 26, 2024 23:35

Merge remote-tracking branch 'upstream/master' into windowFunctionOnMSQ

ea7882c

Fixing a testcase after the merge

83c96b9

Fixing a test by using correct in filters for sql compat mode

cfca6a5

cryptoe reviewed Mar 27, 2024

View reviewed changes

docs/multi-stage-query/known-issues.md Outdated Show resolved Hide resolved

somu-imply added 2 commits March 27, 2024 09:52

Not documenting context flag and 1 more test change

c3e2c29

Merge remote-tracking branch 'upstream/master' into windowFunctionOnMSQ

1464dae

somu-imply force-pushed the windowFunctionOnMSQ branch from 735b621 to 1464dae Compare March 27, 2024 21:55

cryptoe reviewed Mar 28, 2024

View reviewed changes

...ry/src/main/java/org/apache/druid/msq/querykit/WindowOperatorQueryFrameProcessorFactory.java Show resolved Hide resolved

somu-imply added 2 commits March 27, 2024 20:21

New test for inner limit on group by

520ab4e

Adding to known issues

16b75ce

cryptoe approved these changes Mar 28, 2024

View reviewed changes

cryptoe merged commit 524842a into apache:master Mar 28, 2024
85 checks passed

adarshsanjeev added this to the 30.0.0 milestone May 6, 2024

cryptoe mentioned this pull request May 22, 2024

Restore context flag for window functions #16229

Merged

10 tasks

adarshsanjeev mentioned this pull request May 28, 2024

[DRAFT] 30.0.0 release notes #16505

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Window function on msq #15470

Window function on msq #15470

somu-imply commented Dec 1, 2023 •

edited by cryptoe

Loading

somu-imply Dec 14, 2023

somu-imply Dec 14, 2023

LakshSingla Dec 15, 2023

github-advanced-security bot left a comment

cryptoe Mar 28, 2024

cryptoe Mar 28, 2024

soumyava Mar 28, 2024

cryptoe Mar 28, 2024

cryptoe Mar 28, 2024 •

edited

Loading

cryptoe left a comment

cryptoe Mar 28, 2024

somu-imply Mar 28, 2024

cryptoe Mar 28, 2024

cryptoe Mar 28, 2024

cryptoe left a comment

cryptoe commented May 22, 2024

Window function on msq #15470

Window function on msq #15470

Conversation

somu-imply commented Dec 1, 2023 • edited by cryptoe Loading

Release notes

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

github-advanced-security bot left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cryptoe Mar 28, 2024 • edited Loading

Choose a reason for hiding this comment

cryptoe left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cryptoe left a comment

Choose a reason for hiding this comment

cryptoe commented May 22, 2024

somu-imply commented Dec 1, 2023 •

edited by cryptoe

Loading

cryptoe Mar 28, 2024 •

edited

Loading