[SPARK-48358][SQL] Support for REPEAT statement #47756

dusantism-db · 2024-08-14T11:17:29Z

What changes were proposed in this pull request?

In this PR, support for REPEAT statement in SQL scripting is introduced.

Changes summary:

Grammar/parser changes

repeatStatement grammar rule
visitRepeatStatement rule visitor
RepeatStatement logical operetor

RepeatStatementExec execution node
Internal sates - Condition and Body
Iterator implementation - switch between body and condition until condition evaluates to true
SqlScriptingInterpreter - added logic to transform RepeatStatement logical operator to RepeatStatementExec execution node

Why are the changes needed?

This is a part of SQL Scripting introduced to Spark, REPEAT statement is a basic control flow construct in the SQL language.

Does this PR introduce any user-facing change?

No

How was this patch tested?

New tests are introduced to all of the three scripting test suites: SqlScriptingParserSuite, SqlScriptingExecutionNodeSuite and SqlScriptingInterpreterSuite.

Was this patch authored or co-authored using generative AI tooling?

No

sql/core/src/main/scala/org/apache/spark/sql/scripting/SqlScriptingExecutionNode.scala

miland-db · 2024-08-14T12:25:11Z

sql/core/src/main/scala/org/apache/spark/sql/scripting/SqlScriptingExecutionNode.scala

+  body: CompoundBodyExec,
+  session: SparkSession) extends NonLeafStatementExec {
+
+  private object RepeatState extends Enumeration {


Is there a way to unify parts of the while and repeat? The only difference is starting state and reset() method. Does it make sense to do it for 2 loops only?

State enum is exactly the same

treeIterator is exactly the same

Constructor parameters are identical

Just a note, treeIterator is not exactly the same because for while it iterates while condition is true, but for repeat it iterates while condition is false.

Yes, I see it. We can always override some new abstraction like continueIteration(session, condition). From my side this is ok, I'm just thinking if there is need to avoid code duplication in cases like this. Let's wait to hear from other folks.

this is a good point, but let's aim to solve this in a follow-up PR to simplify things a bit in this PR, since logic is correct.

sql/core/src/test/scala/org/apache/spark/sql/scripting/SqlScriptingExecutionNodeSuite.scala

davidm-db

LGTM, minor comments.

dusantism-db · 2024-09-06T12:35:39Z

sql/core/src/main/scala/org/apache/spark/sql/scripting/SqlScriptingInterpreter.scala

+
      case WhileStatement(condition, body, label) =>
        val conditionExec =
          new SingleStatementExec(condition.parsedPlan, condition.origin, isInternal = false)
        val bodyExec =
          transformTreeIntoExecutable(body, session).asInstanceOf[CompoundBodyExec]
        new WhileStatementExec(conditionExec, bodyExec, label, session)
+
+      case RepeatStatement(condition, body, label) =>
+        val conditionExec =
+          new SingleStatementExec(condition.parsedPlan, condition.origin, isInternal = false)
+        val bodyExec =
+          transformTreeIntoExecutable(body, session).asInstanceOf[CompoundBodyExec]
+        new RepeatStatementExec(conditionExec, bodyExec, label, session)
+
      case leaveStatement: LeaveStatement =>
        new LeaveStatementExec(leaveStatement.label)
+
      case iterateStatement: IterateStatement =>
        new IterateStatementExec(iterateStatement.label)
+


The empty lines are a suggestion for better readability of this method

miland-db · 2024-09-06T13:01:34Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala

+      case c: RepeatStatementContext
+        if Option(c.beginLabel()).isDefined &&
+          c.beginLabel().multipartIdentifier().getText.toLowerCase(Locale.ROOT).equals(label)
+      => true


Can we merge this with the above case for WhileStatementContext? You have multiple options for this, using | operator or Either type.

This wouldn't be trivial, as there is no common supertype between RepeatStatementContext and WhileStatementContext with the beginLabel() method defined. Maybe we could have a labeledStatement grammar rule, or something similar, to abstract some of this label logic from all statements with labels (ifElse, while, repeat..)

sql/core/src/test/scala/org/apache/spark/sql/scripting/SqlScriptingInterpreterSuite.scala

...alyst/src/main/scala/org/apache/spark/sql/catalyst/parser/SqlScriptingLogicalOperators.scala

sql/core/src/main/scala/org/apache/spark/sql/scripting/SqlScriptingExecutionNode.scala

MaxGekk

Please, fix indentations and it would be nice to add a little bit more negative tests.

sql/core/src/main/scala/org/apache/spark/sql/scripting/SqlScriptingExecutionNode.scala

MaxGekk · 2024-09-09T12:38:01Z

sql/core/src/main/scala/org/apache/spark/sql/scripting/SqlScriptingExecutionNode.scala

+/**
+ * Executable node for RepeatStatement.
+ * @param condition Executable node for the condition.
+ * @param body Executable node for the body.


Is there any contract/restriction for the body?

Except for the type, not really. The body can be anything which extends CompoundBodyExec

MaxGekk · 2024-09-09T12:39:41Z

sql/core/src/main/scala/org/apache/spark/sql/scripting/SqlScriptingExecutionNode.scala

+              curr = Some(condition)
+              condition.reset()
+              return retStmt
+            case _ =>


In which cases does this happen? Shall we raise an error here like unsupported or unexpected statement?

This match checks if the returned statement is LEAVE or ITERATE, because in those cases it should return early. If it's not LEAVE or ITERATE, then the method will continue normally. Because we only check if we should return early, we shouldn't throw any error. Similar logic exists in WhileStatementExec.

MaxGekk · 2024-09-11T07:50:01Z

+1, LGTM. Merging to master.
Thank you, @dusantism-db and @miland-db @davidm-db for review.

MaxGekk · 2024-09-11T07:53:26Z

@dusantism-db Congratulations with your first contribution to Apache Spark!

dusantism-db · 2024-09-11T08:48:33Z

@dusantism-db Congratulations with your first contribution to Apache Spark!

Thanks for your help, Max!

…Class` in `SqlScriptingInterpreterSuite` ### What changes were proposed in this pull request? In the PR, I propose to replace `errorClass` by `condition` in `SqlScriptingInterpreterSuite` ### Why are the changes needed? The changes from the PR #47756 conflict to #48027 ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? By running the modified test: ``` $ build/sbt "test:testOnly *SqlScriptingInterpreterSuite" ``` ### Was this patch authored or co-authored using generative AI tooling? No. Closes #48072 from MaxGekk/fix-errorClass-REPEAT. Authored-by: Max Gekk <max.gekk@gmail.com> Signed-off-by: Max Gekk <max.gekk@gmail.com>

### What changes were proposed in this pull request? In this PR, support for REPEAT statement in SQL scripting is introduced. Changes summary: Grammar/parser changes - `repeatStatement` grammar rule - `visitRepeatStatement` rule visitor - `RepeatStatement` logical operetor `RepeatStatementExec` execution node Internal sates - `Condition` and `Body` Iterator implementation - switch between body and condition until condition evaluates to true SqlScriptingInterpreter - added logic to transform RepeatStatement logical operator to RepeatStatementExec execution node ### Why are the changes needed? This is a part of SQL Scripting introduced to Spark, REPEAT statement is a basic control flow construct in the SQL language. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? New tests are introduced to all of the three scripting test suites: `SqlScriptingParserSuite`, `SqlScriptingExecutionNodeSuite` and `SqlScriptingInterpreterSuite`. ### Was this patch authored or co-authored using generative AI tooling? No Closes apache#47756 from dusantism-db/sql-scripting-repeat-statement. Authored-by: Dušan Tišma <dusan.tisma@databricks.com> Signed-off-by: Max Gekk <max.gekk@gmail.com>

…Class` in `SqlScriptingInterpreterSuite` ### What changes were proposed in this pull request? In the PR, I propose to replace `errorClass` by `condition` in `SqlScriptingInterpreterSuite` ### Why are the changes needed? The changes from the PR apache#47756 conflict to apache#48027 ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? By running the modified test: ``` $ build/sbt "test:testOnly *SqlScriptingInterpreterSuite" ``` ### Was this patch authored or co-authored using generative AI tooling? No. Closes apache#48072 from MaxGekk/fix-errorClass-REPEAT. Authored-by: Max Gekk <max.gekk@gmail.com> Signed-off-by: Max Gekk <max.gekk@gmail.com>

github-actions bot added SQL DOCS labels Aug 14, 2024

miland-db reviewed Aug 14, 2024

View reviewed changes

sql/core/src/main/scala/org/apache/spark/sql/scripting/SqlScriptingExecutionNode.scala Outdated Show resolved Hide resolved

miland-db reviewed Aug 14, 2024

View reviewed changes

davidm-db reviewed Aug 30, 2024

View reviewed changes

sql/core/src/test/scala/org/apache/spark/sql/scripting/SqlScriptingExecutionNodeSuite.scala Outdated Show resolved Hide resolved

davidm-db reviewed Aug 30, 2024

View reviewed changes

sql/core/src/test/scala/org/apache/spark/sql/scripting/SqlScriptingExecutionNodeSuite.scala Show resolved Hide resolved

davidm-db approved these changes Aug 30, 2024

View reviewed changes

dusantism-db force-pushed the sql-scripting-repeat-statement branch 2 times, most recently from 585dc0c to c3ef7a0 Compare September 5, 2024 16:23

dusantism-db commented Sep 6, 2024

View reviewed changes

miland-db reviewed Sep 6, 2024

View reviewed changes

miland-db approved these changes Sep 6, 2024

View reviewed changes

MaxGekk reviewed Sep 7, 2024

View reviewed changes

MaxGekk requested changes Sep 7, 2024

View reviewed changes

dusantism-db added 10 commits September 9, 2024 11:00

add initial repeat statement

c67d6ca

scripting interpreter suite tests for repeat

e6e27ec

scripting execution node suite tests for repeat

391ae32

scripting parser suite tests for repeat

8719f75

add REPEAT and UNTIL keywords

934e952

remove unnecessary assert

75ac771

refactor execution tests for while and repeat to use shared logic

a6ca945

add leave and iterate support for repeat statement + tests

8e8e690

finish tests for iterate and leave repeat statement

c21d782

fix formatting

e99566a

dusantism-db force-pushed the sql-scripting-repeat-statement branch from b572135 to e99566a Compare September 9, 2024 09:05

add negative tests

51080c2

dusantism-db closed this Sep 9, 2024

dusantism-db reopened this Sep 9, 2024

dusantism-db requested a review from MaxGekk September 9, 2024 09:52

MaxGekk reviewed Sep 9, 2024

View reviewed changes

address comments

27ee356

dusantism-db requested a review from MaxGekk September 9, 2024 14:19

MaxGekk approved these changes Sep 11, 2024

View reviewed changes

MaxGekk closed this in da89322 Sep 11, 2024

MaxGekk mentioned this pull request Sep 11, 2024

[SPARK-48358][SQL][TESTS][FOLLOWUP] Use condition instead of errorClass in SqlScriptingInterpreterSuite #48072

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-48358][SQL] Support for REPEAT statement #47756

[SPARK-48358][SQL] Support for REPEAT statement #47756

dusantism-db commented Aug 14, 2024

miland-db Aug 14, 2024

dusantism-db Aug 14, 2024

miland-db Aug 14, 2024

davidm-db Aug 30, 2024

davidm-db left a comment

dusantism-db Sep 6, 2024

miland-db Sep 6, 2024

dusantism-db Sep 6, 2024

MaxGekk left a comment

MaxGekk Sep 9, 2024

dusantism-db Sep 9, 2024

MaxGekk Sep 9, 2024

dusantism-db Sep 9, 2024 •

edited

Loading

MaxGekk commented Sep 11, 2024

MaxGekk commented Sep 11, 2024

dusantism-db commented Sep 11, 2024

[SPARK-48358][SQL] Support for REPEAT statement #47756

[SPARK-48358][SQL] Support for REPEAT statement #47756

Conversation

dusantism-db commented Aug 14, 2024

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Was this patch authored or co-authored using generative AI tooling?

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

davidm-db left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

MaxGekk left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dusantism-db Sep 9, 2024 • edited Loading

Choose a reason for hiding this comment

MaxGekk commented Sep 11, 2024

MaxGekk commented Sep 11, 2024

dusantism-db commented Sep 11, 2024

dusantism-db Sep 9, 2024 •

edited

Loading