[SPARK-49306][SQL] Create new SQL functions 'zeroifnull' and 'nullifzero' #47817

dtenedor · 2024-08-20T17:57:43Z

What changes were proposed in this pull request?

This PR creates new SQL functions zeroifnull and nullifzero.

These are equivalent to nvl(input, 0) and if(input = 0, null, input), respectively. The analyzer performs these replacements inline in this PR to take advantage of existing query optimizations for nvl and if expressions.

For example:

> select zeroifnull(null);
 0
> select zeroifnull(2);
 2

> select nullifzero(0);
 null
> select nullifzero(2);
 2

Why are the changes needed?

We are adding these functions for compatibility with other systems which also have these SQL functions.

Does this PR introduce any user-facing change?

Yes, see above.

How was this patch tested?

This PR adds unit test coverage.

Was this patch authored or co-authored using generative AI tooling?

No.

commit

dtenedor · 2024-08-20T18:10:56Z

cc @HyukjinKwon @MaxGekk can one of you guys maybe help with this one :)

HyukjinKwon · 2024-08-21T00:19:27Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/FunctionRegistry.scala

@@ -384,6 +385,7 @@ object FunctionRegistry {
    expression[Rand]("random", true, Some("3.0.0")),
    expression[Randn]("randn"),
    expression[Stack]("stack"),
+    expression[ZeroIfNull]("zeroifnull"),


should probably add them into functions.scala and functions.py. could be done in a separate PR tho.

Thanks, I will add DataFrame support in a separate PR :)

MaxGekk · 2024-08-21T07:06:51Z

sql/core/src/test/scala/org/apache/spark/sql/MiscFunctionsSuite.scala

@@ -285,6 +285,48 @@ class MiscFunctionsSuite extends QueryTest with SharedSparkSession {
    assert(df.selectExpr("random(1)").collect() != null)
    assert(df.select(random(lit(1))).collect() != null)
  }
+
+  test("SPARK-49306 nullifzero and zeroifnull functions") {


This test suite is for misc functions mostly. Could you move expression tests to ConditionalExpressionSuite and end-to-end tests to conditional-functions.sql, please.

Sure, it turns out all these were end to end tests so I just moved them all to conditional-functions.sql.

respond to code review comments

dtenedor

Thanks @MaxGekk @HyukjinKwon for your reviews, please take another look.

dtenedor · 2024-08-21T21:33:50Z

sql/core/src/test/scala/org/apache/spark/sql/MiscFunctionsSuite.scala

@@ -285,6 +285,48 @@ class MiscFunctionsSuite extends QueryTest with SharedSparkSession {
    assert(df.selectExpr("random(1)").collect() != null)
    assert(df.select(random(lit(1))).collect() != null)
  }
+
+  test("SPARK-49306 nullifzero and zeroifnull functions") {


Sure, it turns out all these were end to end tests so I just moved them all to conditional-functions.sql.

allisonwang-db

LGTM!

MaxGekk · 2024-08-22T05:18:55Z

+1, LGTM. Merging to master.
Thank you, @dtenedor and @HyukjinKwon @allisonwang-db for review.

…l' and 'nullifzero' ### What changes were proposed in this pull request? In #47817 we added new SQL functions `zeroifnull` and `nullifzero`. In this PR we add Scala and Python DataFrame API endpoints for them. For example, in Scala: ``` var df = Seq((0)).toDF("a") df.selectExpr("nullifzero(0)").collect() > null df.select(nullifzero(lit(0))).collect() > null df.selectExpr("nullifzero(a)").collect() > null df.select(nullifzero(lit(5))).collect() > 5 df = Seq[(Integer)]((null)).toDF("a") df.selectExpr("zeroifnull(null)").collect() > 5 df.select(nullifzero(lit(null))).collect() > 0 df.selectExpr("zeroifnull(a)").collect() > 0 df.select(zeroifnull(lit(5))) > 5 ``` ### Why are the changes needed? This improves DataFrame parity with the SQL API. ### Does this PR introduce _any_ user-facing change? Yes, see above. ### How was this patch tested? This PR adds unit test coverage. ### Was this patch authored or co-authored using generative AI tooling? No. Closes #47851 from dtenedor/dataframe-zeroifnull. Authored-by: Daniel Tenedorio <daniel.tenedorio@databricks.com> Signed-off-by: Max Gekk <max.gekk@gmail.com>

…l' and 'nullifzero' ### What changes were proposed in this pull request? In apache/spark#47817 we added new SQL functions `zeroifnull` and `nullifzero`. In this PR we add Scala and Python DataFrame API endpoints for them. For example, in Scala: ``` var df = Seq((0)).toDF("a") df.selectExpr("nullifzero(0)").collect() > null df.select(nullifzero(lit(0))).collect() > null df.selectExpr("nullifzero(a)").collect() > null df.select(nullifzero(lit(5))).collect() > 5 df = Seq[(Integer)]((null)).toDF("a") df.selectExpr("zeroifnull(null)").collect() > 5 df.select(nullifzero(lit(null))).collect() > 0 df.selectExpr("zeroifnull(a)").collect() > 0 df.select(zeroifnull(lit(5))) > 5 ``` ### Why are the changes needed? This improves DataFrame parity with the SQL API. ### Does this PR introduce _any_ user-facing change? Yes, see above. ### How was this patch tested? This PR adds unit test coverage. ### Was this patch authored or co-authored using generative AI tooling? No. Closes #47851 from dtenedor/dataframe-zeroifnull. Authored-by: Daniel Tenedorio <daniel.tenedorio@databricks.com> Signed-off-by: Max Gekk <max.gekk@gmail.com>

…ero' ### What changes were proposed in this pull request? This PR creates new SQL functions `zeroifnull` and `nullifzero`. These are equivalent to `nvl(input, 0)` and `if(input = 0, null, input)`, respectively. The analyzer performs these replacements inline in this PR to take advantage of existing query optimizations for `nvl` and `if` expressions. For example: ``` > select zeroifnull(null); 0 > select zeroifnull(2); 2 > select nullifzero(0); null > select nullifzero(2); 2 ``` ### Why are the changes needed? We are adding these functions for compatibility with other systems which also have these SQL functions. ### Does this PR introduce _any_ user-facing change? Yes, see above. ### How was this patch tested? This PR adds unit test coverage. ### Was this patch authored or co-authored using generative AI tooling? No. Closes apache#47817 from dtenedor/zeroifnull. Authored-by: Daniel Tenedorio <daniel.tenedorio@databricks.com> Signed-off-by: Max Gekk <max.gekk@gmail.com>

…l' and 'nullifzero' ### What changes were proposed in this pull request? In apache#47817 we added new SQL functions `zeroifnull` and `nullifzero`. In this PR we add Scala and Python DataFrame API endpoints for them. For example, in Scala: ``` var df = Seq((0)).toDF("a") df.selectExpr("nullifzero(0)").collect() > null df.select(nullifzero(lit(0))).collect() > null df.selectExpr("nullifzero(a)").collect() > null df.select(nullifzero(lit(5))).collect() > 5 df = Seq[(Integer)]((null)).toDF("a") df.selectExpr("zeroifnull(null)").collect() > 5 df.select(nullifzero(lit(null))).collect() > 0 df.selectExpr("zeroifnull(a)").collect() > 0 df.select(zeroifnull(lit(5))) > 5 ``` ### Why are the changes needed? This improves DataFrame parity with the SQL API. ### Does this PR introduce _any_ user-facing change? Yes, see above. ### How was this patch tested? This PR adds unit test coverage. ### Was this patch authored or co-authored using generative AI tooling? No. Closes apache#47851 from dtenedor/dataframe-zeroifnull. Authored-by: Daniel Tenedorio <daniel.tenedorio@databricks.com> Signed-off-by: Max Gekk <max.gekk@gmail.com>

…ero' ### What changes were proposed in this pull request? This PR creates new SQL functions `zeroifnull` and `nullifzero`. These are equivalent to `nvl(input, 0)` and `if(input = 0, null, input)`, respectively. The analyzer performs these replacements inline in this PR to take advantage of existing query optimizations for `nvl` and `if` expressions. For example: ``` > select zeroifnull(null); 0 > select zeroifnull(2); 2 > select nullifzero(0); null > select nullifzero(2); 2 ``` ### Why are the changes needed? We are adding these functions for compatibility with other systems which also have these SQL functions. ### Does this PR introduce _any_ user-facing change? Yes, see above. ### How was this patch tested? This PR adds unit test coverage. ### Was this patch authored or co-authored using generative AI tooling? No. Closes apache#47817 from dtenedor/zeroifnull. Authored-by: Daniel Tenedorio <daniel.tenedorio@databricks.com> Signed-off-by: Max Gekk <max.gekk@gmail.com>

…l' and 'nullifzero' ### What changes were proposed in this pull request? In apache#47817 we added new SQL functions `zeroifnull` and `nullifzero`. In this PR we add Scala and Python DataFrame API endpoints for them. For example, in Scala: ``` var df = Seq((0)).toDF("a") df.selectExpr("nullifzero(0)").collect() > null df.select(nullifzero(lit(0))).collect() > null df.selectExpr("nullifzero(a)").collect() > null df.select(nullifzero(lit(5))).collect() > 5 df = Seq[(Integer)]((null)).toDF("a") df.selectExpr("zeroifnull(null)").collect() > 5 df.select(nullifzero(lit(null))).collect() > 0 df.selectExpr("zeroifnull(a)").collect() > 0 df.select(zeroifnull(lit(5))) > 5 ``` ### Why are the changes needed? This improves DataFrame parity with the SQL API. ### Does this PR introduce _any_ user-facing change? Yes, see above. ### How was this patch tested? This PR adds unit test coverage. ### Was this patch authored or co-authored using generative AI tooling? No. Closes apache#47851 from dtenedor/dataframe-zeroifnull. Authored-by: Daniel Tenedorio <daniel.tenedorio@databricks.com> Signed-off-by: Max Gekk <max.gekk@gmail.com>

commit

39832b8

commit

github-actions bot added the SQL label Aug 20, 2024

fix test

38143fb

dtenedor changed the title ~~[SPARK-49306][SQL] Create SQL function aliases for 'zeroifnull' and 'nullifzero'~~ [SPARK-49306][SQL] Create new SQL functions 'zeroifnull' and 'nullifzero' Aug 20, 2024

HyukjinKwon reviewed Aug 21, 2024

View reviewed changes

HyukjinKwon approved these changes Aug 21, 2024

View reviewed changes

MaxGekk requested changes Aug 21, 2024

View reviewed changes

respond to code review comments

0e4b576

respond to code review comments

dtenedor commented Aug 21, 2024

View reviewed changes

allisonwang-db approved these changes Aug 21, 2024

View reviewed changes

dtenedor requested a review from MaxGekk August 21, 2024 23:29

MaxGekk approved these changes Aug 22, 2024

View reviewed changes

MaxGekk closed this in ab3cd91 Aug 22, 2024

dtenedor mentioned this pull request Aug 23, 2024

[SPARK-49306][PYTHON][SQL] Create DataFrame API support for new 'zeroifnull' and 'nullifzero' SQL functions #47851

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-49306][SQL] Create new SQL functions 'zeroifnull' and 'nullifzero' #47817

[SPARK-49306][SQL] Create new SQL functions 'zeroifnull' and 'nullifzero' #47817

dtenedor commented Aug 20, 2024 •

edited

Loading

dtenedor commented Aug 20, 2024

HyukjinKwon Aug 21, 2024

dtenedor Aug 21, 2024

MaxGekk Aug 21, 2024

dtenedor Aug 21, 2024

dtenedor left a comment

dtenedor Aug 21, 2024

allisonwang-db left a comment

MaxGekk commented Aug 22, 2024

[SPARK-49306][SQL] Create new SQL functions 'zeroifnull' and 'nullifzero' #47817

[SPARK-49306][SQL] Create new SQL functions 'zeroifnull' and 'nullifzero' #47817

Conversation

dtenedor commented Aug 20, 2024 • edited Loading

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Was this patch authored or co-authored using generative AI tooling?

dtenedor commented Aug 20, 2024

HyukjinKwon Aug 21, 2024

Choose a reason for hiding this comment

dtenedor Aug 21, 2024

Choose a reason for hiding this comment

MaxGekk Aug 21, 2024

Choose a reason for hiding this comment

dtenedor Aug 21, 2024

Choose a reason for hiding this comment

dtenedor left a comment

Choose a reason for hiding this comment

dtenedor Aug 21, 2024

Choose a reason for hiding this comment

allisonwang-db left a comment

Choose a reason for hiding this comment

MaxGekk commented Aug 22, 2024

dtenedor commented Aug 20, 2024 •

edited

Loading