[SPARK-23926][SQL] Extending reverse function to support ArrayType arguments #21034

mn-mikke · 2018-04-10T21:00:06Z

What changes were proposed in this pull request?

This PR extends reverse functions to be able to operate over array columns and covers:

Introduction of Reverse expression that represents logic for reversing arrays and also strings
Removal of StringReverse expression
A wrapper for PySpark

How was this patch tested?

New tests added into:

CollectionExpressionsSuite
DataFrameFunctionsSuite

Codegen examples

Primitive type

val df = Seq(
  Seq(1, 3, 4, 2),
  null
).toDF("i")
df.filter($"i".isNotNull || $"i".isNull).select(reverse($"i")).debugCodegen

Result:

/* 032 */         boolean inputadapter_isNull = inputadapter_row.isNullAt(0);
/* 033 */         ArrayData inputadapter_value = inputadapter_isNull ?
/* 034 */         null : (inputadapter_row.getArray(0));
/* 035 */
/* 036 */         boolean filter_value = true;
/* 037 */
/* 038 */         if (!(!inputadapter_isNull)) {
/* 039 */           filter_value = inputadapter_isNull;
/* 040 */         }
/* 041 */         if (!filter_value) continue;
/* 042 */
/* 043 */         ((org.apache.spark.sql.execution.metric.SQLMetric) references[0] /* numOutputRows */).add(1);
/* 044 */
/* 045 */         boolean project_isNull = inputadapter_isNull;
/* 046 */         ArrayData project_value = null;
/* 047 */
/* 048 */         if (!inputadapter_isNull) {
/* 049 */           final int project_length = inputadapter_value.numElements();
/* 050 */           project_value = inputadapter_value.copy();
/* 051 */           for(int k = 0; k < project_length / 2; k++) {
/* 052 */             int l = project_length - k - 1;
/* 053 */             boolean isNullAtK = project_value.isNullAt(k);
/* 054 */             boolean isNullAtL = project_value.isNullAt(l);
/* 055 */             if(!isNullAtK) {
/* 056 */               int el = project_value.getInt(k);
/* 057 */               if(!isNullAtL) {
/* 058 */                 project_value.setInt(k, project_value.getInt(l));
/* 059 */               } else {
/* 060 */                 project_value.setNullAt(k);
/* 061 */               }
/* 062 */               project_value.setInt(l, el);
/* 063 */             } else if (!isNullAtL) {
/* 064 */               project_value.setInt(k, project_value.getInt(l));
/* 065 */               project_value.setNullAt(l);
/* 066 */             }
/* 067 */           }
/* 068 */
/* 069 */         }

Non-primitive type

val df = Seq(
  Seq("a", "c", "d", "b"),
  null
).toDF("s")
df.filter($"s".isNotNull || $"s".isNull).select(reverse($"s")).debugCodegen

Result:

/* 032 */         boolean inputadapter_isNull = inputadapter_row.isNullAt(0);
/* 033 */         ArrayData inputadapter_value = inputadapter_isNull ?
/* 034 */         null : (inputadapter_row.getArray(0));
/* 035 */
/* 036 */         boolean filter_value = true;
/* 037 */
/* 038 */         if (!(!inputadapter_isNull)) {
/* 039 */           filter_value = inputadapter_isNull;
/* 040 */         }
/* 041 */         if (!filter_value) continue;
/* 042 */
/* 043 */         ((org.apache.spark.sql.execution.metric.SQLMetric) references[0] /* numOutputRows */).add(1);
/* 044 */
/* 045 */         boolean project_isNull = inputadapter_isNull;
/* 046 */         ArrayData project_value = null;
/* 047 */
/* 048 */         if (!inputadapter_isNull) {
/* 049 */           final int project_length = inputadapter_value.numElements();
/* 050 */           project_value = new org.apache.spark.sql.catalyst.util.GenericArrayData(new Object[project_length]);
/* 051 */           for(int k = 0; k < project_length; k++) {
/* 052 */             int l = project_length - k - 1;
/* 053 */             project_value.update(k, inputadapter_value.getUTF8String(l));
/* 054 */           }
/* 055 */
/* 056 */         }

…guments.

viirya · 2018-04-11T00:07:10Z

...catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala

+
+  lazy val elementType: DataType = dataType.asInstanceOf[ArrayType].elementType
+
+  override def checkInputDataTypes(): TypeCheckResult = {


I think you can just make the inputTypes as Seq(TypeCollection(StringType, ArrayType)). ExpectsInputTypes can check the types.

Great idea!

gatorsmile · 2018-04-11T06:09:35Z

ok to test

SparkQA · 2018-04-11T07:05:02Z

Test build #89175 has finished for PR 21034 at commit 3a76d87.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds the following public classes (experimental):
case class Reverse(child: Expression) extends UnaryExpression with ImplicitCastInputTypes

kiszk · 2018-04-11T09:17:11Z

retest this please

SparkQA · 2018-04-11T11:04:11Z

Test build #89183 has finished for PR 21034 at commit 3a76d87.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
case class Reverse(child: Expression) extends UnaryExpression with ImplicitCastInputTypes

HyukjinKwon · 2018-04-11T12:40:31Z

...catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala

+      > SELECT _FUNC_(array(2, 1, 4, 3));
+       [3, 4, 1, 2]
+  """,
+  since = "2.4.0")


Hmmm. I think the function reverse in SQL is there from 1.5 - SPARK-8258. Can we use 1.5.0 here and add a note via note field saying array type is supported from 2.4.0? there's an example for note:

spark/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/regexpExpressions.scala

Lines 101 to 103 in 2ce37b5

note = """

Use RLIKE to match with standard regular expressions.

""")

kiszk · 2018-04-11T18:16:24Z

...catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala

+    }
+
+    s"""
+    |final int $length = $childName.numElements();


nit: more spaces

SparkQA · 2018-04-11T18:39:33Z

Test build #89209 has finished for PR 21034 at commit 28ed664.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2018-04-11T22:57:42Z

Test build #89219 has finished for PR 21034 at commit 638a0ff.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2018-04-12T13:12:07Z

Test build #89256 has finished for PR 21034 at commit 30af8cd.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
logInfo(s\"Failed to load main class $childMainClass.\")
error(s\"Cannot load main class from JAR $primaryResource\")
error(\"No main class set in JAR; please specify one with --class\")
checkArgument(mainClass != null, \"Missing example class name.\");
sealed trait Node extends Serializable
sealed trait ClassificationNode extends Node
sealed trait RegressionNode extends Node
sealed trait LeafNode extends Node
sealed trait InternalNode extends Node
class VectorAssembler(JavaTransformer, HasInputCols, HasOutputCol, HasHandleInvalid, JavaMLReadable,
class KolmogorovSmirnovTest(object):
case class ExprCode(var code: String, var isNull: ExprValue, var value: ExprValue)
case class SubExprEliminationState(isNull: ExprValue, value: ExprValue)
|class SpecificUnsafeProjection extends $
trait JavaCode
trait ExprValue extends JavaCode
case class SimpleExprValue(expr: String, javaType: Class[_]) extends ExprValue
case class VariableValue(variableName: String, javaType: Class[_]) extends ExprValue
case class GlobalValue(value: String, javaType: Class[_]) extends ExprValue
class LiteralValue(val value: String, val javaType: Class[_]) extends ExprValue with Serializable

…verse-to-master

SparkQA · 2018-04-13T00:37:45Z

Test build #89301 has finished for PR 21034 at commit ad17d49.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2018-04-13T17:10:37Z

Test build #89345 has finished for PR 21034 at commit aca511c.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

mn-mikke · 2018-04-15T19:01:11Z

Any other comments?

SparkQA · 2018-04-16T16:53:59Z

Test build #89396 has finished for PR 21034 at commit 460fea6.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

ueshin · 2018-04-17T02:44:01Z

LGTM.

ueshin · 2018-04-18T09:40:58Z

Thanks! merging to master.

mn-mikke · 2018-04-18T09:44:24Z

Thanks everybody for code reviews!

[SPARK-23926][SQL] Extending reverse function to support ArrayType ar…

3a76d87

…guments.

viirya reviewed Apr 11, 2018

View reviewed changes

HyukjinKwon reviewed Apr 11, 2018

View reviewed changes

[SPARK-23926][SQL] Adding more tests + fixing a bug in codegen.

28ed664

kiszk reviewed Apr 11, 2018

View reviewed changes

[SPARK-23926][SQL] Styling fix - indentation

638a0ff

[SPARK-23926][SQL] Merging current master to the feature branch

30af8cd

Merge remote-tracking branch 'spark/master' into feature/array-api-re…

ad17d49

…verse-to-master

[SPARK-23736][SQL] Fixing Python test

aca511c

[SPARK-23926][SQL] Merging current master to the feature branch.

460fea6

asfgit closed this in f81fa47 Apr 18, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-23926][SQL] Extending reverse function to support ArrayType arguments #21034

[SPARK-23926][SQL] Extending reverse function to support ArrayType arguments #21034

mn-mikke commented Apr 10, 2018 •

edited

Loading

viirya Apr 11, 2018 •

edited

Loading

mn-mikke Apr 11, 2018

gatorsmile commented Apr 11, 2018

SparkQA commented Apr 11, 2018

kiszk commented Apr 11, 2018

SparkQA commented Apr 11, 2018

HyukjinKwon Apr 11, 2018 •

edited

Loading

kiszk Apr 11, 2018

SparkQA commented Apr 11, 2018

SparkQA commented Apr 11, 2018

SparkQA commented Apr 12, 2018

SparkQA commented Apr 13, 2018

SparkQA commented Apr 13, 2018

mn-mikke commented Apr 15, 2018

SparkQA commented Apr 16, 2018

ueshin commented Apr 17, 2018

ueshin commented Apr 18, 2018

mn-mikke commented Apr 18, 2018


		lazy val elementType: DataType = dataType.asInstanceOf[ArrayType].elementType

		override def checkInputDataTypes(): TypeCheckResult = {

	note = """
	Use RLIKE to match with standard regular expressions.
	""")

[SPARK-23926][SQL] Extending reverse function to support ArrayType arguments #21034

[SPARK-23926][SQL] Extending reverse function to support ArrayType arguments #21034

Conversation

mn-mikke commented Apr 10, 2018 • edited Loading

What changes were proposed in this pull request?

How was this patch tested?

Codegen examples

Primitive type

Non-primitive type

viirya Apr 11, 2018 • edited Loading

Choose a reason for hiding this comment

mn-mikke Apr 11, 2018

Choose a reason for hiding this comment

gatorsmile commented Apr 11, 2018

SparkQA commented Apr 11, 2018

kiszk commented Apr 11, 2018

SparkQA commented Apr 11, 2018

HyukjinKwon Apr 11, 2018 • edited Loading

Choose a reason for hiding this comment

kiszk Apr 11, 2018

Choose a reason for hiding this comment

SparkQA commented Apr 11, 2018

SparkQA commented Apr 11, 2018

SparkQA commented Apr 12, 2018

SparkQA commented Apr 13, 2018

SparkQA commented Apr 13, 2018

mn-mikke commented Apr 15, 2018

SparkQA commented Apr 16, 2018

ueshin commented Apr 17, 2018

ueshin commented Apr 18, 2018

mn-mikke commented Apr 18, 2018

mn-mikke commented Apr 10, 2018 •

edited

Loading

viirya Apr 11, 2018 •

edited

Loading

HyukjinKwon Apr 11, 2018 •

edited

Loading