[SPARK-8371][SQL] improve unit test for MaxOf and MinOf and fix bugs #6825

cloud-fan · 2015-06-15T08:16:31Z

a follow up of #6813

cloud-fan · 2015-06-15T08:24:08Z

...yst/src/test/scala/org/apache/spark/sql/catalyst/expressions/ArithmeticExpressionSuite.scala

+    checkEvaluation(MaxOf(true, false), true)
+    checkEvaluation(MaxOf("abc", "bcd"), "bcd")
+//    checkEvaluation(MaxOf(Array(1.toByte, 2.toByte), Array(1.toByte, 3.toByte)),
+//      Array(1.toByte, 3.toByte))


This test will fail at checkEvaluationWithGeneratedProjection for hash code not equal. I'm not sure why we want to compare the hash code, and I looked into the hashCode implementation of GenericRow and Row, they are different. I'm trying to figure it out.

cloud-fan · 2015-06-15T08:24:27Z

cc @rxin @davies

cloud-fan · 2015-06-15T08:26:05Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/arithmetic.scala

        }
      """
    } else {
-      super.genCode(ctx, ev)
+      ctx.defaultCodeGen(ev, this)


The origin code is not right as super here refers to BinaryArithmetic, not Expression that we wanted.

yes,i think you are right. because their symbol is empty.

SparkQA · 2015-06-15T09:25:59Z

Test build #34923 has finished for PR 6825 at commit cd51a7d.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2015-06-15T10:50:55Z

Test build #34933 has finished for PR 6825 at commit 03021bc.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

lianhuiwang · 2015-06-15T11:24:33Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/arithmetic.scala

      val eval1 = left.gen(ctx)
      val eval2 = right.gen(ctx)
+      val result = if (dataType == BooleanType) {
+        s"${ev.primitive} = ${eval1.primitive} || ${eval2.primitive};"


i think at first it needs to check whether eval1 or eval2 isNull. and then calculate primitive.

the code result is inserted after null check.

cloud-fan · 2015-06-15T14:04:28Z

retest it please.

davies · 2015-06-16T02:53:23Z

@cloud-fan There are some changes related to Max/Min in #6726 , is it useful for you? It will be good if you could merge them here.

SparkQA · 2015-06-16T16:20:38Z

Test build #34987 has finished for PR 6825 at commit f153380.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class IsNull(child: Expression) extends UnaryExpression with Predicate
- case class IsNotNull(child: Expression) extends UnaryExpression with Predicate

davies · 2015-06-17T03:49:31Z

...yst/src/test/scala/org/apache/spark/sql/catalyst/expressions/ArithmeticExpressionSuite.scala

+    checkEvaluation(MaxOf("abc", "bcd"), "bcd")
+//    checkEvaluation(MaxOf(Array(1.toByte, 2.toByte), Array(1.toByte, 3.toByte)),
+//      Array(1.toByte, 3.toByte))
+    checkEvaluation(MaxOf(java.sql.Date.valueOf("2012-11-12"), java.sql.Date.valueOf("2012-12-22")),


All expressions only work with InternalRow, which only contain internal types (int, long, UTF8String), no Date or Timestamp. so we don't need to test Date/Timestamp, or use DateUtils.fromDate() to do the conversion before calling expression on them.

cloud-fan · 2015-06-17T15:02:16Z

...yst/src/test/scala/org/apache/spark/sql/catalyst/expressions/ArithmeticExpressionSuite.scala

+    checkEvaluation(MaxOf(true, false), true)
+    checkEvaluation(MaxOf("abc", "bcd"), UTF8String.fromString("bcd"))
+//    checkEvaluation(MaxOf(Array(1.toByte, 2.toByte), Array(1.toByte, 3.toByte)),
+//      Array(1.toByte, 3.toByte))


This test will fail at checkEvaluationWithGeneratedProjection for hash code not equal. I'm not sure why we want to compare the hash code, and I looked into the hashCode implementation of GenericRow and Row, they are different. Does anyone know why we want to compare the hash code there?

SparkQA · 2015-06-17T15:14:10Z

Test build #35044 has finished for PR 6825 at commit 97066a6.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

cloud-fan · 2015-06-17T15:48:02Z

...catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/ExpressionEvalHelper.scala

@@ -109,7 +109,7 @@ trait ExpressionEvalHelper {
    }

    val actual = plan(inputRow)
-    val expectedRow = new GenericRow(Array[Any](CatalystTypeConverters.convertToCatalyst(expected)))
+    val expectedRow = InternalRow.fromSeq(Array(CatalystTypeConverters.convertToCatalyst(expected)))


@davies , we need the conversion here, as we use String instead of UTF8String at many tests...

Sound reasonable, it's anonying to have so many UTF8String.fromString in test cases.

SparkQA · 2015-06-17T17:40:24Z

Test build #35045 has finished for PR 6825 at commit 8c3fbde.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

davies · 2015-06-23T19:07:35Z

@cloud-fan Now #6876 is merged, could you update this PR?

SparkQA · 2015-06-24T06:06:14Z

Test build #35636 has finished for PR 6825 at commit 43170cc.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

davies · 2015-06-24T06:10:19Z

...atalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeGenerator.scala

@@ -175,8 +175,10 @@ class CodeGenContext {
   * Generate code for compare expression in Java
   */
  def genComp(dataType: DataType, c1: String, c2: String): String = dataType match {
+    // java boolean doesn't support > or < operator
+    case BooleanType => s"($c1 == $c2 ? 0 : ($c1 ? 1 : -1))"


Good catch!

davies · 2015-06-24T06:11:22Z

LGTM, the failed PySpark test is not related, merging this into master, thanks!

SparkQA · 2015-06-24T08:02:16Z

Test build #955 has finished for PR 6825 at commit 43170cc.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

cloud-fan reviewed Jun 15, 2015
View reviewed changes

lianhuiwang reviewed Jun 15, 2015
View reviewed changes

cloud-fan force-pushed the cg branch from 03021bc to f153380 Compare June 16, 2015 14:36

davies reviewed Jun 17, 2015
View reviewed changes

cloud-fan force-pushed the cg branch from f153380 to 97066a6 Compare June 17, 2015 14:57

cloud-fan reviewed Jun 17, 2015
View reviewed changes

cloud-fan mentioned this pull request Jun 18, 2015

[WIP] remove expensive api from InternalRow #6869

Closed

fix bugs in code gen

43170cc

cloud-fan force-pushed the cg branch from 3c9aa51 to 43170cc Compare June 24, 2015 04:43

davies reviewed Jun 24, 2015
View reviewed changes

asfgit closed this in 09fcf96 Jun 24, 2015

cloud-fan deleted the cg branch June 24, 2015 06:32

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-8371][SQL] improve unit test for MaxOf and MinOf and fix bugs #6825

[SPARK-8371][SQL] improve unit test for MaxOf and MinOf and fix bugs #6825

cloud-fan commented Jun 15, 2015

cloud-fan Jun 15, 2015

cloud-fan commented Jun 15, 2015

cloud-fan Jun 15, 2015

lianhuiwang Jun 15, 2015

SparkQA commented Jun 15, 2015

SparkQA commented Jun 15, 2015

lianhuiwang Jun 15, 2015

cloud-fan Jun 15, 2015

cloud-fan commented Jun 15, 2015

davies commented Jun 16, 2015

SparkQA commented Jun 16, 2015

davies Jun 17, 2015

cloud-fan Jun 17, 2015

SparkQA commented Jun 17, 2015

cloud-fan Jun 17, 2015

davies Jun 17, 2015

SparkQA commented Jun 17, 2015

davies commented Jun 23, 2015

SparkQA commented Jun 24, 2015

davies Jun 24, 2015

davies commented Jun 24, 2015

SparkQA commented Jun 24, 2015

[SPARK-8371][SQL] improve unit test for MaxOf and MinOf and fix bugs #6825

[SPARK-8371][SQL] improve unit test for MaxOf and MinOf and fix bugs #6825

Conversation

cloud-fan commented Jun 15, 2015

Choose a reason for hiding this comment

cloud-fan commented Jun 15, 2015

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SparkQA commented Jun 15, 2015

SparkQA commented Jun 15, 2015

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cloud-fan commented Jun 15, 2015

davies commented Jun 16, 2015

SparkQA commented Jun 16, 2015

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SparkQA commented Jun 17, 2015

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SparkQA commented Jun 17, 2015

davies commented Jun 23, 2015

SparkQA commented Jun 24, 2015

Choose a reason for hiding this comment

davies commented Jun 24, 2015

SparkQA commented Jun 24, 2015