[SPARK-8238][SPARK-8239][SPARK-8242][SPARK-8243][SPARK-8268][SQL]Add ascii/base64/unbase64/encode/decode functions #6843

chenghao-intel · 2015-06-16T14:36:42Z

Add ascii,base64,unbase64,encode and decode expressions.

SparkQA · 2015-06-16T14:41:24Z

Test build #34986 has finished for PR 6843 at commit b341b21.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class Ascii(child: Expression) extends UnaryExpression with ExpectsInputTypes
- case class Base64(child: Expression) extends UnaryExpression with ExpectsInputTypes
- case class UnBase64(child: Expression) extends UnaryExpression with ExpectsInputTypes
- case class Decode(bin: Expression, charset: Expression)
- case class Encode(value: Expression, charset: Expression)

chenghao-intel · 2015-06-16T15:16:49Z

@adrian-wang @zhichao-li can you review the code for me?

SparkQA · 2015-06-16T17:10:12Z

Test build #34990 has finished for PR 6843 at commit 6a71481.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class Ascii(child: Expression) extends UnaryExpression with ExpectsInputTypes
- case class Base64(child: Expression) extends UnaryExpression with ExpectsInputTypes
- case class UnBase64(child: Expression) extends UnaryExpression with ExpectsInputTypes
- case class Decode(bin: Expression, charset: Expression)
- case class Encode(value: Expression, charset: Expression)

adrian-wang · 2015-06-17T02:57:30Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringOperations.scala

+    }
+  }
+
+  override def toString: String = s"ascii($child)"


Upper case these all?

SparkQA · 2015-06-18T15:24:41Z

Test build #35132 has finished for PR 6843 at commit 05cc18e.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class Ascii(child: Expression) extends UnaryExpression
- case class Base64(child: Expression) extends UnaryExpression
- case class UnBase64(child: Expression) extends UnaryExpression
- case class Decode(bin: Expression, charset: Expression) extends Expression
- case class Encode(value: Expression, charset: Expression) extends Expression

chenghao-intel · 2015-06-19T00:36:46Z

@rxin, any more comments on this?

rxin · 2015-06-19T05:31:20Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringOperations.scala

+    }
+  }
+
+  override def toString: String = s"ASCII($child)"


i don't think this is necessary since it is a case class?

btw i don't think we should be making everything uppercase..

Probably the registered function name will make more sense, than giving the user a class name. What do you think?

SparkQA · 2015-06-23T08:19:33Z

Test build #35522 has finished for PR 6843 at commit a5e89b6.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class Ascii(child: Expression) extends UnaryExpression
- case class Base64(child: Expression) extends UnaryExpression
- case class UnBase64(child: Expression) extends UnaryExpression
- case class Decode(bin: Expression, charset: Expression) extends Expression
- case class Encode(value: Expression, charset: Expression) extends Expression

SparkQA · 2015-07-02T04:54:37Z

Test build #36330 has finished for PR 6843 at commit da6a788.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class Ascii(child: Expression) extends UnaryExpression with AutoCastInputTypes
- case class Base64(child: Expression) extends UnaryExpression with AutoCastInputTypes
- case class UnBase64(child: Expression) extends UnaryExpression with AutoCastInputTypes
- case class Decode(bin: Expression, charset: Expression)
- case class Encode(value: Expression, charset: Expression)

SparkQA · 2015-07-02T06:59:03Z

Test build #36341 has finished for PR 6843 at commit 947a88a.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- trait ExpectsInputTypes
- trait AutoCastInputTypes
- abstract class BinaryExpression extends Expression with trees.BinaryNode[Expression]
- abstract class BinaryOperator extends BinaryExpression
- abstract class BinaryArithmetic extends BinaryOperator
- abstract class BinaryComparison extends BinaryOperator with Predicate
- case class Ascii(child: Expression) extends UnaryExpression with AutoCastInputTypes
- case class Base64(child: Expression) extends UnaryExpression with AutoCastInputTypes
- case class UnBase64(child: Expression) extends UnaryExpression with AutoCastInputTypes
- case class Decode(bin: Expression, charset: Expression)
- case class Encode(value: Expression, charset: Expression)

SparkQA · 2015-07-03T15:19:29Z

Test build #36500 has finished for PR 6843 at commit 9d6f9f4.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class Factorial(child: Expression) extends UnaryExpression with ExpectsInputTypes
- case class Ascii(child: Expression) extends UnaryExpression with ExpectsInputTypes
- case class Base64(child: Expression) extends UnaryExpression with ExpectsInputTypes
- case class UnBase64(child: Expression) extends UnaryExpression with ExpectsInputTypes
- case class Decode(bin: Expression, charset: Expression) extends Expression with ExpectsInputTypes
- case class Encode(value: Expression, charset: Expression)

SparkQA · 2015-07-03T15:24:17Z

Test build #36501 has finished for PR 6843 at commit 78dee7d.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class Ascii(child: Expression) extends UnaryExpression with ExpectsInputTypes
- case class Base64(child: Expression) extends UnaryExpression with ExpectsInputTypes
- case class UnBase64(child: Expression) extends UnaryExpression with ExpectsInputTypes
- case class Decode(bin: Expression, charset: Expression) extends Expression with ExpectsInputTypes
- case class Encode(value: Expression, charset: Expression)

rxin · 2015-07-04T06:39:11Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringOperations.scala

+ * (one of 'US-ASCII', 'ISO-8859-1', 'UTF-8', 'UTF-16BE', 'UTF-16LE', 'UTF-16').
+ * If either argument is null, the result will also be null. (As of Hive 0.12.0.).
+ */
+case class Decode(bin: Expression, charset: Expression) extends Expression with ExpectsInputTypes {


can you make this extend BinaryExpression? You can just define def bin = left, and def charset = right.

Actually that's my intention, as I think the parameters is asymmetric semantically. Not sure if you are thinking the code impovement like #7157?

rxin · 2015-07-04T06:39:18Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringOperations.scala

+/**
+ * Decodes the first argument into a String using the provided character set
+ * (one of 'US-ASCII', 'ISO-8859-1', 'UTF-8', 'UTF-16BE', 'UTF-16LE', 'UTF-16').
+ * If either argument is null, the result will also be null. (As of Hive 0.12.0.).


remove "As of Hive 0.12.0"

rxin · 2015-07-04T06:45:11Z

I'm going to merge this first. Please submit a cleanup pr.

This is a the follow up of #6843. Author: Cheng Hao <hao.cheng@intel.com> Closes #7230 from chenghao-intel/str_funcs2_followup and squashes the following commits: 52cc553 [Cheng Hao] update the code as comment

chenghao-intel changed the title ~~[SPARK-8238][SPARK-8239][SPARK-8242][SPARK-8243][SPARK-8268][SQL]add ascii/base64/unbase64/encode/decode functions~~ [SPARK-8238][SPARK-8239][SPARK-8242][SPARK-8243][SPARK-8268][SQL][WIP] Add ascii/base64/unbase64/encode/decode functions Jun 16, 2015

adrian-wang reviewed Jun 17, 2015
View reviewed changes

chenghao-intel changed the title ~~[SPARK-8238][SPARK-8239][SPARK-8242][SPARK-8243][SPARK-8268][SQL][WIP] Add ascii/base64/unbase64/encode/decode functions~~ [SPARK-8238][SPARK-8239][SPARK-8242][SPARK-8243][SPARK-8268][SQL]Add ascii/base64/unbase64/encode/decode functions Jun 18, 2015

rxin reviewed Jun 19, 2015
View reviewed changes

chenghao-intel force-pushed the str_funcs2 branch from a5e89b6 to da6a788 Compare July 2, 2015 03:11

chenghao-intel force-pushed the str_funcs2 branch from da6a788 to 947a88a Compare July 2, 2015 05:00

chenghao-intel added 5 commits July 3, 2015 21:26

add ascii/base64/unbase64/encode/decode functions

491ce7b

remove the unused import

e2df768

scalastyle issues

96170fc

update code as comments

ed5c19c

remove the toString method for expressions

9d6f9f4

chenghao-intel force-pushed the str_funcs2 branch from 947a88a to 9d6f9f4 Compare July 3, 2015 13:42

base 64 -> base64

78dee7d

rxin reviewed Jul 4, 2015
View reviewed changes

asfgit closed this in f35b0c3 Jul 4, 2015

chenghao-intel mentioned this pull request Jul 6, 2015

[SQL][Minor] Update the DataFrame API for encode/decode #7230

Closed

chenghao-intel deleted the str_funcs2 branch July 6, 2015 00:44

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-8238][SPARK-8239][SPARK-8242][SPARK-8243][SPARK-8268][SQL]Add ascii/base64/unbase64/encode/decode functions #6843

[SPARK-8238][SPARK-8239][SPARK-8242][SPARK-8243][SPARK-8268][SQL]Add ascii/base64/unbase64/encode/decode functions #6843

chenghao-intel commented Jun 16, 2015

SparkQA commented Jun 16, 2015

chenghao-intel commented Jun 16, 2015

SparkQA commented Jun 16, 2015

adrian-wang Jun 17, 2015

SparkQA commented Jun 18, 2015

chenghao-intel commented Jun 19, 2015

rxin Jun 19, 2015

rxin Jun 19, 2015

chenghao-intel Jun 19, 2015

SparkQA commented Jun 23, 2015

SparkQA commented Jul 2, 2015

SparkQA commented Jul 2, 2015

SparkQA commented Jul 3, 2015

SparkQA commented Jul 3, 2015

rxin Jul 4, 2015

chenghao-intel Jul 5, 2015

rxin Jul 4, 2015

rxin commented Jul 4, 2015

[SPARK-8238][SPARK-8239][SPARK-8242][SPARK-8243][SPARK-8268][SQL]Add ascii/base64/unbase64/encode/decode functions #6843

[SPARK-8238][SPARK-8239][SPARK-8242][SPARK-8243][SPARK-8268][SQL]Add ascii/base64/unbase64/encode/decode functions #6843

Conversation

chenghao-intel commented Jun 16, 2015

SparkQA commented Jun 16, 2015

chenghao-intel commented Jun 16, 2015

SparkQA commented Jun 16, 2015

adrian-wang Jun 17, 2015

Choose a reason for hiding this comment

SparkQA commented Jun 18, 2015

chenghao-intel commented Jun 19, 2015

rxin Jun 19, 2015

Choose a reason for hiding this comment

rxin Jun 19, 2015

Choose a reason for hiding this comment

chenghao-intel Jun 19, 2015

Choose a reason for hiding this comment

SparkQA commented Jun 23, 2015

SparkQA commented Jul 2, 2015

SparkQA commented Jul 2, 2015

SparkQA commented Jul 3, 2015

SparkQA commented Jul 3, 2015

rxin Jul 4, 2015

Choose a reason for hiding this comment

chenghao-intel Jul 5, 2015

Choose a reason for hiding this comment

rxin Jul 4, 2015

Choose a reason for hiding this comment

rxin commented Jul 4, 2015