[SPARK-3988][SQL] add public API for date type #2901

adrian-wang · 2014-10-23T02:22:10Z

Add json and python api for date type.
By using Pickle, java.sql.Date was serialized as calendar, and recognized in python as datetime.datetime.

SparkQA · 2014-10-23T02:29:46Z

QA tests have started for PR 2901 at commit 8d7dd22.

This patch merges cleanly.

SparkQA · 2014-10-23T02:30:48Z

QA tests have finished for PR 2901 at commit 8d7dd22.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class DateType(PrimitiveType):

AmplabJenkins · 2014-10-23T02:30:49Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22053/
Test FAILed.

SparkQA · 2014-10-23T02:49:51Z

QA tests have started for PR 2901 at commit 1d74448.

This patch merges cleanly.

liancheng · 2014-10-23T03:38:38Z

sql/core/src/test/java/org/apache/spark/sql/api/java/JavaRowSuite.java

@@ -76,6 +79,7 @@ public void constructSimpleRow() {
      new Boolean(booleanValue),
      stringValue,               // StringType
      binaryValue,               // BinaryType
+      dateValue,            // DateType


Nit: indentation is off

liancheng · 2014-10-23T03:42:02Z

LGTM, thanks!

SparkQA · 2014-10-23T04:06:25Z

QA tests have finished for PR 2901 at commit 1d74448.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class DateType(PrimitiveType):

AmplabJenkins · 2014-10-23T04:06:29Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22055/
Test FAILed.

SparkQA · 2014-10-23T04:39:46Z

QA tests have started for PR 2901 at commit 444f100.

This patch merges cleanly.

SparkQA · 2014-10-23T04:49:52Z

QA tests have started for PR 2901 at commit f760d8e.

This patch merges cleanly.

SparkQA · 2014-10-23T06:13:42Z

QA tests have finished for PR 2901 at commit 444f100.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class DateType(PrimitiveType):

AmplabJenkins · 2014-10-23T06:13:45Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22058/
Test PASSed.

SparkQA · 2014-10-23T06:22:15Z

QA tests have finished for PR 2901 at commit f760d8e.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class DateType(PrimitiveType):

AmplabJenkins · 2014-10-23T06:22:18Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22059/
Test PASSed.

davies · 2014-10-23T22:30:05Z

python/pyspark/sql.py

@@ -1065,7 +1074,9 @@ def applySchema(self, rdd, schema):
        [Row(field1=1, field2=u'row1'),..., Row(field1=3, field2=u'row3')]

        >>> from datetime import datetime
+        >>> from datetime import date


minor: these two lines can be combined.

davies · 2014-10-23T22:53:20Z

Thanks for fix so many typos!

It will be awesome to recognize all Date/Timestamps values in JsonRDD. If it's not easy to do it in this PR, we could do it in another one.

adrian-wang · 2014-10-24T08:59:45Z

python/pyspark/sql.py

+        ...     lambda x: (x.byte1, x.byte2, x.short1, x.short2, x.int, x.float, x.date,
+        ...         x.time, x.map["a"], x.struct.b, x.list, x.null))
+        >>> results.collect()[0] # doctest: +NORMALIZE_WHITESPACE
+        (127, -128, -32768, 32767, 2147483647, 1.0, datetime.datetime(2010, 1, 1, 0, 0),


@davies because of using pyrolite, java.sql.Date is serialized in the same way as java.sql.Timestamp, since they are all subtype of java.util.Date. And this make the dumps() function to generate datetime instead of date for java.util.Date. I think this is related to your comments in JIRA SPARK-2674

Oh, I see. After the data was deserialized in Python, we need to some data coversions, so we can convert datetime to date if DataType is DateType.

so should the convert in python side or scala side, which one would you prefer?

I think we can only do it in Python side.

SparkQA · 2014-10-24T08:59:56Z

Test build #22133 has started for PR 2901 at commit 5670626.

This patch merges cleanly.

SparkQA · 2014-10-24T10:33:02Z

Test build #22133 has finished for PR 2901 at commit 5670626.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class DateType(PrimitiveType):

AmplabJenkins · 2014-10-24T10:33:05Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22133/
Test PASSed.

SparkQA · 2014-10-25T08:09:28Z

Test build #440 has started for PR 2901 at commit 5670626.

This patch merges cleanly.

SparkQA · 2014-10-25T09:30:50Z

Test build #440 has finished for PR 2901 at commit 5670626.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class DateType(PrimitiveType):

adrian-wang · 2014-10-25T11:07:24Z

retest this please.

SparkQA · 2014-10-25T11:14:41Z

Test build #22215 has started for PR 2901 at commit 5670626.

This patch merges cleanly.

SparkQA · 2014-10-25T12:46:25Z

Test build #22215 has finished for PR 2901 at commit 5670626.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class DateType(PrimitiveType):

AmplabJenkins · 2014-10-25T12:46:29Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22215/
Test PASSed.

marmbrus · 2014-10-26T22:01:06Z

@davies does anything else remain to be done here?

davies · 2014-10-27T04:42:55Z

@marmbrus There is a bug: DateType object will be datetime in Python. Also we could improve DataType/TimestampType support in jsonRDD, it could be done separately.

@adrian-wang You could convert datetime object into date object in _create_properties/_create_object

davies · 2014-10-27T04:48:17Z

@adrian-wang You can do it like this:

diff --git a/python/pyspark/sql.py b/python/pyspark/sql.py
index 7daf306..db256a4 100644
--- a/python/pyspark/sql.py
+++ b/python/pyspark/sql.py
@@ -782,6 +782,8 @@ def _restore_object(dataType, obj):

 def _create_object(cls, v):
     """ Create an customized object with class `cls`. """
+    if cls is datetime.date and isinstance(v, datetime.datetime):
+        return v.date()
     return cls(v) if v is not None else v


@@ -796,9 +798,11 @@ def _create_getter(dt, i):


 def _has_struct(dt):
-    """Return whether `dt` is or has StructType in it"""
+    """Return whether `dt` is or has StructType/DateType in it"""
     if isinstance(dt, StructType):
         return True
+    if isinstance(dt, DateType):
+        return True
     elif isinstance(dt, ArrayType):
         return _has_struct(dt.elementType)
     elif isinstance(dt, MapType):
@@ -870,6 +874,9 @@ def _create_cls(dataType):

         return Dict

+    elif isinstance(dataType, DateType):
+        return datetime.date
+
     elif not isinstance(dataType, StructType):
         raise Exception("unexpected data type: %s" % dataType)

adrian-wang · 2014-10-27T05:09:26Z

@davies
Thanks for the tips! I'll follow up very soon.

SparkQA · 2014-10-27T05:34:48Z

Test build #22274 has started for PR 2901 at commit c51a24d.

This patch merges cleanly.

davies · 2014-10-27T05:57:16Z

LGTM, waiting for the tests.

SparkQA · 2014-10-27T07:15:37Z

Test build #22274 has finished for PR 2901 at commit c51a24d.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class DateType(PrimitiveType):

AmplabJenkins · 2014-10-27T07:15:40Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22274/
Test PASSed.

marmbrus · 2014-10-28T20:43:42Z

Thanks! Merged to master.

This implement the feature davies mentioned in #2901 (diff) Author: Daoyuan Wang <daoyuan.wang@intel.com> Closes #3012 from adrian-wang/iso8601 and squashes the following commits: 50df6e7 [Daoyuan Wang] json data timestamp ISO8601 support

This implement the feature davies mentioned in #2901 (diff) Author: Daoyuan Wang <daoyuan.wang@intel.com> Closes #3012 from adrian-wang/iso8601 and squashes the following commits: 50df6e7 [Daoyuan Wang] json data timestamp ISO8601 support (cherry picked from commit a1fc059) Signed-off-by: Michael Armbrust <michael@databricks.com>

add json and python api for date type

8d7dd22

fix scala style

1d74448

liancheng reviewed Oct 23, 2014
View reviewed changes

fix a typo

444f100

fix indent

f760d8e

davies reviewed Oct 23, 2014
View reviewed changes

minor line combine

5670626

adrian-wang reviewed Oct 24, 2014
View reviewed changes

convert datetime to date

c51a24d

asfgit closed this in 47a40f6 Oct 28, 2014

adrian-wang mentioned this pull request Oct 30, 2014

[SPARK-4149][SQL] ISO 8601 support for json date time strings #3012

Closed

[SPARK-3988][SQL] add public API for date type #2901

[SPARK-3988][SQL] add public API for date type #2901

Conversation

adrian-wang commented Oct 23, 2014

SparkQA commented Oct 23, 2014

SparkQA commented Oct 23, 2014

AmplabJenkins commented Oct 23, 2014

SparkQA commented Oct 23, 2014

liancheng Oct 23, 2014

Choose a reason for hiding this comment

liancheng commented Oct 23, 2014

SparkQA commented Oct 23, 2014

AmplabJenkins commented Oct 23, 2014

SparkQA commented Oct 23, 2014

SparkQA commented Oct 23, 2014

SparkQA commented Oct 23, 2014

AmplabJenkins commented Oct 23, 2014

SparkQA commented Oct 23, 2014

AmplabJenkins commented Oct 23, 2014

davies Oct 23, 2014

Choose a reason for hiding this comment

davies commented Oct 23, 2014

adrian-wang Oct 24, 2014

Choose a reason for hiding this comment

davies Oct 24, 2014

Choose a reason for hiding this comment

adrian-wang Oct 24, 2014

Choose a reason for hiding this comment

davies Oct 24, 2014

Choose a reason for hiding this comment

SparkQA commented Oct 24, 2014

SparkQA commented Oct 24, 2014

AmplabJenkins commented Oct 24, 2014

SparkQA commented Oct 25, 2014

SparkQA commented Oct 25, 2014

adrian-wang commented Oct 25, 2014

SparkQA commented Oct 25, 2014

SparkQA commented Oct 25, 2014

AmplabJenkins commented Oct 25, 2014

marmbrus commented Oct 26, 2014

davies commented Oct 27, 2014

davies commented Oct 27, 2014

adrian-wang commented Oct 27, 2014

SparkQA commented Oct 27, 2014

davies commented Oct 27, 2014

SparkQA commented Oct 27, 2014

AmplabJenkins commented Oct 27, 2014

marmbrus commented Oct 28, 2014