[SPARK-4508] [SQL] build native date type to conform behavior to Hive #3381

adrian-wang · 2014-11-20T07:32:58Z

Store daysSinceEpoch as an Int value(4 bytes) to represent DateType, instead of using java.sql.Date(8 bytes as Long) in catalyst row. This ensures the same comparison behavior of Hive and Catalyst.

SparkQA · 2014-11-20T07:37:50Z

Test build #23664 has started for PR 3381 at commit f7c704a.

This patch merges cleanly.

SparkQA · 2014-11-20T07:41:13Z

Test build #23664 has finished for PR 3381 at commit f7c704a.

This patch fails to build.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- final class Date extends Ordered[Date] with Serializable

AmplabJenkins · 2014-11-20T07:41:14Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/23664/
Test FAILed.

adrian-wang · 2014-11-20T07:53:13Z

retest this please.

adrian-wang · 2014-11-20T07:55:50Z

This is a weird build error. @shaneknapp

SparkQA · 2014-11-20T08:00:02Z

Test build #23665 has started for PR 3381 at commit f7c704a.

This patch merges cleanly.

SparkQA · 2014-11-20T08:03:41Z

Test build #23665 has finished for PR 3381 at commit f7c704a.

This patch fails to build.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- final class Date extends Ordered[Date] with Serializable

AmplabJenkins · 2014-11-20T08:03:43Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/23665/
Test FAILed.

shaneknapp · 2014-11-20T17:02:31Z

looks like a compilation error, and the reason why no unit test results
were stored was that none were run (unless i'm missing something).

On Wed, Nov 19, 2014 at 11:56 PM, Daoyuan Wang notifications@github.com
wrote:

This is a weird build error. @shaneknapp https://github.com/shaneknapp

—
Reply to this email directly or view it on GitHub
#3381 (comment).

shaneknapp · 2014-11-20T18:04:49Z

Jenkins, test this please

SparkQA · 2014-11-20T18:35:39Z

Test build #23676 has started for PR 3381 at commit f7c704a.

This patch merges cleanly.

SparkQA · 2014-11-20T18:41:29Z

Test build #23676 has finished for PR 3381 at commit f7c704a.

This patch fails to build.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- final class Date extends Ordered[Date] with Serializable

AmplabJenkins · 2014-11-20T18:41:30Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/23676/
Test FAILed.

adrian-wang · 2014-11-21T01:12:19Z

This builds successfully locally, and the build error is very confusing, since I never changed anything related to that.

adrian-wang · 2014-11-25T04:44:12Z

@marmbrus @rxin @liancheng Can you help verify this build error? I cloned this branch separately and the build is successful.

liancheng · 2014-11-25T05:18:54Z

The history of this build has already timed out...

liancheng · 2014-11-25T05:18:58Z

retest this please

SparkQA · 2014-11-25T05:25:09Z

Test build #23816 has started for PR 3381 at commit f7c704a.

This patch merges cleanly.

SparkQA · 2014-11-25T05:31:31Z

Test build #23816 has finished for PR 3381 at commit f7c704a.

This patch fails to build.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- final class Date extends Ordered[Date] with Serializable

AmplabJenkins · 2014-11-25T05:31:32Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/23816/
Test FAILed.

adrian-wang · 2014-11-25T05:59:38Z

@liancheng can you reproduce the build error locally? I think it is a bug of Jenkins.

shaneknapp · 2014-11-25T06:40:26Z

i'll have the time to take a closer look at this tomorrow.

On Mon, Nov 24, 2014 at 9:59 PM, Daoyuan Wang notifications@github.com
wrote:

@liancheng https://github.com/liancheng can you reproduce the build
error locally? I think it is a bug of Jenkins.

—
Reply to this email directly or view it on GitHub
#3381 (comment).

liancheng · 2014-11-25T07:18:32Z

@adrian-wang Actually I can reproduce this compilation error with this:

./sbt/sbt -Pyarn -Phadoop-2.3 -Dhadoop.version=2.3.0 -Pkinesis-asl -Phive -Phive-0.12.0 -Phive-thriftserver clean compile

Those profiles used by Jenkins are unfortunately not printed when compiling against Hive 0.12.0. You may refer to dev/run-tests for details.

liancheng · 2014-11-25T07:19:45Z

And the compilation failure is caused by this line https://github.com/apache/spark/pull/3381/files#diff-ff50aea397a607b79df9bec6f2a841dbL23

adrian-wang · 2014-11-25T07:56:44Z

@liancheng thanks for your help! My local compilation with -Phive-0.13.1 is totally OK......
Anyway, it is ok now, with that import line back again.:)

SparkQA · 2014-11-25T08:00:03Z

Test build #23827 has started for PR 3381 at commit 0c6cab6.

This patch merges cleanly.

SparkQA · 2014-11-25T09:07:56Z

Test build #23827 has finished for PR 3381 at commit 0c6cab6.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- final class Date extends Ordered[Date] with Serializable

AmplabJenkins · 2014-11-25T09:07:59Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/23827/
Test PASSed.

liancheng · 2014-11-25T09:39:08Z

@adrian-wang Yea, Jenkins compiles Spark SQL with both Hive 0.12.0 and 0.13.1, and then runs SQL tests against 0.13.1.

marmbrus · 2014-12-17T19:57:29Z

Hi @adrian-wang, thanks for working on this and I'm very sorry for letting it fall out of date. In general this seems like a reasonable change to make, but we do need to be very careful that we don't change the output types as those are part of the public API. I could be wrong, but it doesn't seem like we are doing that now.

So I propose the following:

Close this issue for now, and reopen it once it has been updated. (we are trying very hard to keep the PR queue small and limited to active PRs so that things don't fall through the cracks).
Add tests to scala/java/python to make sure we are not changing the types that are output.

What do you think?

adrian-wang · 2014-12-18T02:39:05Z

Thanks for comments! Sorry for letting it out of date. I have no authority to reopen PRs, so I'll start a new session for this.

marmbrus · 2014-12-18T02:41:00Z

Oh, sorry I thought you could reopen your own PRs. I guess that is not the case. Either way, please open a new one when ready.

Store daysSinceEpoch as an Int value(4 bytes) to represent DateType, instead of using java.sql.Date(8 bytes as Long) in catalyst row. This ensures the same comparison behavior of Hive and Catalyst. Subsumes #3381 I thinks there are already some tests in JavaSQLSuite, and for python it will not affect python's datetime class. Author: Daoyuan Wang <daoyuan.wang@intel.com> Closes #3732 from adrian-wang/datenative and squashes the following commits: 0ed0fdc [Daoyuan Wang] fix test data a2fdd4e [Daoyuan Wang] getDate c37832b [Daoyuan Wang] row to catalyst f0005b1 [Daoyuan Wang] add date in sql parser and java type conversion 024c9a6 [Daoyuan Wang] clean some import order d6715fc [Daoyuan Wang] refactoring Date as Primitive Int internally 374abd5 [Daoyuan Wang] spark native date type support

adrian-wang added 3 commits November 19, 2014 23:23

spark native date type

4341214

api change

9a91bd8

remove outdated files

f7c704a

fix build error

0c6cab6

asfgit closed this in ca12608 Dec 17, 2014

adrian-wang mentioned this pull request Dec 18, 2014

[SPARK-4508] [SQL] build native date type to conform behavior to Hive #3732

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-4508] [SQL] build native date type to conform behavior to Hive #3381

[SPARK-4508] [SQL] build native date type to conform behavior to Hive #3381

adrian-wang commented Nov 20, 2014

SparkQA commented Nov 20, 2014

SparkQA commented Nov 20, 2014

AmplabJenkins commented Nov 20, 2014

adrian-wang commented Nov 20, 2014

adrian-wang commented Nov 20, 2014

SparkQA commented Nov 20, 2014

SparkQA commented Nov 20, 2014

AmplabJenkins commented Nov 20, 2014

shaneknapp commented Nov 20, 2014

shaneknapp commented Nov 20, 2014

SparkQA commented Nov 20, 2014

SparkQA commented Nov 20, 2014

AmplabJenkins commented Nov 20, 2014

adrian-wang commented Nov 21, 2014

adrian-wang commented Nov 25, 2014

liancheng commented Nov 25, 2014

liancheng commented Nov 25, 2014

SparkQA commented Nov 25, 2014

SparkQA commented Nov 25, 2014

AmplabJenkins commented Nov 25, 2014

adrian-wang commented Nov 25, 2014

shaneknapp commented Nov 25, 2014

liancheng commented Nov 25, 2014

liancheng commented Nov 25, 2014

adrian-wang commented Nov 25, 2014

SparkQA commented Nov 25, 2014

SparkQA commented Nov 25, 2014

AmplabJenkins commented Nov 25, 2014

liancheng commented Nov 25, 2014

marmbrus commented Dec 17, 2014

adrian-wang commented Dec 18, 2014

marmbrus commented Dec 18, 2014

[SPARK-4508] [SQL] build native date type to conform behavior to Hive #3381

[SPARK-4508] [SQL] build native date type to conform behavior to Hive #3381

Conversation

adrian-wang commented Nov 20, 2014

SparkQA commented Nov 20, 2014

SparkQA commented Nov 20, 2014

AmplabJenkins commented Nov 20, 2014

adrian-wang commented Nov 20, 2014

adrian-wang commented Nov 20, 2014

SparkQA commented Nov 20, 2014

SparkQA commented Nov 20, 2014

AmplabJenkins commented Nov 20, 2014

shaneknapp commented Nov 20, 2014

shaneknapp commented Nov 20, 2014

SparkQA commented Nov 20, 2014

SparkQA commented Nov 20, 2014

AmplabJenkins commented Nov 20, 2014

adrian-wang commented Nov 21, 2014

adrian-wang commented Nov 25, 2014

liancheng commented Nov 25, 2014

liancheng commented Nov 25, 2014

SparkQA commented Nov 25, 2014

SparkQA commented Nov 25, 2014

AmplabJenkins commented Nov 25, 2014

adrian-wang commented Nov 25, 2014

shaneknapp commented Nov 25, 2014

liancheng commented Nov 25, 2014

liancheng commented Nov 25, 2014

adrian-wang commented Nov 25, 2014

SparkQA commented Nov 25, 2014

SparkQA commented Nov 25, 2014

AmplabJenkins commented Nov 25, 2014

liancheng commented Nov 25, 2014

marmbrus commented Dec 17, 2014

adrian-wang commented Dec 18, 2014

marmbrus commented Dec 18, 2014