[SPARK-4205][SQL] Timestamp and Date with comparisons #3158

culler · 2014-11-07T16:53:41Z

This PR adds new RichDate and RichTimestamp classes that provide comparison operators, allowing them to be used in DSL expressions, and initializers which accept string representations of dates or times.

AmplabJenkins · 2014-11-07T16:57:10Z

Can one of the admins verify this patch?

culler · 2014-11-07T17:00:37Z

@liangcheng and @rxin, I am reopening pull request #3066 as #3158 so it can be based on a current commit of the spark source. I messed up #3066 by trying to rebase it after a new test file was added which required minor changes to compile. Sorry for any confusion this causes.

marmbrus · 2014-11-07T21:01:48Z

Thanks for working on this. Is a pretty big change so I'll review it next week after we get some critical fixes in for 1.2.

culler · 2014-11-07T22:09:04Z

Thanks @marmbrus. That sounds good. If it isn't too much trouble, I'd appreciate seeing what jenkins has to say before you review it next week.

marmbrus · 2014-11-07T22:14:00Z

ok to test

SparkQA · 2014-11-07T22:20:08Z

Test build #23075 has started for PR 3158 at commit ef5e4a4.

This patch merges cleanly.

SparkQA · 2014-11-07T22:29:00Z

Test build #23075 has finished for PR 3158 at commit ef5e4a4.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class LhsLiteral(x: Any)
- final class MutableDate extends MutableValue
- final class MutableTimestamp extends MutableValue
- class RichDate(milliseconds: Long) extends Date(milliseconds)
- class RichTimestamp(milliseconds: Long) extends Timestamp(milliseconds)

AmplabJenkins · 2014-11-07T22:29:02Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/23075/
Test FAILed.

AmplabJenkins · 2014-11-07T22:43:17Z

Can one of the admins verify this patch?

AmplabJenkins · 2014-11-07T22:52:17Z

Can one of the admins verify this patch?

culler · 2014-11-08T14:07:13Z

Oh well. Some new tests were added Friday to sql/core/src/test/scala/org/apache/spark/sql/parquet/ParquetQuerySuite.scala which use the construction assert(X === Y). Except for adding a few tests for the new features in this PR, all of the changes to the test files consisted of replacing
assert(X === Y) by assert(convertToEqualizer(X).===(Y)), which is what the scalatest documentation recommends in cases where an === operator is already defined for X. In this situation the === operators are being added by the implicit conversion which enables DSL expressions to begin with a literal. The affected tests are the ones which import these implicit conversions.

SparkQA · 2014-11-08T19:53:31Z

Test build #23104 has started for PR 3158 at commit b61540e.

This patch merges cleanly.

SparkQA · 2014-11-08T20:01:23Z

Test build #23104 has finished for PR 3158 at commit b61540e.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class LhsLiteral(x: Any)
- final class MutableDate extends MutableValue
- final class MutableTimestamp extends MutableValue
- class RichDate(milliseconds: Long) extends Date(milliseconds)
- class RichTimestamp(milliseconds: Long) extends Timestamp(milliseconds)

AmplabJenkins · 2014-11-08T20:01:27Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/23104/
Test FAILed.

SparkQA · 2014-11-08T20:59:59Z

Test build #23105 has started for PR 3158 at commit e1ba56e.

This patch merges cleanly.

SparkQA · 2014-11-08T21:02:36Z

Test build #23106 has started for PR 3158 at commit 97d00a8.

This patch merges cleanly.

SparkQA · 2014-11-08T21:08:12Z

Test build #23105 has finished for PR 3158 at commit e1ba56e.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class LhsLiteral(x: Any)
- final class MutableDate extends MutableValue
- final class MutableTimestamp extends MutableValue
- class RichDate(milliseconds: Long) extends Date(milliseconds)
- class RichTimestamp(milliseconds: Long) extends Timestamp(milliseconds)

AmplabJenkins · 2014-11-08T21:08:15Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/23105/
Test FAILed.

SparkQA · 2014-11-08T22:16:15Z

Test build #23106 has finished for PR 3158 at commit 97d00a8.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class LhsLiteral(x: Any)
- final class MutableDate extends MutableValue
- final class MutableTimestamp extends MutableValue
- class RichDate(milliseconds: Long) extends Date(milliseconds)
- class RichTimestamp(milliseconds: Long) extends Timestamp(milliseconds)

AmplabJenkins · 2014-11-08T22:16:19Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/23106/
Test PASSed.

SparkQA · 2014-11-09T16:57:44Z

Test build #23123 has started for PR 3158 at commit c5fb299.

This patch merges cleanly.

SparkQA · 2014-11-09T17:32:57Z

Test build #23123 has finished for PR 3158 at commit c5fb299.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class LhsLiteral(x: Any)
- final class MutableDate extends MutableValue
- final class MutableTimestamp extends MutableValue
- class RichDate(milliseconds: Long) extends Date(milliseconds)
- class RichTimestamp(milliseconds: Long) extends Timestamp(milliseconds)

AmplabJenkins · 2014-11-09T17:32:59Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/23123/
Test FAILed.

SparkQA · 2014-11-10T03:39:54Z

Test build #23130 has started for PR 3158 at commit 0af7a3d.

This patch merges cleanly.

allowing them to be used in DSL expressions. These classes provide initializers which accept string representations of dates or times. They are renamed as Date and Timestamp when the members of an SQLContext are in scope.

with a literal, e.g. 0 < 'x . These conversions expose a conflict with the scalatest === operator if assert(X === Y) is used when the conversions are in scope. To fix this, several tests are modified, as recommended in the scalatest documentation, by making the change: assert(X === Y) --> assert(convertToEqualizer(X).===(Y))

a test which would have detected the problem.

available after importing the members of an SQLContext.

SparkQA · 2014-11-11T00:21:41Z

Test build #23171 has finished for PR 3158 at commit dabcaf2.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class LhsLiteral(x: Any)
- class RichDate(milliseconds: Long) extends Date(milliseconds)
- class RichTimestamp(milliseconds: Long) extends Timestamp(milliseconds)

AmplabJenkins · 2014-11-11T00:21:43Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/23171/
Test FAILed.

SparkQA · 2014-11-11T05:10:06Z

Test build #23193 has started for PR 3158 at commit f917e06.

This patch merges cleanly.

AmplabJenkins · 2014-11-11T05:17:20Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/23192/
Test FAILed.

SparkQA · 2014-11-11T05:17:34Z

Test build #23193 has finished for PR 3158 at commit f917e06.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class RichDate(milliseconds: Long) extends Date(milliseconds)
- class RichTimestamp(milliseconds: Long) extends Timestamp(milliseconds)

AmplabJenkins · 2014-11-11T05:17:37Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/23193/
Test FAILed.

SparkQA · 2014-11-11T05:30:04Z

Test build #23195 has started for PR 3158 at commit 4c32c04.

This patch merges cleanly.

SparkQA · 2014-11-11T06:45:34Z

Test build #23195 has finished for PR 3158 at commit 4c32c04.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class RichDate(milliseconds: Long) extends Date(milliseconds)
- class RichTimestamp(milliseconds: Long) extends Timestamp(milliseconds)

AmplabJenkins · 2014-11-11T06:45:37Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/23195/
Test PASSed.

culler · 2014-11-11T06:46:36Z

OK @marmbrus and @liancheng, this PR is now fit and slim. The LhsLiterals have been removed and will appear in a new PR. This one passes all tests and I think it is ready for review, finally.

谢谢, @liancheng, for your patience, thoughtful comments and careful reading.

culler · 2014-11-11T20:54:05Z

Thanks!

On Mon, Nov 10, 2014 at 6:17 PM, Michael Armbrust notifications@github.com
wrote:

Quick note: you don't need a fork per PR. You can have multiple branches
in a single fork, each of them with a corresponding PR.

—
Reply to this email directly or view it on GitHub
#3158 (comment).

culler · 2014-11-15T01:34:07Z

Maybe next week will be better. The PR is now much smaller, with the
changes to test files removed and the DSL part moved into PR #3210.

On Fri, Nov 7, 2014 at 3:02 PM, Michael Armbrust notifications@github.com
wrote:

Thanks for working on this. Is a pretty big change so I'll review it next
week after we get some critical fixes in for 1.2.

—
Reply to this email directly or view it on GitHub
#3158 (comment).

SparkQA · 2014-12-09T21:15:24Z

Test build #24259 has started for PR 3158 at commit 4cfb864.

This patch merges cleanly.

SparkQA · 2014-12-09T22:58:22Z

Test build #24259 has finished for PR 3158 at commit 4cfb864.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class RichDate(milliseconds: Long) extends Date(milliseconds)
- class RichTimestamp(milliseconds: Long) extends Timestamp(milliseconds)

AmplabJenkins · 2014-12-09T22:58:26Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24259/
Test PASSed.

SparkQA · 2015-01-06T19:22:47Z

Test build #25108 has started for PR 3158 at commit dd39a89.

This patch merges cleanly.

SparkQA · 2015-01-06T20:12:45Z

Test build #25110 has started for PR 3158 at commit 31f0d7e.

This patch merges cleanly.

SparkQA · 2015-01-06T20:29:09Z

Test build #25108 has finished for PR 3158 at commit dd39a89.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-01-06T20:29:12Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25108/
Test PASSed.

SparkQA · 2015-01-06T21:20:21Z

Test build #25110 has finished for PR 3158 at commit 31f0d7e.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-01-06T21:20:26Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25110/
Test PASSed.

culler added 6 commits November 9, 2014 22:07

Clarification of one comment.

492feef

Fixed implicit conversions for RichDate and RichTimestamp and added

53aced3

a test which would have detected the problem.

Rebased and fixed the assert commands in ParquetQuerySuite.scala.

bd539b6

Add unapply methods to RichDate and RichTimestamp and make their types

353d12c

available after importing the members of an SQLContext.

Removed LhsLiterals for another PR. Cleaned up.

ad2940d

culler changed the title ~~[SPARK-4205][SQL] Timestamp and Date with comparisons / DSL literals~~ [SPARK-4205][SQL] Timestamp and Date with comparisons Nov 11, 2014

Removed test which is no longer part of this PR.

f917e06

Restored lost import of scalatest.Assertions.convertToEqualizer.

4c32c04

culler mentioned this pull request Nov 11, 2014

[SPARK-4205][SQL] DSL expressions with a literal on the left hand side #3210

Closed

Merge branch 'master' of https://github.com/apache/spark

4cfb864

Update timetypes.scala

dd39a89

Update ExpressionEvaluationSuite.scala

31f0d7e

asfgit closed this in ffe8cc9 Apr 3, 2015

[SPARK-4205][SQL] Timestamp and Date with comparisons #3158

[SPARK-4205][SQL] Timestamp and Date with comparisons #3158

Conversation

culler commented Nov 7, 2014

AmplabJenkins commented Nov 7, 2014

culler commented Nov 7, 2014

marmbrus commented Nov 7, 2014

culler commented Nov 7, 2014

marmbrus commented Nov 7, 2014

SparkQA commented Nov 7, 2014

SparkQA commented Nov 7, 2014

AmplabJenkins commented Nov 7, 2014

AmplabJenkins commented Nov 7, 2014

AmplabJenkins commented Nov 7, 2014

culler commented Nov 8, 2014

SparkQA commented Nov 8, 2014

SparkQA commented Nov 8, 2014

AmplabJenkins commented Nov 8, 2014

SparkQA commented Nov 8, 2014

SparkQA commented Nov 8, 2014

SparkQA commented Nov 8, 2014

AmplabJenkins commented Nov 8, 2014

SparkQA commented Nov 8, 2014

AmplabJenkins commented Nov 8, 2014

SparkQA commented Nov 9, 2014

SparkQA commented Nov 9, 2014

AmplabJenkins commented Nov 9, 2014

SparkQA commented Nov 10, 2014

SparkQA commented Nov 11, 2014

AmplabJenkins commented Nov 11, 2014

SparkQA commented Nov 11, 2014

AmplabJenkins commented Nov 11, 2014

SparkQA commented Nov 11, 2014

AmplabJenkins commented Nov 11, 2014

SparkQA commented Nov 11, 2014

SparkQA commented Nov 11, 2014

AmplabJenkins commented Nov 11, 2014

culler commented Nov 11, 2014

culler commented Nov 11, 2014

culler commented Nov 15, 2014

SparkQA commented Dec 9, 2014

SparkQA commented Dec 9, 2014

AmplabJenkins commented Dec 9, 2014

SparkQA commented Jan 6, 2015

SparkQA commented Jan 6, 2015

SparkQA commented Jan 6, 2015

AmplabJenkins commented Jan 6, 2015

SparkQA commented Jan 6, 2015

AmplabJenkins commented Jan 6, 2015