[SPARK-19228][SQL] Migrate on Java 8 time from FastDateFormat for meet the ISO8601 #21363

sergey-rubtsov · 2018-05-18T12:19:36Z

What changes were proposed in this pull request?

Add support for inferring DateType and custom "dateFormat" option.
Fix an old bug with parse string to SQL's timestamp value in microseconds accuracy.
Add a type-widening rule in findTightestCommonType between DateType and TimestampType.

How was this patch tested?

Fix some tests to accord with an internationally agreed way to represent dates.
Add an end-to-end test case and unit tests.

…t the ISO8601 and parsing dates in csv correctly. Add support for inferring DateType and custom "dateFormat" option.

sergey-rubtsov · 2018-05-18T12:21:30Z

Previous pull request #20140 is closed

sergey-rubtsov · 2018-05-18T12:22:31Z

@HyukjinKwon, @gatorsmile please take a look

sergey-rubtsov · 2018-05-20T16:55:47Z

cc @viirya @mgaido91 @dongjoon-hyun

HyukjinKwon · 2018-05-21T01:44:25Z

ok to test

SparkQA · 2018-05-21T01:49:55Z

Test build #90870 has finished for PR 21363 at commit 65179a2.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

HyukjinKwon · 2018-05-21T02:27:09Z

@sergey-rubtsov, would we be able to add a configuration to control this behaviour? Sounds we should better have a configuration to control this behaviour for now since the date / timestamp parsing logic is affected by the library change.

HyukjinKwon · 2018-05-21T02:28:00Z

...ore/src/test/scala/org/apache/spark/sql/execution/datasources/csv/UnivocityParserSuite.scala

    val castedDate =
-      parser.makeConverter("_1", DateType, nullable = true, options = dateOptions)


I would keep this line as was.

HyukjinKwon · 2018-05-21T02:29:12Z

sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/csv/CSVOptions.scala

@@ -119,7 +119,6 @@ class CSVOptions(
  val positiveInf = parameters.getOrElse("positiveInf", "Inf")
  val negativeInf = parameters.getOrElse("negativeInf", "-Inf")

-


Sounds unrelated change.

viirya · 2018-05-21T03:14:41Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/DateTimeUtils.scala

+  def dateTimeToMicroseconds(localDateTime: LocalDateTime, timeZone: TimeZone): Long = {
+    val microOfSecond = localDateTime.getLong(ChronoField.MICRO_OF_SECOND)
+    val epochSecond = localDateTime.atZone(timeZone.toZoneId).toInstant.getEpochSecond
+    epochSecond * 1000000L + microOfSecond


1000000L -> MICROS_PER_SECOND?

viirya · 2018-05-21T03:51:35Z

sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/csv/CSVInferSchema.scala

@@ -140,14 +141,23 @@ private[csv] object CSVInferSchema {
  private def tryParseDouble(field: String, options: CSVOptions): DataType = {
    if ((allCatch opt field.toDouble).isDefined || isInfOrNan(field, options)) {
      DoubleType
+    } else {
+      tryParseDate(field, options)


Is this a behavior change? Previously timestamp type, now date type/timestamp type?

For example, by mistake we have identical "timestampFormat" and "dateFormat" options.
Let it be "yyyy-MM-dd"
'TimestampType' (8 bytes) is larger than 'DateType' (4 bytes)
So if they can overlap, we need to try parse it as date firstly, because both of these types are suitable, but you need to try to use a more compact by default and it will be correct inferring of type

At the moment, DateType here is ignored at all, I'm not sure that it was conceived when the type was created

viirya · 2018-05-21T04:01:58Z

...core/src/test/scala/org/apache/spark/sql/execution/datasources/csv/CSVInferSchemaSuite.scala

-  test("Timestamp field types are inferred correctly via custom data format") {
-    var options = new CSVOptions(Map("timestampFormat" -> "yyyy-mm"), "GMT")
+  test("Timestamp field types are inferred correctly via custom date format") {
+    var options = new CSVOptions(Map("timestampFormat" -> "yyyy-MM"), "GMT")


Why we need to change this? The format string change is also a behavior change for users. We might need a config for that.

"yyyy-mm" means years and minutes, this is a date format, not a time format
"yyyy-MM" means years and months, but I do not insist on this change

mgaido91 · 2018-05-21T08:37:59Z

sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/csv/CSVInferSchema.scala

@@ -90,6 +90,7 @@ private[csv] object CSVInferSchema {
          // DecimalTypes have different precisions and scales, so we try to find the common type.
          findTightestCommonType(typeSoFar, tryParseDecimal(field, options)).getOrElse(StringType)
        case DoubleType => tryParseDouble(field, options)
+        case DateType => tryParseDate(field, options)


this also is a behavior change. Shall we document it?

I can do it, but where exactly it should be documented?

sergey-rubtsov · 2018-05-21T13:42:36Z

@HyukjinKwon please, clarify, how to add a configuration to control this behaviour? Do you mean to keep backward compatibility?
But it seems, now, we don't handle a Spark "DataType" at all.
I can change the behaviour in next way:
Hanlde "DataType" if option "dateFormat" was set by customer, if it is not set, ignore it at all, is it okay?

mgaido91 · 2018-05-21T15:11:39Z

@sergey-rubtsov we have to keep backward compatibility. If a user upgrades, with your change a running application may break because of data not being anymore timestamp but date. We can add a new entry in SQLConf for that.

Moreover, we have to mention in the migration guide all the behavioral changes we introduce.

HyukjinKwon · 2018-06-09T08:51:48Z

ok to test

SparkQA · 2018-06-09T08:55:02Z

Test build #91599 has finished for PR 21363 at commit 65179a2.

This patch fails Scala style tests.
This patch does not merge cleanly.
This patch adds no public classes.

AmplabJenkins · 2018-10-22T12:15:56Z

Can one of the admins verify this patch?

srowen

I agree we can probably use Java 8 Date classes now instead of 3rd party ones, but can we do so without a behavior change? is it actually required?

HyukjinKwon · 2018-11-09T00:17:21Z

Looks difficult because the behaviours themselves are different. One possibility is a fallback and the other possibility is configuration.

HyukjinKwon · 2018-11-11T03:16:57Z

I am going to suggest to close this since it's being active more then few weeks. It should be good to fix. Let me leave some cc's who might be interested in this just FYI. Feel free to take over this when you guys find some time or are interested in this.

Adding @xuanyuanking, @mgaido91, @viirya, @MaxGekk, @softmanu who I could think of for now. Feel free to ignore my cc if you guys are busy or having more important fixes you guys are working on.

xuanyuanking · 2018-11-11T13:24:31Z

@HyukjinKwon Great thanks for ping me, I'll try to work on this and cc all reviewer in this PR.

MaxGekk · 2018-11-18T15:47:06Z

@xuanyuanking Are you still working on this? If not, I could continue.

xuanyuanking · 2018-11-19T02:28:12Z

@MaxGekk Sorry for the late, something inserted in the my scheduler, I plan to start this PR in this weekend, if its too late please just take it, sorry for the late again.

MaxGekk · 2018-11-26T23:04:44Z

Here is the PR: #23150 which allows to switch on java.time API (in CSV so far).

srowen · 2018-12-04T14:47:09Z

@MaxGekk now that your change is merge, can this proceed, @xuanyuanking ? or is it obsolete?

MaxGekk · 2018-12-04T14:56:42Z

@srowen There is another PR with related changes - inferring DateType from CSV: #23202

HyukjinKwon · 2018-12-18T11:19:08Z

I'm going to close this since @sergey-rubtsov is inactive.

Most of feasible changes extracted from here are merged. One last part is date type inference which's going to need some more discussions (IMHO).

[SPARK-19228][SQL] Migrate on Java 8 time from FastDateFormat for mee…

65179a2

…t the ISO8601 and parsing dates in csv correctly. Add support for inferring DateType and custom "dateFormat" option.

HyukjinKwon reviewed May 21, 2018

View reviewed changes

viirya reviewed May 21, 2018

View reviewed changes

mgaido91 reviewed May 21, 2018

View reviewed changes

HyukjinKwon mentioned this pull request Sep 25, 2018

[SPARK-25517][SQL] Detect/Infer date type in CSV file #22539

Closed

srowen reviewed Nov 8, 2018

View reviewed changes

HyukjinKwon mentioned this pull request Nov 11, 2018

[SPARK-25977][SQL] Parsing decimals from CSV using locale #22979

Closed

HyukjinKwon closed this Dec 18, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-19228][SQL] Migrate on Java 8 time from FastDateFormat for meet the ISO8601 #21363

[SPARK-19228][SQL] Migrate on Java 8 time from FastDateFormat for meet the ISO8601 #21363

sergey-rubtsov commented May 18, 2018

sergey-rubtsov commented May 18, 2018

sergey-rubtsov commented May 18, 2018 •

edited

Loading

sergey-rubtsov commented May 20, 2018

HyukjinKwon commented May 21, 2018

SparkQA commented May 21, 2018

HyukjinKwon commented May 21, 2018 •

edited

Loading

HyukjinKwon May 21, 2018

sergey-rubtsov May 21, 2018

HyukjinKwon May 21, 2018

viirya May 21, 2018

sergey-rubtsov May 21, 2018

viirya May 21, 2018

sergey-rubtsov May 21, 2018

sergey-rubtsov May 21, 2018

viirya May 21, 2018 •

edited

Loading

sergey-rubtsov May 21, 2018 •

edited

Loading

mgaido91 May 21, 2018

sergey-rubtsov May 21, 2018

sergey-rubtsov commented May 21, 2018

mgaido91 commented May 21, 2018

HyukjinKwon commented Jun 9, 2018

SparkQA commented Jun 9, 2018

AmplabJenkins commented Oct 22, 2018

srowen left a comment

HyukjinKwon commented Nov 9, 2018

HyukjinKwon commented Nov 11, 2018

xuanyuanking commented Nov 11, 2018

MaxGekk commented Nov 18, 2018

xuanyuanking commented Nov 19, 2018

MaxGekk commented Nov 26, 2018

srowen commented Dec 4, 2018

MaxGekk commented Dec 4, 2018

HyukjinKwon commented Dec 18, 2018

		val castedDate =
		parser.makeConverter("_1", DateType, nullable = true, options = dateOptions)

		@@ -119,7 +119,6 @@ class CSVOptions(
		val positiveInf = parameters.getOrElse("positiveInf", "Inf")
		val negativeInf = parameters.getOrElse("negativeInf", "-Inf")

[SPARK-19228][SQL] Migrate on Java 8 time from FastDateFormat for meet the ISO8601 #21363

[SPARK-19228][SQL] Migrate on Java 8 time from FastDateFormat for meet the ISO8601 #21363

Conversation

sergey-rubtsov commented May 18, 2018

What changes were proposed in this pull request?

How was this patch tested?

sergey-rubtsov commented May 18, 2018

sergey-rubtsov commented May 18, 2018 • edited Loading

sergey-rubtsov commented May 20, 2018

HyukjinKwon commented May 21, 2018

SparkQA commented May 21, 2018

HyukjinKwon commented May 21, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

viirya May 21, 2018 • edited Loading

Choose a reason for hiding this comment

sergey-rubtsov May 21, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sergey-rubtsov commented May 21, 2018

mgaido91 commented May 21, 2018

HyukjinKwon commented Jun 9, 2018

SparkQA commented Jun 9, 2018

AmplabJenkins commented Oct 22, 2018

srowen left a comment

Choose a reason for hiding this comment

HyukjinKwon commented Nov 9, 2018

HyukjinKwon commented Nov 11, 2018

xuanyuanking commented Nov 11, 2018

MaxGekk commented Nov 18, 2018

xuanyuanking commented Nov 19, 2018

MaxGekk commented Nov 26, 2018

srowen commented Dec 4, 2018

MaxGekk commented Dec 4, 2018

HyukjinKwon commented Dec 18, 2018

sergey-rubtsov commented May 18, 2018 •

edited

Loading

HyukjinKwon commented May 21, 2018 •

edited

Loading

viirya May 21, 2018 •

edited

Loading

sergey-rubtsov May 21, 2018 •

edited

Loading