[SPARK-31762][SQL] Fix perf regression of date/timestamp formatting in toHiveString #28582

MaxGekk · 2020-05-19T19:56:13Z

What changes were proposed in this pull request?

Add new methods that accept date-time Java types to the DateFormatter and TimestampFormatter traits. The methods format input date-time instances to strings:
- TimestampFormatter:
  - def format(ts: Timestamp): String
  - def format(instant: Instant): String
- DateFormatter:
  - def format(date: Date): String
  - def format(localDate: LocalDate): String
Re-use the added methods from HiveResult.toHiveString
Borrow the code for formatting of java.sql.Timestamp from Spark 2.4 DateTimeUtils.timestampToString to FractionTimestampFormatter because legacy formatters don't support variable length patterns for seconds fractions.

Why are the changes needed?

To avoid unnecessary overhead of converting Java date-time types to micros/days before formatting. Also formatters have to convert input micros/days back to Java types to pass instances to standard library API.

Does this PR introduce any user-facing change?

No

How was this patch tested?

By existing tests for toHiveString and new tests in TimestampFormatterSuite.

SparkQA · 2020-05-19T21:58:36Z

Test build #122854 has finished for PR 28582 at commit d8b0ed4.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

MaxGekk · 2020-05-20T10:00:06Z

@cloud-fan @HyukjinKwon @bogdanghit Please, review this PR.

SparkQA · 2020-05-20T12:02:49Z

Test build #122887 has finished for PR 28582 at commit 3914313.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

cloud-fan · 2020-05-20T12:10:22Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/TimestampFormatter.scala

@@ -100,10 +111,26 @@ class Iso8601TimestampFormatter(
 */
 class FractionTimestampFormatter(zoneId: ZoneId)
  extends Iso8601TimestampFormatter(
-    "", zoneId, TimestampFormatter.defaultLocale, needVarLengthSecondFraction = false) {
+    TimestampFormatter.defaultPattern,


what's wrong if we still pass the empty string as the pattern?

empty result

ok, it's used by the legacy formatter.

cloud-fan · 2020-05-20T12:11:29Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/TimestampFormatter.scala


  @transient
  override protected lazy val formatter = DateTimeFormatterHelper.fractionFormatter
+
+  // Converts Timestamp to string according to Hive TimestampWritable convention.
+  // The code is borrowed from Spark 2.4 DateTimeUtils.timestampToString


is it really needed? DateTimeFormatterHelper.fractionFormatter should omit tailing 0 already.

should omit tailing 0 already.

The reason of making this PR is to pass java.sql.Timestamp to a legacy formatter which can accept the type but the legacy formatter of fractionFormatter is SimpleDateFormat which cannot omit tailing 0.

is it really needed?

I think so. What do you propose instead of it?

Let's make the comment more clear:

// The new formatter will omit the trailing 0 in the timestamp string, but the legacy formatter can't. // Here we borrow the code from Spark 2.4 DateTimeUtils.timestampToString to omit the // trailing 0 for the legacy formatter as well.

We don't need to mention hive at all. This is just for internal consistency.

Thanks. I updated the comment.

SparkQA · 2020-05-20T13:45:04Z

Test build #122883 has finished for PR 28582 at commit 6d8c2ae.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

MaxGekk · 2020-05-20T14:10:53Z

sql/core/src/test/resources/sql-tests/results/postgreSQL/date.sql.out

@@ -584,7 +584,7 @@ select make_date(-44, 3, 15)
 -- !query schema
 struct<make_date(-44, 3, 15):date>
 -- !query output
-0044-03-15
+0045-03-15


This is compatible with Spark 2.4 because it uses SimpleDateFormat which cannot format negative years.

SparkQA · 2020-05-20T14:55:22Z

Test build #122893 has finished for PR 28582 at commit 29d8e51.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-05-20T19:52:12Z

Test build #122902 has finished for PR 28582 at commit 31e8ffd.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

cloud-fan · 2020-05-21T04:01:13Z

thanks, merging to master/3.0!

…n toHiveString ### What changes were proposed in this pull request? 1. Add new methods that accept date-time Java types to the DateFormatter and TimestampFormatter traits. The methods format input date-time instances to strings: - TimestampFormatter: - `def format(ts: Timestamp): String` - `def format(instant: Instant): String` - DateFormatter: - `def format(date: Date): String` - `def format(localDate: LocalDate): String` 2. Re-use the added methods from `HiveResult.toHiveString` 3. Borrow the code for formatting of `java.sql.Timestamp` from Spark 2.4 `DateTimeUtils.timestampToString` to `FractionTimestampFormatter` because legacy formatters don't support variable length patterns for seconds fractions. ### Why are the changes needed? To avoid unnecessary overhead of converting Java date-time types to micros/days before formatting. Also formatters have to convert input micros/days back to Java types to pass instances to standard library API. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? By existing tests for toHiveString and new tests in `TimestampFormatterSuite`. Closes #28582 from MaxGekk/opt-format-old-types. Authored-by: Max Gekk <max.gekk@gmail.com> Signed-off-by: Wenchen Fan <wenchen@databricks.com> (cherry picked from commit 5d67331) Signed-off-by: Wenchen Fan <wenchen@databricks.com>

…ectly ### What changes were proposed in this pull request? Use `format()` methods for Java date-time types in `Row.jsonValue`. The PR #28582 added the methods to avoid conversions to days and microseconds. ### Why are the changes needed? To avoid unnecessary overhead of converting Java date-time types to micros/days before formatting. Also formatters have to convert input micros/days back to Java types to pass instances to standard library API. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? By existing tests in `RowJsonSuite`. Closes #28620 from MaxGekk/toJson-format-Java-datetime-types. Authored-by: Max Gekk <max.gekk@gmail.com> Signed-off-by: HyukjinKwon <gurwls223@apache.org>

…ectly ### What changes were proposed in this pull request? Use `format()` methods for Java date-time types in `Row.jsonValue`. The PR #28582 added the methods to avoid conversions to days and microseconds. ### Why are the changes needed? To avoid unnecessary overhead of converting Java date-time types to micros/days before formatting. Also formatters have to convert input micros/days back to Java types to pass instances to standard library API. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? By existing tests in `RowJsonSuite`. Closes #28620 from MaxGekk/toJson-format-Java-datetime-types. Authored-by: Max Gekk <max.gekk@gmail.com> Signed-off-by: HyukjinKwon <gurwls223@apache.org> (cherry picked from commit 7f36310) Signed-off-by: HyukjinKwon <gurwls223@apache.org>

MaxGekk added 4 commits May 19, 2020 22:28

Support formatting java.sql.Timestamp and java.time.Instant directly

2545366

Re-use new timestamp formatting functions from toHiveSting

b325c21

Support formatting java.util.Date and java.time.LocalDate directly

af816e6

Re-use new date formatting functions from toHiveSting

d8b0ed4

probot-autolabeler bot added the SQL label May 19, 2020

MaxGekk added 4 commits May 20, 2020 11:52

Fix FractionTimestampFormatter for java.sql.Timestamp

6d8c2ae

Bug fix

1b2bc2f

Add tests

bfae42e

Test Fraction Formatter

3914313

MaxGekk changed the title ~~[WIP][SPARK-31762][SQL] Fix perf regression of date/timestamp formatting in toHiveString~~ [SPARK-31762][SQL] Fix perf regression of date/timestamp formatting in toHiveString May 20, 2020

MaxGekk added 2 commits May 20, 2020 14:08

Add tests to DateFormatterSuite

864efdd

More tests to TimestampFormatterSuite

29d8e51

cloud-fan reviewed May 20, 2020

View reviewed changes

MaxGekk added 2 commits May 20, 2020 17:06

Re-gen postgreSQL/date.sql.out

5222a0a

Update comment

31e8ffd

MaxGekk commented May 20, 2020

View reviewed changes

cloud-fan approved these changes May 20, 2020

View reviewed changes

cloud-fan closed this in 5d67331 May 21, 2020

MaxGekk mentioned this pull request May 23, 2020

[SPARK-31802][SQL] Format Java date-time types in Row.jsonValue directly #28620

Closed

MaxGekk mentioned this pull request Jun 4, 2020

[SPARK-28624][SQL][TESTS][3.0] Run date.sql via Thrift Server #28723

Closed

MaxGekk deleted the opt-format-old-types branch June 5, 2020 19:49

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-31762][SQL] Fix perf regression of date/timestamp formatting in toHiveString #28582

[SPARK-31762][SQL] Fix perf regression of date/timestamp formatting in toHiveString #28582

MaxGekk commented May 19, 2020 •

edited

Loading

SparkQA commented May 19, 2020

MaxGekk commented May 20, 2020

SparkQA commented May 20, 2020

cloud-fan May 20, 2020

MaxGekk May 20, 2020

cloud-fan May 20, 2020

cloud-fan May 20, 2020

MaxGekk May 20, 2020

cloud-fan May 20, 2020

MaxGekk May 20, 2020

SparkQA commented May 20, 2020

MaxGekk May 20, 2020

SparkQA commented May 20, 2020

SparkQA commented May 20, 2020

cloud-fan commented May 21, 2020

[SPARK-31762][SQL] Fix perf regression of date/timestamp formatting in toHiveString #28582

[SPARK-31762][SQL] Fix perf regression of date/timestamp formatting in toHiveString #28582

Conversation

MaxGekk commented May 19, 2020 • edited Loading

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

SparkQA commented May 19, 2020

MaxGekk commented May 20, 2020

SparkQA commented May 20, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SparkQA commented May 20, 2020

Choose a reason for hiding this comment

SparkQA commented May 20, 2020

SparkQA commented May 20, 2020

cloud-fan commented May 21, 2020

MaxGekk commented May 19, 2020 •

edited

Loading