[SPARK-29486][SQL] CalendarInterval should have 3 fields: months, days and microseconds #26134

LinhongLiu · 2019-10-16T06:02:47Z

What changes were proposed in this pull request?

Current CalendarInterval has 2 fields: months and microseconds. This PR try to change it
to 3 fields: months, days and microseconds. This is because one logical day interval may
have different number of microseconds (daylight saving).

Why are the changes needed?

One logical day interval may have different number of microseconds (daylight saving).
For example, in PST timezone, there will be 25 hours from 2019-11-2 12:00:00 to
2019-11-3 12:00:00

Does this PR introduce any user-facing change?

no

How was this patch tested?

unit test and new added test cases

LinhongLiu · 2019-10-16T06:08:07Z

sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/Triggers.scala

-      throw new IllegalArgumentException(s"Doesn't support month or year interval: $interval")
+    if (cal.months > 0 || cal.days > 0) {
+      throw new IllegalArgumentException(
+        s"Doesn't support day, week, month or year interval: $interval")


This will fail a current valid usage: Trigger.ProcessingTime("10 day")
Maybe we should still keep converting day interval and use a 24 hours = 1 day assumption

LinhongLiu · 2019-10-16T06:08:56Z

sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/util/DateTimeUtilsSuite.scala

+    assert(timestampAddInterval(ts3, 36, 0, 123000, TimeZoneGMT.toZoneId) === ts5)
+  }
+
+  test("timestamp add days") {


test case to demonstrate the daylight saving case

LinhongLiu · 2019-10-16T06:10:24Z

cc @xuanyuanking @cloud-fan

cloud-fan · 2019-10-16T06:21:03Z

Let's create a ticket in https://issues.apache.org/jira/secure/Dashboard.jspa and put it in the title. You can follow other PRs: https://github.com/apache/spark/pulls

LinhongLiu · 2019-10-16T06:27:32Z

Let's create a ticket in https://issues.apache.org/jira/secure/Dashboard.jspa and put it in the title. You can follow other PRs: https://github.com/apache/spark/pulls

sorry, my mistake. ticket id is attached now.

cloud-fan · 2019-10-16T06:56:53Z

ok to test

SparkQA · 2019-10-16T07:05:02Z

Test build #112141 has finished for PR 26134 at commit 3d954d6.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

xuanyuanking · 2019-10-16T08:26:41Z

retest this please

MaxGekk · 2019-10-16T08:06:30Z

common/unsafe/src/main/java/org/apache/spark/unsafe/types/CalendarInterval.java

    microseconds += toLong(m.group(6)) * MICROS_PER_MINUTE;
    microseconds += toLong(m.group(7)) * MICROS_PER_SECOND;
    microseconds += toLong(m.group(8)) * MICROS_PER_MILLI;
    microseconds += toLong(m.group(9));
-    return new CalendarInterval((int) months, microseconds);
+    return new CalendarInterval((int) months, (int) days, microseconds);


I think it is better to catch int overflow by Math.toIntExact instead of silently ignore it.

or maybe we can change this to long

Valid range of dates/timestamps is [0..10000). So, we should support days up to 10000 * 366 = 3660000. Even int is so much but it is ok.

MaxGekk · 2019-10-16T08:07:08Z

common/unsafe/src/main/java/org/apache/spark/unsafe/types/CalendarInterval.java

@@ -195,7 +194,7 @@ public static CalendarInterval fromDayTimeString(String s, String from, String t
    } else {
      try {
        int sign = m.group(1) != null && m.group(1).equals("-") ? -1 : 1;
-        long days = m.group(2) == null ? 0 : toLongWithRange("day", m.group(3),
+        int days = m.group(2) == null ? 0 : (int) toLongWithRange("day", m.group(3),


Use Math.toIntExact and in other similar places

can we use Math.toIntExact for only days field? as you know, there are long -> int in months filed as well, should we cover both of them?

This is OK here, we call toLongWithRange with Integer.MAX_VALUE so it's safe to cast to int.

MaxGekk · 2019-10-16T08:14:10Z

common/unsafe/src/main/java/org/apache/spark/unsafe/types/CalendarInterval.java

@@ -332,31 +331,35 @@ public static long parseSecondNano(String secondNano) throws IllegalArgumentExce
  }

  public final int months;
+  public final int days;
  public final long microseconds;

  public long milliseconds() {


The method will return different values because you exclude days from microseconds

MaxGekk · 2019-10-16T08:32:26Z

sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/codegen/UnsafeWriter.java

@@ -132,16 +132,17 @@ private void writeUnalignedBytes(

  public final void write(int ordinal, CalendarInterval input) {
    // grow the global buffer before writing data.
-    grow(16);
+    grow(24);

    // Write the months and microseconds fields of Interval to the variable length portion.


and the days field

MaxGekk · 2019-10-16T08:34:10Z

sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/codegen/UnsafeWriter.java


    // Write the months and microseconds fields of Interval to the variable length portion.
    Platform.putLong(getBuffer(), cursor(), input.months);
-    Platform.putLong(getBuffer(), cursor() + 8, input.microseconds);
+    Platform.putLong(getBuffer(), cursor() + 8, input.days);


Just in case, why we put long if days is int?

MaxGekk · 2019-10-16T08:36:58Z

...catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala


    private val backedSequenceImpl = new IntegralSequenceImpl[T](dt)
-    private val microsPerMonth = 28 * CalendarInterval.MICROS_PER_DAY
+    private val microsPerDay = 24 * CalendarInterval.MICROS_PER_HOUR
+    private val microsPerMonth = 28 * microsPerDay


Not related to the PR but it is interesting why 28 days per months here.

the code will use microsPerMonth to estimate the array length first. use the minimum days in one month can make sure the array long enough. (smaller days means smaller steps in micro and means longer array length)

not related to this PR, but would be great if we can add a comment to explain it in the code.

MaxGekk · 2019-10-16T08:51:06Z

sql/core/src/test/resources/sql-tests/results/ansi/interval.sql.out

@@ -238,7 +238,7 @@ from interval_arithmetic
 -- !query 17 schema
 struct<tsval:timestamp,CAST(tsval - interval 14 weeks 1 days 11 hours 22 minutes 33 seconds 123 milliseconds 456 microseconds AS TIMESTAMP):timestamp,CAST(tsval - interval -14 weeks -1 days -11 hours -22 minutes -33 seconds -123 milliseconds -456 microseconds AS TIMESTAMP):timestamp,CAST(tsval + interval 14 weeks 1 days 11 hours 22 minutes 33 seconds 123 milliseconds 456 microseconds AS TIMESTAMP):timestamp,CAST(tsval + interval -14 weeks -1 days -11 hours -22 minutes -33 seconds -123 milliseconds -456 microseconds AS TIMESTAMP):timestamp,CAST(tsval + (- interval 14 weeks 1 days 11 hours 22 minutes 33 seconds 123 milliseconds 456 microseconds) AS TIMESTAMP):timestamp,CAST(tsval + interval 14 weeks 1 days 11 hours 22 minutes 33 seconds 123 milliseconds 456 microseconds AS TIMESTAMP):timestamp>
 -- !query 17 output
-2012-01-01 00:00:00	2011-09-23 13:37:26.876544	2012-04-09 12:22:33.123456	2012-04-09 12:22:33.123456	2011-09-23 13:37:26.876544	2011-09-23 13:37:26.876544	2012-04-09 12:22:33.123456
+2012-01-01 00:00:00	2011-09-23 12:37:26.876544	2012-04-09 11:22:33.123456	2012-04-09 11:22:33.123456	2011-09-23 12:37:26.876544	2011-09-23 12:37:26.876544	2012-04-09 11:22:33.123456


Why it there difference of 1 hour?

as you know it's caused by daylight saving

MaxGekk · 2019-10-16T08:55:16Z

common/unsafe/src/main/java/org/apache/spark/unsafe/types/CalendarInterval.java

  }

  @Override
  public int hashCode() {
-    return 31 * months + (int) microseconds;
+    return 31 * (31 * months + days) + (int) microseconds;


Not related to the PR but I am wondering why the most significant bits of microseconds are completely excluded from the hash.

MaxGekk · 2019-10-16T09:22:57Z

Why are the changes needed?
One logical day interval may have different number of microseconds (daylight saving).
For example, in PST timezone, there will be 25 hours from 2019-11-2 12:00:00 to
2019-11-3 12:00:00

Yes, actual duration of the daylight saving day can be 25 hours but logically any day has 24 hours. Spark INTERVAL type defines not duration, it is adjustment of local date-time. After your changes, it adjusts logical (local) months, days and for some reasons physical time duration of a day. It looks like inconsistent, doesn't it?

As you point out, the daylight saving in PST will happen at 2019-11-03 02:00:00. So, the next timestamp with granularity of 1 microsecond is 2019-11-03 01:00:01. Even more, the local timestamp 2019-11-03 02:00:00 will be showed on local clock twice.
At the first time:

timestamp'2019-11-03 02:00:00' + interval 1 microsecond = timestamp'2019-11-03 01:00:00.000001'

At the second time:

timestamp'2019-11-03 02:00:00' + interval 1 microsecond = timestamp'2019-11-03 02:00:00.000001'

Could you elaborate how your changes resolve this situation?

cloud-fan · 2019-10-16T09:42:26Z

one day is always 24 hours, but one day interval may not. Daylight saving doesn't change the # of hours in a day, but the timezone offset of a region. For example, California is in UTC-7 right now, but will be in UTC-8 after 3 Nov 2019. So adding one day to 3 Nov 2019 at local time maybe different from adding 24 hours.

However, I'd like to know if it's allowed to set timezone to a region name that has undetermined timezone offset. If it's not allowed then one day interval is always 24 hours in SQL.

MaxGekk · 2019-10-16T10:36:32Z

... but one day interval may not.

As I can understand, new microseconds field means actual time duration inside of a day, correct? Back to the example, it means:

timestamp'2019-11-03 01:59:59 PST' + interval 2 seconds = timestamp'2019-11-03 01:00:01 PST'

and to get timestamp'2019-11-03 02:00:01 PST', we should add another interval:

timestamp'2019-11-03 01:59:59 PST' + interval 1 hour 2 seconds = timestamp'2019-11-03 02:00:01 PST'

However, I'd like to know if it's allowed to set timezone to a region name that has undetermined timezone offset.

Allowed by whom (SQL standard?)? For example, Java 8 time API allows to get 0, 1 or 2 zone offsets depend on local timestamp, see https://docs.oracle.com/javase/8/docs/api/java/time/zone/ZoneRules.html#getValidOffsets-java.time.LocalDateTime- , see:

The mapping from a local date-time to an offset is not straightforward. There are three cases:
 -   Normal, with one valid offset. For the vast majority of the year, the normal case applies, where there is a single valid offset for the local date-time.
 -   Gap, with zero valid offsets. This is when clocks jump forward typically due to the spring daylight savings change from "winter" to "summer". In a gap there are local date-time values with no valid offset.
 -   Overlap, with two valid offsets. This is when clocks are set back typically due to the autumn daylight savings change from "summer" to "winter". In an overlap there are local date-time values with two valid offsets.

If it's not allowed then one day interval is always 24 hours in SQL.

The SQL standard defines time zones as TIMEZONE_HOUR:TIMEZONE_MINUTE. Due to this, there is no issue of converting zone id (region name) to offsets.

SparkQA · 2019-10-16T10:49:50Z

Test build #112154 has finished for PR 26134 at commit 3d954d6.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

MaxGekk · 2019-10-16T11:21:32Z

Since SQL has always determined zone offsets, a day has 24 hours always, and interval hours are in 0..23:

Current implementation of CalendarIntervalType is correct union of SQL year-month INTERVAL and day-time INTERVAL. Also there is no issue of adding intervals to timestamps on Daylight saving day if we assume that the time part of intervals is just "logical" adjustment.

Taking into account that local hours of timestamps are always in the range of [0, 23], @LinhongLiu could you show concrete example when old (current) implementation returns wrong results, and your changes fix that. And it would be nice to put such example in the description of this PR.

LinhongLiu · 2019-10-16T11:40:19Z

@MaxGekk @cloud-fan thanks for your review, here is my answer for the questions:

local timestamp 2019-11-03 02:00:00 will be showed on local clock twice

my solution won't fix this, but the original code won't, too. actually I don't think this matters.

could you show concrete example when old (current) implementation returns wrong results, and your changes fix that

the first case is adding 1 day on 2019-11-02 12:00:00 PST in current code will result in 2019-11-03 11:00:00 PST. But as you said, this won't produce a wrong results.

the second case is in SS, if we do a window agg like this:

df.groupBy(window($"time", "1 day")).sum

Since this function is considering timezone, in current implementation, we may have window like this:
2019-11-01 00:00:00 - 2019-11-01 23:59:59
2019-11-02 00:00:00 - 2019-11-02 23:59:59
2019-11-03 00:00:00 - 2019-11-03 22:59:59
2019-11-03 23:00:00 - 2019-11-04 22:59:59
I think the user don't want to see aggregation in this kind of window.

xuanyuanking · 2019-10-16T13:28:05Z

common/unsafe/src/main/java/org/apache/spark/unsafe/types/CalendarInterval.java

@@ -365,12 +368,12 @@ public boolean equals(Object other) {
    if (other == null || !(other instanceof CalendarInterval)) return false;

    CalendarInterval o = (CalendarInterval) other;
-    return this.months == o.months && this.microseconds == o.microseconds;
+    return this.months == o.months && this.days == o.days && this.microseconds == o.microseconds;


Is this line changed manually? We'd better use IDE to generate the equals and hashCode.

cloud-fan · 2019-10-16T14:48:14Z

I spend more time thinking about it. There are 3 kinds of timestamps in the SQL world:

TIMESTAMP WITHOUT TIMEZONE: It's timezone-agnostic, and is simply a combination of the datetime fields YEAR, MONTH, DAY, HOUR, MINUTE and SECOND. So adding one day is the same as adding 24 hours.
TIMESTAMP WITH TIMEZONE: It's similar to the above one but has a timezone associated with each value. Since SQL spec requires a timezone offset (TIMEZONE_HOUR:TIMEZONE_MINUTE), there is no ambiguity and adding 1 day is the same as adding 24 hours.
TIMESTAMP WITH LOCAL TIMEZONE (the Spark timestamp): It's similar to the first one but has a local timezone associated with all the values. Since SQL spec requires a timezone offset, the local timezone should also be a specific offset, and has no ambiguity as well.

However, things are different in the Java world. The java time API use ZoneId, which can be a region name with undeterminate zone offset (see the doc).

@MaxGekk do you think Spark violates the SQL standard if ZoneId is used to represent the session local timezone? I see many datetime operations in DateTimeUtils are using ZoneId.

cloud-fan · 2019-10-28T16:31:56Z

...catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala

-        // about a month length in microseconds
-        val intervalStepInMicros = stepMicros + stepMonths * microsPerMonth
+        // about a month length in days and a day length in microseconds
+        // We choose a minimum days(28) in one month to make sure the estimated


can we move this comment to https://github.com/apache/spark/pull/26134/files#diff-9853dcf5ce3d2ac1e94d473197ff5768R2617 ?

cloud-fan · 2019-10-28T16:40:08Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/IntervalUtils.scala

@@ -87,6 +87,7 @@ object IntervalUtils {
  // Returns total number of seconds with microseconds fractional part in the given interval.
  def getEpoch(interval: CalendarInterval): Decimal = {
    var result = interval.microseconds
+    result += DateTimeUtils.MICROS_PER_DAY * interval.days
    result += MICROS_PER_YEAR * (interval.months / MONTHS_PER_YEAR)
    result += MICROS_PER_MONTH * (interval.months % MONTHS_PER_YEAR)


not related to this PR, but isn't it simply result += MICROS_PER_MONTH * interval.months?

It could be if MICROS_PER_YEAR = 12 * MICROS_PER_MONTH . I strictly followed PostgreSQL implementation: https://github.com/postgres/postgres/blob/bffe1bd68457e43925c362d8728ce3b25bdf1c94/src/backend/utils/adt/timestamp.c#L5016-L5022

SparkQA · 2019-10-29T05:19:21Z

Test build #112813 has finished for PR 26134 at commit 336f0db.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2019-10-29T05:50:32Z

Test build #112814 has finished for PR 26134 at commit 4e97bc0.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeExpressions.scala

SparkQA · 2019-10-31T11:29:50Z

Test build #113013 has finished for PR 26134 at commit 0e87e2d.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
trait HasRelativeError extends Params
class _ImputerParams(HasInputCol, HasInputCols, HasOutputCol, HasOutputCols, HasRelativeError):
class _OneHotEncoderParams(HasInputCol, HasInputCols, HasOutputCol, HasOutputCols,
class _RobustScalerParams(HasInputCol, HasOutputCol, HasRelativeError):
class HasRelativeError(Params):
class ResolveCoalesceHints(conf: SQLConf) extends Rule[LogicalPlan]
case class AlterTableRecoverPartitionsStatement(
case class DropNamespaceStatement(
case class LoadDataStatement(
case class ShowColumnsStatement(
case class DropNamespace(
case class DropNamespaceExec(

cloud-fan · 2019-10-31T11:31:23Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/IntervalUtils.scala

      micros = Math.addExact(micros, Math.multiplyExact(hours, MICROS_PER_HOUR))
      micros = Math.addExact(micros, Math.multiplyExact(minutes, MICROS_PER_MINUTE))
      micros = Math.addExact(micros, Math.multiplyExact(seconds, DateTimeUtils.MICROS_PER_SECOND))
-      new CalendarInterval(0, sign * micros)
+      new CalendarInterval(0, sign * days.toInt, sign * micros)


nit: we can cast to int at the beginning

val days = if (m.group(2) == null) { 0 } else { toLongWithRange("day", m.group(3), 0, Integer.MAX_VALUE).toInt }

cloud-fan · 2019-10-31T11:33:55Z

sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CastSuite.scala

      StringType),
-      "interval 1 years 3 months -3 days")
+      "interval 1 years 3 months 1 weeks 2 days -3 hours")


@MaxGekk @dongjoon-hyun this is another instance we can think of: I feel 9 days is more readable than 1 week 2 days.

cloud-fan · 2019-10-31T11:41:51Z

LGTM except a few minor comments, thanks for your work and patience @LinhongLiu !

cloud-fan · 2019-10-31T15:16:34Z

...catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala

@@ -2702,7 +2715,10 @@ object Sequence {
         |    $arr[$i] = ($elemType) ($t / ${scale}L);
         |    $i += 1;
         |    $t = org.apache.spark.sql.catalyst.util.DateTimeUtils.timestampAddInterval(
-         |       $startMicros, $i * $stepMonths, $i * $stepMicros, $zid);
+         |       $startMicros,
+         |       new org.apache.spark.unsafe.types.CalendarInterval(


Sorry I missed this place. It has performance penalty if we create an object every time. Let's go back to the previous version. Sorry for the back and forth!

np. I reset the changes back.

SparkQA · 2019-10-31T17:29:43Z

Test build #113034 has finished for PR 26134 at commit fb1591e.

This patch fails SparkR unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2019-11-01T04:18:51Z

Test build #113061 has finished for PR 26134 at commit 2dd8e43.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2019-11-01T08:23:54Z

Test build #113073 has finished for PR 26134 at commit 2f90189.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

LinhongLiu · 2019-11-01T08:28:50Z

@cloud-fan tests are all pass, could you please check this PR again?

cloud-fan · 2019-11-01T10:12:45Z

thanks, merging to master!

LinhongLiu · 2019-11-01T10:55:13Z

thanks @cloud-fan @xuanyuanking @MaxGekk for your reviewing.

zsxwing · 2019-11-26T19:01:55Z

common/unsafe/src/main/java/org/apache/spark/unsafe/types/CalendarInterval.java

  public final long microseconds;

  public long milliseconds() {
    return this.microseconds / MICROS_PER_MILLI;
  }

-  public CalendarInterval(int months, long microseconds) {
+  public CalendarInterval(int months, int days, long microseconds) {


@LinhongLiu Could you send out a follow up PR to document why we need days and how do we use it in this file? This is definitely worth to document. Otherwise, everyone reading this class may need to git blame and go through the long discussion in this PR.

@LinhongLiu Could you send out a follow up PR to document why we need days and how do we use it in this file? This is definitely worth to document. Otherwise, everyone reading this class may need to git blame and go through the long discussion in this PR.

Sure, will do.

### What changes were proposed in this pull request? Follow up of #26134 to document the reason to add days filed and explain how do we use it ### Why are the changes needed? only comment ### Does this PR introduce any user-facing change? no ### How was this patch tested? no need test Closes #26701 from LinhongLiu/spark-29486-followup. Authored-by: Liu,Linhong <liulinhong@baidu.com> Signed-off-by: Sean Owen <sean.owen@databricks.com>

### What changes were proposed in this pull request? Follow up of apache#26134 to document the reason to add days filed and explain how do we use it ### Why are the changes needed? only comment ### Does this PR introduce any user-facing change? no ### How was this patch tested? no need test Closes apache#26701 from LinhongLiu/spark-29486-followup. Authored-by: Liu,Linhong <liulinhong@baidu.com> Signed-off-by: Sean Owen <sean.owen@databricks.com>

LinhongLiu added 7 commits October 16, 2019 13:51

modify class CalendarInterval and CalendarIntervalSuite

44d66d6

modify DateTimeUtils, DateTimeUtilsSuite, TemporalSequenceImpl

4979e1e

modify CalendarInterval related classes in sql/catalyst

f09b529

fix test in sql/catalyst

0db5b7a

fix remaining tests

ce879c2

fix failed tests

4ee8354

code clean and more bug fix

3d954d6

LinhongLiu commented Oct 16, 2019

View reviewed changes

LinhongLiu changed the title ~~Calendarinterval~~ CalendarInterval should have 3 fields: months, days and microseconds Oct 16, 2019

LinhongLiu changed the title ~~CalendarInterval should have 3 fields: months, days and microseconds~~ [SPARK-29486][SQL] CalendarInterval should have 3 fields: months, days and microseconds Oct 16, 2019

cloud-fan mentioned this pull request Oct 16, 2019

[SPARK-29448][SQL] Support the INTERVAL type by Parquet datasource #26102

Closed

MaxGekk reviewed Oct 16, 2019

View reviewed changes

xuanyuanking reviewed Oct 16, 2019

View reviewed changes

LinhongLiu added 2 commits October 16, 2019 22:25

address comments and fix tests

05042e8

Merge remote-tracking branch 'origin/master' into calendarinterval

4c31401

cloud-fan reviewed Oct 28, 2019

View reviewed changes

fix comment

4e97bc0

LinhongLiu force-pushed the calendarinterval branch from 336f0db to 4e97bc0 Compare October 29, 2019 03:07

Merge remote-tracking branch 'origin/master' into calendarinterval

0e87e2d

cloud-fan reviewed Oct 31, 2019

View reviewed changes

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeExpressions.scala Show resolved Hide resolved

cloud-fan reviewed Oct 31, 2019

View reviewed changes

LinhongLiu force-pushed the calendarinterval branch from fb1591e to 2dd8e43 Compare November 1, 2019 02:14

address comments

2f90189

LinhongLiu force-pushed the calendarinterval branch from 2dd8e43 to 2f90189 Compare November 1, 2019 05:44

cloud-fan closed this in a4382f7 Nov 1, 2019

zsxwing reviewed Nov 26, 2019

View reviewed changes

LinhongLiu mentioned this pull request Nov 28, 2019

[SPARK-29486][SQL][FOLLOWUP] Document the reason to add days field #26701

Closed

yikf mentioned this pull request Jan 25, 2022

[SPARK-38018][SQL] Fix ColumnVectorUtils.populate to handle CalendarIntervalType correctly #35314

Closed

[SPARK-29486][SQL] CalendarInterval should have 3 fields: months, days and microseconds #26134

[SPARK-29486][SQL] CalendarInterval should have 3 fields: months, days and microseconds #26134

Conversation

LinhongLiu commented Oct 16, 2019

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Choose a reason for hiding this comment

Choose a reason for hiding this comment

LinhongLiu commented Oct 16, 2019

cloud-fan commented Oct 16, 2019

LinhongLiu commented Oct 16, 2019

cloud-fan commented Oct 16, 2019

SparkQA commented Oct 16, 2019

xuanyuanking commented Oct 16, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

MaxGekk commented Oct 16, 2019

cloud-fan commented Oct 16, 2019

MaxGekk commented Oct 16, 2019 • edited Loading

SparkQA commented Oct 16, 2019

MaxGekk commented Oct 16, 2019

LinhongLiu commented Oct 16, 2019 • edited Loading

Choose a reason for hiding this comment

cloud-fan commented Oct 16, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SparkQA commented Oct 29, 2019

SparkQA commented Oct 29, 2019

SparkQA commented Oct 31, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cloud-fan commented Oct 31, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SparkQA commented Oct 31, 2019

SparkQA commented Nov 1, 2019

SparkQA commented Nov 1, 2019

LinhongLiu commented Nov 1, 2019

cloud-fan commented Nov 1, 2019

LinhongLiu commented Nov 1, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

MaxGekk commented Oct 16, 2019 •

edited

Loading

LinhongLiu commented Oct 16, 2019 •

edited

Loading