[SPARK-24360][SQL] Support Hive 3.1 metastore #23694

dongjoon-hyun · 2019-01-30T07:16:36Z

What changes were proposed in this pull request?

Hive 3.1.1 is released. This PR aims to support Hive 3.1.x metastore.
Please note that Hive 3.0.0 Metastore is skipped intentionally.

How was this patch tested?

Pass the Jenkins with the updated test cases including 3.1.

HyukjinKwon · 2019-01-30T07:21:05Z

sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveShim.scala

@@ -1179,3 +1180,128 @@ private[client] class Shim_v2_1 extends Shim_v2_0 {
 private[client] class Shim_v2_2 extends Shim_v2_1

 private[client] class Shim_v2_3 extends Shim_v2_1
+
+private[client] class Shim_v3_1 extends Shim_v2_3 {


I think it's fine if the signatures are matched and the tests pass. Let me double check them within tomorrow.

SparkQA · 2019-01-30T08:05:01Z

Test build #101874 has finished for PR 23694 at commit a6a0810.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2019-01-30T08:05:02Z

Test build #101876 has finished for PR 23694 at commit 9f621e7.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

dongjoon-hyun · 2019-01-30T08:06:12Z

Retest this please.

gatorsmile · 2019-01-30T08:20:42Z

sql/hive/src/main/scala/org/apache/spark/sql/hive/client/IsolatedClientLoader.scala

@@ -100,6 +100,7 @@ private[hive] object IsolatedClientLoader extends Logging {
    case "2.1" | "2.1.0" | "2.1.1" => hive.v2_1
    case "2.2" | "2.2.0" => hive.v2_2
    case "2.3" | "2.3.0" | "2.3.1" | "2.3.2" | "2.3.3" | "2.3.4" => hive.v2_3
+    case "3.1" | "3.1.0" | "3.1.1" => hive.v3_1


What is the reason we do not support 3.0?

It's not stable like Hadoop 3.0. Spark skipped Hadoop 3.0 and go with Hadoop 3.1.

I think we face the same issue in 1.0, 2.0, but we still support them. Could we simply support it and let the end-users decide it by themselves?

Then, hopefully, I can proceed that in another PR.

SparkQA · 2019-01-30T10:34:29Z

Test build #101877 has finished for PR 23694 at commit 9f621e7.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

dongjoon-hyun · 2019-01-30T23:58:12Z

Also, cc @wangyum since he is working on Hive upgrade.

HyukjinKwon · 2019-01-31T03:31:49Z

sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveShim.scala

+      numDP: JInteger, listBucketingLevel, isAcid, writeIdInLoadTableOrPartition,
+      stmtIdInLoadTableOrPartition, hasFollowingStatsTask, AcidUtils.Operation.NOT_ACID,
+      replace: JBoolean)
+  }


@dongjoon-hyun, not a big deal but how about adding dropIndex with, say, unsupported exception?

It's possible, but it doesn't help much. As we see here, Hive.getIndexes raises NoSuchMethodError before we call shim.dropIndex.

Hive.java in 2.3 has getIndexes.

Hive.java in 3.0 and master doesn't have it.

Yea, I noticed it. I am leaving this comment orthogonally just as a sanity check.

HyukjinKwon

LGTM except one question

HyukjinKwon · 2019-01-31T03:47:37Z

sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveShim.scala

+      classOf[AcidUtils.Operation],
+      JBoolean.TYPE)
+
+  override def loadPartition(


HyukjinKwon · 2019-01-31T03:47:43Z

sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveShim.scala

+    assert(session.nonEmpty)
+    val database = session.get.sessionState.catalog.getCurrentDatabase
+    val table = hive.getTable(database, tableName)
+    val loadFileType = if (replace) {


HyukjinKwon · 2019-01-31T03:56:04Z

sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveShim.scala

+      clazzLoadFileType.getEnumConstants.find(_.toString.equalsIgnoreCase("KEEP_EXISTING"))
+    }
+    assert(loadFileType.isDefined)
+    loadDynamicPartitionsMethod.invoke(hive, loadPath, tableName, partSpec, loadFileType.get,


HyukjinKwon · 2019-01-31T03:56:07Z

sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveShim.scala

+      writeIdInLoadTableOrPartition, stmtIdInLoadTableOrPartition: JInteger, replace: JBoolean)
+  }
+
+  override def loadDynamicPartitions(


HyukjinKwon · 2019-01-31T04:00:45Z

sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveShim.scala

+
+private[client] class Shim_v3_1 extends Shim_v2_3 {
+  // Spark supports only non-ACID operations
+  protected lazy val isAcidIUDoperation = JBoolean.FALSE


@dongjoon-hyun, isn't it isAcid defined at Shim_v0_14? Was wondering why this was separate variable again. Do you see any possibility that this is different specifically for 3.1? Then, it's fine.

Historically, isAcidIUDoperation is an evolved one from isAcid.

In Hive code, isAcid was a general ACID operation and isAcidIUDoperation is now used for ACID Insert/Update/Delete operations. Also, they checks isFullAcidTable and use it together like this.

else if(!isAcidIUDoperation && isFullAcidTable) { destPath = fixFullAcidPathForLoadData(loadFileType, destPath, txnId, stmtId, tbl); }

And yes for your last question. We don't know the future of Hive. So, for the different parameter name, we had better handle differently. That was my logic.

HyukjinKwon · 2019-01-31T04:01:23Z

sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveShim.scala

+      writeIdInLoadTableOrPartition, stmtIdInLoadTableOrPartition, replace: JBoolean)
+  }
+
+  override def loadTable(


HyukjinKwon · 2019-01-31T04:01:30Z

sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveShim.scala

+      clazzLoadFileType.getEnumConstants.find(_.toString.equalsIgnoreCase("KEEP_EXISTING"))
+    }
+    assert(loadFileType.isDefined)
+    loadTableMethod.invoke(hive, loadPath, tableName, loadFileType.get, isSrcLocal: JBoolean,


dongjoon-hyun · 2019-01-31T04:32:46Z

Thank you so much for review, @HyukjinKwon and @gatorsmile .
Merged to master.

## What changes were proposed in this pull request? Hive 3.1.1 is released. This PR aims to support Hive 3.1.x metastore. Please note that Hive 3.0.0 Metastore is skipped intentionally. ## How was this patch tested? Pass the Jenkins with the updated test cases including 3.1. Closes apache#23694 from dongjoon-hyun/SPARK-24360-3.1. Authored-by: Dongjoon Hyun <dongjoon@apache.org> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>

Hive 3.1.1 is released. This PR aims to support Hive 3.1.x metastore. Please note that Hive 3.0.0 Metastore is skipped intentionally. Pass the Jenkins with the updated test cases including 3.1. Closes apache#23694 from dongjoon-hyun/SPARK-24360-3.1. Authored-by: Dongjoon Hyun <dongjoon@apache.org> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>

* Compiles with Hops Hive * [SPARK-24360][SQL] Support Hive 3.1 metastore Hive 3.1.1 is released. This PR aims to support Hive 3.1.x metastore. Please note that Hive 3.0.0 Metastore is skipped intentionally. Pass the Jenkins with the updated test cases including 3.1. Closes apache#23694 from dongjoon-hyun/SPARK-24360-3.1. Authored-by: Dongjoon Hyun <dongjoon@apache.org> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>

* Compiles with Hops Hive * [SPARK-24360][SQL] Support Hive 3.1 metastore Hive 3.1.1 is released. This PR aims to support Hive 3.1.x metastore. Please note that Hive 3.0.0 Metastore is skipped intentionally. Pass the Jenkins with the updated test cases including 3.1. Closes apache#23694 from dongjoon-hyun/SPARK-24360-3.1. Authored-by: Dongjoon Hyun <dongjoon@apache.org> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org> [HOPSWORKS-385] append - Bump Hive dependency [HOPSWORKS-385] append - Exclude log4j dependency (#10) * [HOPSWORKS-1105] Bump Hops version to 2.8.2.8 * [HOPSWORKS-385] append - Exclude log4j dependency

## What changes were proposed in this pull request? Hive 3.1.1 is released. This PR aims to support Hive 3.1.x metastore. Please note that Hive 3.0.0 Metastore is skipped intentionally. ## How was this patch tested? Pass the Jenkins with the updated test cases including 3.1. Closes apache#23694 from dongjoon-hyun/SPARK-24360-3.1. Authored-by: Dongjoon Hyun <dongjoon@apache.org> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>

[SPARK-24360][SQL] Support Hive 3.1 metastore

a6a0810

dongjoon-hyun mentioned this pull request Jan 30, 2019

[SPARK-24360][SQL] Support Hive 3.0 metastore #21404

Closed

HyukjinKwon reviewed Jan 30, 2019

View reviewed changes

fix

9f621e7

gatorsmile reviewed Jan 30, 2019

View reviewed changes

HyukjinKwon reviewed Jan 31, 2019

View reviewed changes

HyukjinKwon approved these changes Jan 31, 2019

View reviewed changes

dongjoon-hyun closed this in aeff69b Jan 31, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-24360][SQL] Support Hive 3.1 metastore #23694

[SPARK-24360][SQL] Support Hive 3.1 metastore #23694

dongjoon-hyun commented Jan 30, 2019

HyukjinKwon Jan 30, 2019

dongjoon-hyun Jan 30, 2019

SparkQA commented Jan 30, 2019

SparkQA commented Jan 30, 2019

dongjoon-hyun commented Jan 30, 2019

gatorsmile Jan 30, 2019

dongjoon-hyun Jan 30, 2019 •

edited

Loading

gatorsmile Jan 30, 2019

dongjoon-hyun Jan 30, 2019

SparkQA commented Jan 30, 2019

dongjoon-hyun commented Jan 30, 2019

HyukjinKwon Jan 31, 2019

dongjoon-hyun Jan 31, 2019 •

edited

Loading

HyukjinKwon Jan 31, 2019

HyukjinKwon left a comment

HyukjinKwon Jan 31, 2019

HyukjinKwon Jan 31, 2019

HyukjinKwon Jan 31, 2019

HyukjinKwon Jan 31, 2019

HyukjinKwon Jan 31, 2019

dongjoon-hyun Jan 31, 2019 •

edited

Loading

HyukjinKwon Jan 31, 2019

HyukjinKwon Jan 31, 2019

dongjoon-hyun commented Jan 31, 2019 •

edited

Loading

[SPARK-24360][SQL] Support Hive 3.1 metastore #23694

[SPARK-24360][SQL] Support Hive 3.1 metastore #23694

Conversation

dongjoon-hyun commented Jan 30, 2019

What changes were proposed in this pull request?

How was this patch tested?

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SparkQA commented Jan 30, 2019

SparkQA commented Jan 30, 2019

dongjoon-hyun commented Jan 30, 2019

Choose a reason for hiding this comment

dongjoon-hyun Jan 30, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SparkQA commented Jan 30, 2019

dongjoon-hyun commented Jan 30, 2019

Choose a reason for hiding this comment

dongjoon-hyun Jan 31, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

HyukjinKwon left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dongjoon-hyun Jan 31, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dongjoon-hyun commented Jan 31, 2019 • edited Loading

dongjoon-hyun Jan 30, 2019 •

edited

Loading

dongjoon-hyun Jan 31, 2019 •

edited

Loading

dongjoon-hyun Jan 31, 2019 •

edited

Loading

dongjoon-hyun commented Jan 31, 2019 •

edited

Loading