[SPARK-25313][SQL][FOLLOW-UP] Fix InsertIntoHiveDirCommand output schema in Parquet issue #22359
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| @@ -803,6 +803,23 @@ class HiveDDLSuite | |||
| } | |||
| } | |||
|
|
|||
| test("Insert overwrite directory should output correct schema") { | |||
| withSQLConf(CONVERT_METASTORE_PARQUET.key -> "false") { | |||
There was a problem hiding this comment.
Add withTable("tbl") { here.
gengliangwang
approved these changes
Sep 7, 2018
|
Test build #95788 has finished for PR 22359 at commit
|
|
Test build #95791 has finished for PR 22359 at commit
|
|
Hi, @wangyum .
|
| @@ -803,6 +803,25 @@ class HiveDDLSuite | |||
| } | |||
| } | |||
|
|
|||
| test("Insert overwrite directory should output correct schema") { | |||
There was a problem hiding this comment.
Since this is a bug fix, can we have SPARK-25313 prefix?
There was a problem hiding this comment.
There was a problem hiding this comment.
In this PR, let's handle this test case only.
wangyum
changed the title
|
I removed my previous comment. It seems to be the Parquet behavior from the beginning of this command at 2.3.0. I was confused because it's different from ORC. |
|
Test build #95818 has finished for PR 22359 at commit
|
|
Since this is related to Parquet behavior only, can we have |
wangyum
changed the title
|
cc @cloud-fan |
|
thanks, merging to master/2.4! |
asfgit
pushed a commit
that referenced
this pull request
Sep 10, 2018
…ema in Parquet issue
## What changes were proposed in this pull request?
How to reproduce:
```scala
spark.sql("CREATE TABLE tbl(id long)")
spark.sql("INSERT OVERWRITE TABLE tbl VALUES 4")
spark.sql("CREATE VIEW view1 AS SELECT id FROM tbl")
spark.sql(s"INSERT OVERWRITE LOCAL DIRECTORY '/tmp/spark/parquet' " +
"STORED AS PARQUET SELECT ID FROM view1")
spark.read.parquet("/tmp/spark/parquet").schema
scala> spark.read.parquet("/tmp/spark/parquet").schema
res10: org.apache.spark.sql.types.StructType = StructType(StructField(id,LongType,true))
```
The schema should be `StructType(StructField(ID,LongType,true))` as we `SELECT ID FROM view1`.
This pr fix this issue.
## How was this patch tested?
unit tests
Closes #22359 from wangyum/SPARK-25313-FOLLOW-UP.
Authored-by: Yuming Wang <yumwang@ebay.com>
Signed-off-by: Wenchen Fan <wenchen@databricks.com>
(cherry picked from commit f8b4d5a)
Signed-off-by: Wenchen Fan <wenchen@databricks.com>
asfgit
pushed a commit
that referenced
this pull request
Sep 11, 2018
…and output schema in Parquet issue ## What changes were proposed in this pull request? Backport #22359 to branch-2.3. ## How was this patch tested? unit tests Closes #22387 from wangyum/SPARK-25313-FOLLOW-UP-branch-2.3. Authored-by: Yuming Wang <yumwang@ebay.com> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
How to reproduce:
The schema should be
StructType(StructField(ID,LongType,true))as weSELECT ID FROM view1.This pr fix this issue.
How was this patch tested?
unit tests