-
Notifications
You must be signed in to change notification settings - Fork 28.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[SPARK-8961] [SQL] Makes BaseWriterContainer.outputWriterForRow accepts InternalRow instead of Row #7331
Conversation
Test build #36970 has finished for PR 7331 at commit
|
cc @rxin |
@@ -19,6 +19,8 @@ package org.apache.spark.sql.sources | |||
|
|||
import java.util.{Date, UUID} | |||
|
|||
import scala.collection.JavaConversions._ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I explicitly didn't include this wildcard implicit import because I didn't want future code to accidentally introduce a scala wrapper on the java hashmap.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Got it. Converting the iterator explicitly now.
lgtm |
Test build #37062 has finished for PR 7331 at commit
|
Thanks - merging. |
Actually couldn't merge. Not sure what's going on.
|
Let me try to merge it. |
Merged to master. |
…ts InternalRow instead of Row This is a follow-up of [SPARK-8888] [1], which also aims to optimize writing dynamic partitions. Three more changes can be made here: 1. Using `InternalRow` instead of `Row` in `BaseWriterContainer.outputWriterForRow` 2. Using `Cast` expressions to convert partition columns to strings, so that we can leverage code generation. 3. Replacing the FP-style `zip` and `map` calls with a faster imperative `while` loop. [1]: https://issues.apache.org/jira/browse/SPARK-8888 Author: Cheng Lian <lian@databricks.com> Closes #7331 from liancheng/spark-8961 and squashes the following commits: b5ab9ae [Cheng Lian] Casts Java iterator to Scala iterator explicitly 719e63b [Cheng Lian] Makes BaseWriterContainer.outputWriterForRow accepts InternalRow instead of Row
This is a follow-up of SPARK-8888, which also aims to optimize writing dynamic partitions.
Three more changes can be made here:
InternalRow
instead ofRow
inBaseWriterContainer.outputWriterForRow
Cast
expressions to convert partition columns to strings, so that we can leverage code generation.zip
andmap
calls with a faster imperativewhile
loop.