-
Notifications
You must be signed in to change notification settings - Fork 28.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[SPARK-4945] [SQL] Add overwrite option support for SchemaRDD.saveAsParquetFile #3780
Conversation
Test build #24755 has started for PR 3780 at commit
|
Test build #24755 has finished for PR 3780 at commit
|
Test FAILed. |
Test build #24758 has started for PR 3780 at commit
|
Test build #24758 has finished for PR 3780 at commit
|
Test FAILed. |
52adf46
to
a7a380f
Compare
Test build #24802 has started for PR 3780 at commit
|
Test build #24802 has finished for PR 3780 at commit
|
Test PASSed. |
ParquetRelation.create(path, child, sparkContext.hadoopConfiguration, sqlContext) | ||
// Note: overwrite=false because otherwise the metadata we just created will be deleted | ||
InsertIntoParquetTable(relation, planLater(child), overwrite = false) :: Nil | ||
ParquetRelation.createEmpty( |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is it safe to replace create
with createEmpty
? I see create
will do some check and then call createEmpty
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It should be safe, as we believe the logical plan has been resolved already during the analysis phase.
/cc @rxin More API questions. |
a7a380f
to
72c4a4b
Compare
Test build #25409 has started for PR 3780 at commit
|
Test build #25409 has finished for PR 3780 at commit
|
Test PASSed. |
} catch { | ||
case e: IOException => | ||
throw new IOException( | ||
s"Unable to clear output directory ${path}") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
put this on the previous line?
The changes lgtm. I left couple small comments. Do you mind fixing those? |
@rxin what about the question of |
For my understanding, |
Test build #25860 has started for PR 3780 at commit
|
Or probably we can cleanup those APIs, kind of like:
Any ideas? |
Test build #25860 has finished for PR 3780 at commit
|
Test PASSed. |
I kind of like a proposal like that, but we need to make it Java friendly as well. |
OK, I will close this. |
No description provided.