[deprecated] [SPARK-5821] [SPARK-5746] [SQL] JSON external data source INSERT operation improvements

[yanboliang]

This PR was deprecated, please move to #4610 .
JSON external data source INSERT operation improvements and bug fix:
1, The path in "CREATE TABLE AS SELECT" must be a directory. Because in this scenario we need to write or append files to the existed table, underlying directory is more reasonable for append operation, authentication and authorization.
For SPARK-5821, if we don't have write permission for the parent directory, the CTAS command will failure.
Another reason is that we can't append to HDFS files which represent RDD, if we want to implement append semantics, we need new files and add to a specific directory.
2, New INSERT OVERWRITE implementation.
First insert the new generated table to a temporary directory which named as "_temporary" under the path directory. After insert finished, we deleted the original files. At last we rename "_temporary" for "data".
This can fix the bug which mentioned at SPARK-5746.
3, Why to rename "_temporary" for "data" rather than move all files in "_temporary" to path and then delete "_temporary"? Because that spark RDD.saveAsTextFile(path) related operation will store the whole RDD to HDFS files which named as "part-* " like files under the path. If the original files were produced by this mean, and then we use "INSERT" without overwrite, the new generated table files are also named as "part-* " which will produce corrupted table.

This is the initial draft and need optimization. Looking forward your opinions and comments.

[SparkQA]

Test build #27494 has started for PR 4607 at commit 0812dd1.

• This patch merges cleanly.

[SparkQA]

Test build #27494 has finished for PR 4607 at commit 0812dd1.

• This patch fails Scala style tests.
• This patch merges cleanly.
• This patch adds no public classes.

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27494/
Test FAILed.

[SparkQA]

Test build #27495 has started for PR 4607 at commit 29e138a.

• This patch merges cleanly.

[SparkQA]

Test build #27495 has finished for PR 4607 at commit 29e138a.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27495/
Test FAILed.

[SparkQA]

Test build #27502 has started for PR 4607 at commit 41307cd.

• This patch merges cleanly.

[SparkQA]

Test build #27502 has finished for PR 4607 at commit 41307cd.

• This patch fails Scala style tests.
• This patch merges cleanly.
• This patch adds no public classes.

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27502/
Test FAILed.

[yhuai]

@yanbohappy Thank you for working on it! For SPARK-5746, I think it is better to add an analysis rule to do a check and throw an exception when you find that users try to write to a table while reading it. Actually, I have been working on it and will have a PR soon. How about we use this PR to address the issue of SPARK-5821 in JSONRelation? Can you also try the parquet data source and see if SPARK-5821 also affects that?

[yhuai]

Actually, I think we just need to throw an exception if the delete returns false when we try to delete the existing data for OVERWRITE (we also need to make the change at https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/json/JSONRelation.scala#L70).

[yanboliang]

@yhuai Thank you for your reply. Add analysis rule and throw an exception is reasonable and looking forward your PR.
I can address the issue of SPARK-5821, I'm working on another PR #4610 not only resolve SPARK-5821 but also with some improvements.
Could I close this PR and discuss JSON data source improvement related problem at #4610 ？

[yanboliang]

Actually, the insert function (https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/json/JSONRelation.scala#L107) will be not called any time. The "CTAS" command is just executed at https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/json/JSONRelation.scala#L81

[yhuai]

Yes, please close it.

insert is used by INSERT INTO/OVERWRITE and DataFrame.insertInto.