Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[SPARK-7084] improve saveAsTable documentation #5654

Closed
wants to merge 5 commits into from
Closed

[SPARK-7084] improve saveAsTable documentation #5654

wants to merge 5 commits into from

Conversation

phatak-dev
Copy link
Contributor

No description provided.

@AmplabJenkins
Copy link

Can one of the admins verify this patch?

rxin
rxin reviewed Apr 23, 2015
View reviewed changes
sql/core/src/main/scala/org/apache/spark/sql/DataFrame.scala
@@ -1085,6 +1085,9 @@ class DataFrame private[sql](
* there is no notion of a persisted catalog in a standard SQL context. Instead you can write
* an RDD out to a parquet file, and then register that file as a table. This "table" can then
* be the target of an `insertInto`.
*
* Also note that this doesn't create a hive table, but instead creates a Spark data source table.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe say

"Also note that while this function can persist the table metadata into Hive's metastore, the table will NOT be accessible from Hive."

@phatak-dev
Copy link
Contributor Author

Updated.

@rxin
Copy link
Contributor

rxin commented Apr 23, 2015

Jenkins, test this please.

@SparkQA
Copy link

SparkQA commented Apr 23, 2015

Test build #30856 has started for PR 5654 at commit 00bc819.

@SparkQA
Copy link

SparkQA commented Apr 23, 2015

Test build #30856 has finished for PR 5654 at commit 00bc819.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds the following public classes (experimental):
    • class FreqItemset(namedtuple("FreqItemset", ["items", "freq"])):
    • protected[sql] abstract class AtomicType extends DataType
    • abstract class NumericType extends AtomicType
    • class Encoder[T <: AtomicType](columnType: NativeColumnType[T]) extends compression.Encoder[T]
    • class Decoder[T <: AtomicType](buffer: ByteBuffer, columnType: NativeColumnType[T])
    • class Encoder[T <: AtomicType](columnType: NativeColumnType[T]) extends compression.Encoder[T]
    • class Decoder[T <: AtomicType](buffer: ByteBuffer, columnType: NativeColumnType[T])
    • class Encoder[T <: AtomicType](columnType: NativeColumnType[T]) extends compression.Encoder[T]
    • class Decoder[T <: AtomicType](buffer: ByteBuffer, columnType: NativeColumnType[T])
  • This patch does not change any dependencies.

@AmplabJenkins
Copy link

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/30856/
Test PASSed.

rxin
rxin reviewed Apr 23, 2015
View reviewed changes
sql/core/src/main/scala/org/apache/spark/sql/DataFrame.scala
@@ -1085,6 +1085,9 @@ class DataFrame private[sql](
* there is no notion of a persisted catalog in a standard SQL context. Instead you can write
* an RDD out to a parquet file, and then register that file as a table. This "table" can then
* be the target of an `insertInto`.
*
* Also note that while this function can persist the table metadata into Hive's metastore,
* the table will NOT be accessible from Hive.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are multiple saveAsTable functions. Do you mind updating them as well?

@phatak-dev
Copy link
Contributor Author

Added for other methods also.

@AmplabJenkins
Copy link

Can one of the admins verify this patch?

@rxin
Copy link
Contributor

rxin commented May 11, 2015

Jenkins, test this please.

@AmplabJenkins
Copy link

Merged build triggered.

@AmplabJenkins
Copy link

Merged build started.

@SparkQA
Copy link

SparkQA commented May 11, 2015

Test build #32428 has started for PR 5654 at commit 386f407.

@rxin
Copy link
Contributor

rxin commented May 11, 2015

LGTM. I will merge it after Jenkins comes back happy.

@SparkQA
Copy link

SparkQA commented May 11, 2015

Test build #32428 has finished for PR 5654 at commit 386f407.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@AmplabJenkins
Copy link

Merged build finished. Test PASSed.

@AmplabJenkins
Copy link

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/32428/
Test PASSed.

@asfgit asfgit closed this in 57255dc May 12, 2015
asfgit pushed a commit that referenced this pull request May 12, 2015
@phatak-dev @rxin
[SPARK-7084] improve saveAsTable documentation
0dbfe16 
Author: madhukar <phatak.dev@gmail.com>

Closes #5654 from phatak-dev/master and squashes the following commits:

386f407 [madhukar] #5654 updated for all the methods
2c997c5 [madhukar] Merge branch 'master' of https://github.com/apache/spark
00bc819 [madhukar] Merge branch 'master' of https://github.com/apache/spark
2a802c6 [madhukar] #5654 updated the doc according to comments
866e8df [madhukar] [SPARK-7084] improve saveAsTable documentation

(cherry picked from commit 57255dc)
Signed-off-by: Reynold Xin <rxin@databricks.com>
asfgit pushed a commit that referenced this pull request May 12, 2015
@phatak-dev @rxin
[SPARK-7084] improve saveAsTable documentation
8f99a49 
Author: madhukar <phatak.dev@gmail.com>

Closes #5654 from phatak-dev/master and squashes the following commits:

386f407 [madhukar] #5654 updated for all the methods
2c997c5 [madhukar] Merge branch 'master' of https://github.com/apache/spark
00bc819 [madhukar] Merge branch 'master' of https://github.com/apache/spark
2a802c6 [madhukar] #5654 updated the doc according to comments
866e8df [madhukar] [SPARK-7084] improve saveAsTable documentation

(cherry picked from commit 57255dc)
Signed-off-by: Reynold Xin <rxin@databricks.com>
@rxin
Copy link
Contributor

rxin commented May 12, 2015

FYI I submitted a small patch on top of this to add a link to a jira ticket: #6067

jeanlyn pushed a commit to jeanlyn/spark that referenced this pull request May 28, 2015
@phatak-dev @jeanlyn
[SPARK-7084] improve saveAsTable documentation
697c79d 
Author: madhukar <phatak.dev@gmail.com>

Closes apache#5654 from phatak-dev/master and squashes the following commits:

386f407 [madhukar] apache#5654 updated for all the methods
2c997c5 [madhukar] Merge branch 'master' of https://github.com/apache/spark
00bc819 [madhukar] Merge branch 'master' of https://github.com/apache/spark
2a802c6 [madhukar] apache#5654 updated the doc according to comments
866e8df [madhukar] [SPARK-7084] improve saveAsTable documentation
jeanlyn pushed a commit to jeanlyn/spark that referenced this pull request Jun 12, 2015
@phatak-dev @jeanlyn
[SPARK-7084] improve saveAsTable documentation
841fa8b 
Author: madhukar <phatak.dev@gmail.com>

Closes apache#5654 from phatak-dev/master and squashes the following commits:

386f407 [madhukar] apache#5654 updated for all the methods
2c997c5 [madhukar] Merge branch 'master' of https://github.com/apache/spark
00bc819 [madhukar] Merge branch 'master' of https://github.com/apache/spark
2a802c6 [madhukar] apache#5654 updated the doc according to comments
866e8df [madhukar] [SPARK-7084] improve saveAsTable documentation
@chiyingyunhua
Copy link

请问，我使用spark1.3.1的sparksql中dataframe的一个saveastable方法存了一个dataframe，但是用hive读的时候出错了Failed with exception java.io.IOException:java.io.IOException: hdfs://namenode71:8020/user/hive/warehouse/zz5/part-00000 not a SequenceFile。这个怎么破？1.2是可以的

nemccarthy pushed a commit to nemccarthy/spark that referenced this pull request Jun 19, 2015
@phatak-dev @nemccarthy
[SPARK-7084] improve saveAsTable documentation
d5f3162 
Author: madhukar <phatak.dev@gmail.com>

Closes apache#5654 from phatak-dev/master and squashes the following commits:

386f407 [madhukar] apache#5654 updated for all the methods
2c997c5 [madhukar] Merge branch 'master' of https://github.com/apache/spark
00bc819 [madhukar] Merge branch 'master' of https://github.com/apache/spark
2a802c6 [madhukar] apache#5654 updated the doc according to comments
866e8df [madhukar] [SPARK-7084] improve saveAsTable documentation
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@phatak-dev @AmplabJenkins @rxin @SparkQA @chiyingyunhua