Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[SPARK-6824] Fill the docs for DataFrame API in SparkR #5969

Closed
wants to merge 6 commits into from
Closed

[SPARK-6824] Fill the docs for DataFrame API in SparkR #5969

wants to merge 6 commits into from

Conversation

hqzizania
Copy link
Contributor

This patch also removes the RDD docs from being built as a part of roxygen just by the method to delete
" ' '" of " #' ".

@AmplabJenkins
Copy link

Merged build triggered.

@AmplabJenkins
Copy link

Merged build started.

@SparkQA
Copy link

SparkQA commented May 7, 2015

Test build #32087 has started for PR 5969 at commit 6ebcb07.

@hqzizania hqzizania closed this May 7, 2015
@hqzizania hqzizania reopened this May 7, 2015
@AmplabJenkins
Copy link

Merged build triggered.

@AmplabJenkins
Copy link

Merged build started.

@AmplabJenkins
Copy link

Merged build triggered.

@AmplabJenkins
Copy link

Merged build started.

@SparkQA
Copy link

SparkQA commented May 7, 2015

Test build #32095 has started for PR 5969 at commit 6394579.

@SparkQA
Copy link

SparkQA commented May 7, 2015

Test build #32087 has finished for PR 5969 at commit 6ebcb07.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@AmplabJenkins
Copy link

Merged build finished. Test PASSed.

@AmplabJenkins
Copy link

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/32087/
Test PASSed.

@AmplabJenkins
Copy link

Merged build finished. Test FAILed.

@AmplabJenkins
Copy link

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/32094/
Test FAILed.

@SparkQA
Copy link

SparkQA commented May 7, 2015

Test build #32095 has finished for PR 5969 at commit 6394579.

  • This patch fails PySpark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@AmplabJenkins
Copy link

Merged build finished. Test FAILed.

@AmplabJenkins
Copy link

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/32095/
Test FAILed.

@shivaram
Copy link
Contributor

shivaram commented May 7, 2015

Jenkins, retest this please

@shivaram
Copy link
Contributor

shivaram commented May 7, 2015

Thanks @hqzizania - This is the right approach for this PR. There are a couple of more docs that we should make private -- one is the Broadcast class and its members like setValue, clearBroadcastVariables etc. and the other is methods in SparkContext like parallelize, textFile etc. Since RRDD is private, we shouldn't have docs for methods which generate them as well.

@AmplabJenkins
Copy link

Merged build triggered.

@AmplabJenkins
Copy link

Merged build started.

@SparkQA
Copy link

SparkQA commented May 7, 2015

Test build #32147 has started for PR 5969 at commit 6394579.

@SparkQA
Copy link

SparkQA commented May 7, 2015

Test build #32147 has finished for PR 5969 at commit 6394579.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@AmplabJenkins
Copy link

Merged build finished. Test PASSed.

@AmplabJenkins
Copy link

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/32147/
Test PASSed.

@AmplabJenkins
Copy link

Merged build triggered.

@AmplabJenkins
Copy link

Merged build started.

@SparkQA
Copy link

SparkQA commented May 8, 2015

Test build #32172 has started for PR 5969 at commit eb4b095.

@hqzizania
Copy link
Contributor Author

@shivaram I've remove docs about broadcast and context, and check Rd files with ones in NAMESPACE. But I wonder why length(rdd) exists in DataFrame export in NAMESPACE, and why export("sparkRSQL.init",
"sparkRHive.init") written twice in NAMESPACE?

@shivaram
Copy link
Contributor

shivaram commented May 8, 2015

Thanks @hqzizania - I missed that fact that length doesn't exist in DataFrame. We can remove it for now from the namespace. In the future we could add a length and dim function

Also the duplicates should be removed as well

@AmplabJenkins
Copy link

Merged build triggered.

@AmplabJenkins
Copy link

Merged build started.

@SparkQA
Copy link

SparkQA commented May 8, 2015

Test build #32175 has started for PR 5969 at commit 6d27696.

@SparkQA
Copy link

SparkQA commented May 8, 2015

Test build #32172 has finished for PR 5969 at commit eb4b095.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@AmplabJenkins
Copy link

Merged build finished. Test PASSed.

@AmplabJenkins
Copy link

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/32172/
Test PASSed.

@SparkQA
Copy link

SparkQA commented May 8, 2015

Test build #32175 has finished for PR 5969 at commit 6d27696.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@AmplabJenkins
Copy link

Merged build finished. Test PASSed.

@AmplabJenkins
Copy link

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/32175/
Test PASSed.

@shivaram
Copy link
Contributor

shivaram commented May 8, 2015

Thanks @hqzizania - LGTM. We could do a few more cleanups later based on how the knitr documents look (I think some of the rdnames we use might be good to change etc.) but lets do that in a follow up PR

asfgit pushed a commit that referenced this pull request May 8, 2015
@hqzizania @shivaram
[SPARK-6824] Fill the docs for DataFrame API in SparkR
4f01f5b 
This patch also removes the RDD docs from being built as a part of roxygen just by the method to delete
" ' '" of " \#' ".

Author: hqzizania <qian.huang@intel.com>
Author: qhuang <qian.huang@intel.com>

Closes #5969 from hqzizania/R1 and squashes the following commits:

6d27696 [qhuang] fixes in NAMESPACE
eb4b095 [qhuang] remove more docs
6394579 [qhuang] remove RDD docs in generics.R
6813860 [hqzizania] Fill the docs for DataFrame API in SparkR
857220f [hqzizania] remove the pairRDD docs from being built as a part of roxygen
c045d64 [hqzizania] remove the RDD docs from being built as a part of roxygen

(cherry picked from commit 008a60d)
Signed-off-by: Shivaram Venkataraman <shivaram@cs.berkeley.edu>
@asfgit asfgit closed this in 008a60d May 8, 2015
jeanlyn pushed a commit to jeanlyn/spark that referenced this pull request May 28, 2015
@hqzizania @jeanlyn
[SPARK-6824] Fill the docs for DataFrame API in SparkR
73e633c 
This patch also removes the RDD docs from being built as a part of roxygen just by the method to delete
" ' '" of " \#' ".

Author: hqzizania <qian.huang@intel.com>
Author: qhuang <qian.huang@intel.com>

Closes apache#5969 from hqzizania/R1 and squashes the following commits:

6d27696 [qhuang] fixes in NAMESPACE
eb4b095 [qhuang] remove more docs
6394579 [qhuang] remove RDD docs in generics.R
6813860 [hqzizania] Fill the docs for DataFrame API in SparkR
857220f [hqzizania] remove the pairRDD docs from being built as a part of roxygen
c045d64 [hqzizania] remove the RDD docs from being built as a part of roxygen
jeanlyn pushed a commit to jeanlyn/spark that referenced this pull request Jun 12, 2015
@hqzizania @jeanlyn
[SPARK-6824] Fill the docs for DataFrame API in SparkR
6ee32b5 
This patch also removes the RDD docs from being built as a part of roxygen just by the method to delete
" ' '" of " \#' ".

Author: hqzizania <qian.huang@intel.com>
Author: qhuang <qian.huang@intel.com>

Closes apache#5969 from hqzizania/R1 and squashes the following commits:

6d27696 [qhuang] fixes in NAMESPACE
eb4b095 [qhuang] remove more docs
6394579 [qhuang] remove RDD docs in generics.R
6813860 [hqzizania] Fill the docs for DataFrame API in SparkR
857220f [hqzizania] remove the pairRDD docs from being built as a part of roxygen
c045d64 [hqzizania] remove the RDD docs from being built as a part of roxygen
nemccarthy pushed a commit to nemccarthy/spark that referenced this pull request Jun 19, 2015
@hqzizania @nemccarthy
[SPARK-6824] Fill the docs for DataFrame API in SparkR
9d35d43 
This patch also removes the RDD docs from being built as a part of roxygen just by the method to delete
" ' '" of " \#' ".

Author: hqzizania <qian.huang@intel.com>
Author: qhuang <qian.huang@intel.com>

Closes apache#5969 from hqzizania/R1 and squashes the following commits:

6d27696 [qhuang] fixes in NAMESPACE
eb4b095 [qhuang] remove more docs
6394579 [qhuang] remove RDD docs in generics.R
6813860 [hqzizania] Fill the docs for DataFrame API in SparkR
857220f [hqzizania] remove the pairRDD docs from being built as a part of roxygen
c045d64 [hqzizania] remove the RDD docs from being built as a part of roxygen
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@hqzizania @AmplabJenkins @SparkQA @shivaram