Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[SPARK-1594][MLLIB] Cleaning up MLlib APIs and guide #524

Closed
wants to merge 21 commits into from
Closed

[SPARK-1594][MLLIB] Cleaning up MLlib APIs and guide #524

wants to merge 21 commits into from

Conversation

mengxr
Copy link
Contributor

@mengxr mengxr commented Apr 24, 2014

Final pass before the v1.0 release.

  • Remove VectorRDDs
  • Move BinaryClassificationMetrics from evaluation.binary to evaluation
  • Change default value of addIntercept to false and allow to add intercept in Ridge and Lasso.
  • Clean DecisionTree package doc and test suite.
  • Mark model constructors private[spark]
  • Rename loadLibSVMData to loadLibSVMFile and hide LabelParser from users.
  • Add saveAsLibSVMFile.
  • Add appendBias to MLUtils.

@mengxr mengxr changed the title [WIP][SPARK-1594] Cleaning up MLlib APIs and guide [WIP][SPARK-1594][MLLIB] Cleaning up MLlib APIs and guide Apr 24, 2014
@AmplabJenkins
Copy link

Merged build triggered.

@AmplabJenkins
Copy link

Merged build started.

@AmplabJenkins
Copy link

Merged build finished.

@AmplabJenkins
Copy link

Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/14429/

@mateiz
Copy link
Contributor

mateiz commented Apr 24, 2014

Hey Xiangrui, can you also add Scaladocs on the main methods? Otherwise it looks weird to have an experimental method. Really these were meant to give people an easy way to run these algorithms from the command line, but very few are probably using it because we didn't document it, so we could also move these methods into examples instead (e.g. in an org.apache.spark.examples.mllib package in the examples project). They are also useful for perf testing during development.

@mengxr
Copy link
Contributor Author

mengxr commented Apr 25, 2014

@mateiz My concern is that whether we want to encourage users to use those main methods if we know they are definitely going to change in the next release. Now the main methods take arguments in a predefined order. It is hard to remember and there is no help message.

I'm testing scopt for parsing options: https://github.com/mengxr/MLlib-QA/blob/master/src/main/scala/mllib/qa/BinaryClassification.scala

The question is whether we want to make the change now or in the next release.

@AmplabJenkins
Copy link

Merged build triggered.

@AmplabJenkins
Copy link

Merged build started.

@AmplabJenkins
Copy link

Merged build finished.

@AmplabJenkins
Copy link

Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/14563/

@AmplabJenkins
Copy link

Merged build triggered.

@AmplabJenkins
Copy link

Merged build started.

@AmplabJenkins
Copy link

Merged build finished.

@AmplabJenkins
Copy link

Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/14564/

@mengxr mengxr changed the title [WIP][SPARK-1594][MLLIB] Cleaning up MLlib APIs and guide [SPARK-1594][MLLIB] Cleaning up MLlib APIs and guide Apr 29, 2014
@AmplabJenkins
Copy link

Merged build triggered.

@AmplabJenkins
Copy link

Merged build started.

@AmplabJenkins
Copy link

Merged build finished.

@AmplabJenkins
Copy link

Merged build triggered.

@AmplabJenkins
Copy link

Merged build started.

@AmplabJenkins
Copy link

Merged build finished. All automated tests passed.

@AmplabJenkins
Copy link

All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/14594/

@AmplabJenkins
Copy link

Merged build triggered.

@AmplabJenkins
Copy link

Merged build started.

@AmplabJenkins
Copy link

Merged build finished. All automated tests passed.

@AmplabJenkins
Copy link

All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/14671/

@AmplabJenkins
Copy link

Merged build triggered.

@AmplabJenkins
Copy link

Merged build started.

@AmplabJenkins
Copy link

Merged build finished. All automated tests passed.

@AmplabJenkins
Copy link

All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/14684/

@mengxr
Copy link
Contributor Author

mengxr commented May 6, 2014

@mateiz Could you make another scan and see whether this is okay to merge? Thanks!

@mateiz
Copy link
Contributor

mateiz commented May 6, 2014

Thanks, it looks good. I've merged this in.

@asfgit asfgit closed this in 98750a7 May 6, 2014
asfgit pushed a commit that referenced this pull request May 6, 2014
@mengxr @mateiz
[SPARK-1594][MLLIB] Cleaning up MLlib APIs and guide
32c960a 
Final pass before the v1.0 release.

* Remove `VectorRDDs`
* Move `BinaryClassificationMetrics` from `evaluation.binary` to `evaluation`
* Change default value of `addIntercept` to false and allow to add intercept in Ridge and Lasso.
* Clean `DecisionTree` package doc and test suite.
* Mark model constructors `private[spark]`
* Rename `loadLibSVMData` to `loadLibSVMFile` and hide `LabelParser` from users.
* Add `saveAsLibSVMFile`.
* Add `appendBias` to `MLUtils`.

Author: Xiangrui Meng <meng@databricks.com>

Closes #524 from mengxr/mllib-cleaning and squashes the following commits:

295dc8b [Xiangrui Meng] update loadLibSVMFile doc
1977ac1 [Xiangrui Meng] fix doc of appendBias
649fcf0 [Xiangrui Meng] rename loadLibSVMData to loadLibSVMFile; hide LabelParser from user APIs
54b812c [Xiangrui Meng] add appendBias
a71e7d0 [Xiangrui Meng] add saveAsLibSVMFile
d976295 [Xiangrui Meng] Merge branch 'master' into mllib-cleaning
b7e5cec [Xiangrui Meng] remove some experimental annotations and make model constructors private[mllib]
9b02b93 [Xiangrui Meng] minor code style update
a593ddc [Xiangrui Meng] fix python tests
fc28c18 [Xiangrui Meng] mark more classes experimental
f6cbbff [Xiangrui Meng] fix Java tests
0af70b0 [Xiangrui Meng] minor
6e139ef [Xiangrui Meng] Merge branch 'master' into mllib-cleaning
94e6dce [Xiangrui Meng] move BinaryLabelCounter and BinaryConfusionMatrixImpl to evaluation.binary
df34907 [Xiangrui Meng] clean DecisionTreeSuite to use LocalSparkContext
c81807f [Xiangrui Meng] set the default value of AddIntercept to false
03389c0 [Xiangrui Meng] allow to add intercept in Ridge and Lasso
c66c56f [Xiangrui Meng] move tree md to package object doc
a2695df [Xiangrui Meng] update guide for BinaryClassificationMetrics
9194f4c [Xiangrui Meng] move BinaryClassificationMetrics one level up
1c1a0e3 [Xiangrui Meng] remove VectorRDDs because it only contains one function that is not necessary for us to maintain

(cherry picked from commit 98750a7)
Signed-off-by: Matei Zaharia <matei@databricks.com>
@mengxr mengxr deleted the mllib-cleaning branch May 7, 2014 00:06
pwendell pushed a commit to pwendell/spark that referenced this pull request May 12, 2014
@rxin
Merge pull request apache#524 from rxin/doc
ac712e4 
Added spark.shuffle.file.buffer.kb to configuration doc.

Author: Reynold Xin <rxin@apache.org>

== Merge branch commits ==

commit 0eea1d761ff772ff89be234e1e28035d54e5a7de
Author: Reynold Xin <rxin@apache.org>
Date:   Wed Jan 29 14:40:48 2014 -0800

    Added spark.shuffle.file.buffer.kb to configuration doc.
pdeyhim pushed a commit to pdeyhim/spark-1 that referenced this pull request Jun 25, 2014
@mengxr @pdeyhim
[SPARK-1594][MLLIB] Cleaning up MLlib APIs and guide
059f805 
Final pass before the v1.0 release.

* Remove `VectorRDDs`
* Move `BinaryClassificationMetrics` from `evaluation.binary` to `evaluation`
* Change default value of `addIntercept` to false and allow to add intercept in Ridge and Lasso.
* Clean `DecisionTree` package doc and test suite.
* Mark model constructors `private[spark]`
* Rename `loadLibSVMData` to `loadLibSVMFile` and hide `LabelParser` from users.
* Add `saveAsLibSVMFile`.
* Add `appendBias` to `MLUtils`.

Author: Xiangrui Meng <meng@databricks.com>

Closes apache#524 from mengxr/mllib-cleaning and squashes the following commits:

295dc8b [Xiangrui Meng] update loadLibSVMFile doc
1977ac1 [Xiangrui Meng] fix doc of appendBias
649fcf0 [Xiangrui Meng] rename loadLibSVMData to loadLibSVMFile; hide LabelParser from user APIs
54b812c [Xiangrui Meng] add appendBias
a71e7d0 [Xiangrui Meng] add saveAsLibSVMFile
d976295 [Xiangrui Meng] Merge branch 'master' into mllib-cleaning
b7e5cec [Xiangrui Meng] remove some experimental annotations and make model constructors private[mllib]
9b02b93 [Xiangrui Meng] minor code style update
a593ddc [Xiangrui Meng] fix python tests
fc28c18 [Xiangrui Meng] mark more classes experimental
f6cbbff [Xiangrui Meng] fix Java tests
0af70b0 [Xiangrui Meng] minor
6e139ef [Xiangrui Meng] Merge branch 'master' into mllib-cleaning
94e6dce [Xiangrui Meng] move BinaryLabelCounter and BinaryConfusionMatrixImpl to evaluation.binary
df34907 [Xiangrui Meng] clean DecisionTreeSuite to use LocalSparkContext
c81807f [Xiangrui Meng] set the default value of AddIntercept to false
03389c0 [Xiangrui Meng] allow to add intercept in Ridge and Lasso
c66c56f [Xiangrui Meng] move tree md to package object doc
a2695df [Xiangrui Meng] update guide for BinaryClassificationMetrics
9194f4c [Xiangrui Meng] move BinaryClassificationMetrics one level up
1c1a0e3 [Xiangrui Meng] remove VectorRDDs because it only contains one function that is not necessary for us to maintain
andrewor14 pushed a commit to andrewor14/spark that referenced this pull request Jan 8, 2015
@rxin @pwendell
Merge pull request apache#524 from rxin/doc
a414071 
Added spark.shuffle.file.buffer.kb to configuration doc.

Author: Reynold Xin <rxin@apache.org>

== Merge branch commits ==

commit 0eea1d761ff772ff89be234e1e28035d54e5a7de
Author: Reynold Xin <rxin@apache.org>
Date:   Wed Jan 29 14:40:48 2014 -0800

    Added spark.shuffle.file.buffer.kb to configuration doc.
(cherry picked from commit ac712e4)

Signed-off-by: Patrick Wendell <pwendell@gmail.com>
yifeih pushed a commit to yifeih/spark that referenced this pull request May 8, 2019
@mccheah @bulldozer-bot
[SPARK-25299] Local shuffle implementation of the shuffle writer API (a…
bc40da2 
…pache#524)

Implements the shuffle writer API by writing shuffle files to local disk and using the index block resolver to commit data and write index files.

The logic in `BypassMergeSortShuffleWriter` has been refactored to use the base implementation of the plugin instead.

APIs have been slightly renamed to clarify semantics after considering nuances in how these are to be implemented by other developers.

Follow-up commits are to come for `SortShuffleWriter` and `UnsafeShuffleWriter`.

Ported from bloomberg#6, credits to @ifilonenko.
squito pushed a commit to squito/spark that referenced this pull request Jul 18, 2019
@mccheah
[SPARK-25299] Local shuffle implementation of the shuffle writer API (a…
857552a 
…pache#524)

Implements the shuffle writer API by writing shuffle files to local disk and using the index block resolver to commit data and write index files.

The logic in `BypassMergeSortShuffleWriter` has been refactored to use the base implementation of the plugin instead.

APIs have been slightly renamed to clarify semantics after considering nuances in how these are to be implemented by other developers.

Follow-up commits are to come for `SortShuffleWriter` and `UnsafeShuffleWriter`.

Ported from bloomberg#6, credits to @ifilonenko.
bzhaoopenstack pushed a commit to bzhaoopenstack/spark that referenced this pull request Sep 11, 2019
@kiwik
Enable bosh cpi test against openstack stein periodic job (apache#524)
afb9486 
Run bosh cpi test against openstack stable/stein
in periodic pipeline.

Closes: theopenlab/openlab#265
arjunshroff pushed a commit to arjunshroff/spark that referenced this pull request Nov 24, 2020
@ekrivokonmapr
MapR [SPARK-567] Update Hive version for Spark-2.4.3 (apache#524)
784332e
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@mengxr @AmplabJenkins @mateiz