Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[SPARK-3001][MLLIB] Improve Spearman's correlation #1917

Closed
wants to merge 5 commits into from
Closed

[SPARK-3001][MLLIB] Improve Spearman's correlation #1917

wants to merge 5 commits into from

Conversation

mengxr
Copy link
Contributor

@mengxr mengxr commented Aug 13, 2014

The current implementation requires sorting individual columns, which could be done with a global sort.

result on a 32-node cluster:

m n prev this
1000000 50 55s 9s
10000000 50 97s 76s
1000000 100 119s 15s

@mengxr
Copy link
Contributor Author

mengxr commented Aug 13, 2014

Jenkins, test this please.

@SparkQA
Copy link

SparkQA commented Aug 13, 2014

QA tests have started for PR 1917. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18427/consoleFull

@SparkQA
Copy link

SparkQA commented Aug 13, 2014

QA results for PR 1917:
- This patch FAILED unit tests.
- This patch merges cleanly
- This patch adds no public classes

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18427/consoleFull

@mengxr
Copy link
Contributor Author

mengxr commented Aug 13, 2014

Jenkins, retest this please.

@SparkQA
Copy link

SparkQA commented Aug 13, 2014

QA tests have started for PR 1917. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18432/consoleFull

@SparkQA
Copy link

SparkQA commented Aug 13, 2014

QA results for PR 1917:
- This patch FAILED unit tests.
- This patch merges cleanly
- This patch adds no public classes

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18432/consoleFull

@SparkQA
Copy link

SparkQA commented Aug 13, 2014

QA tests have started for PR 1917. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18446/consoleFull

@SparkQA
Copy link

SparkQA commented Aug 13, 2014

QA results for PR 1917:
- This patch PASSES unit tests.
- This patch merges cleanly
- This patch adds no public classes

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18446/consoleFull

@mengxr
Copy link
Contributor Author

mengxr commented Aug 14, 2014

Jenkins, test this please.

@SparkQA
Copy link

SparkQA commented Aug 14, 2014

QA tests have started for PR 1917. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18520/consoleFull

@SparkQA
Copy link

SparkQA commented Aug 14, 2014

QA results for PR 1917:
- This patch FAILED unit tests.
- This patch merges cleanly
- This patch adds no public classes

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18520/consoleFull

@mengxr
Copy link
Contributor Author

mengxr commented Aug 14, 2014

Jenkins, retest this please.

@SparkQA
Copy link

SparkQA commented Aug 14, 2014

QA tests have started for PR 1917. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18528/consoleFull

@SparkQA
Copy link

SparkQA commented Aug 14, 2014

QA results for PR 1917:
- This patch PASSES unit tests.
- This patch merges cleanly
- This patch adds no public classes

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18528/consoleFull

@SparkQA
Copy link

SparkQA commented Aug 14, 2014

QA tests have started for PR 1917. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18546/consoleFull

@SparkQA
Copy link

SparkQA commented Aug 14, 2014

QA results for PR 1917:
- This patch PASSES unit tests.
- This patch merges cleanly
- This patch adds no public classes

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18546/consoleFull

@SparkQA
Copy link

SparkQA commented Aug 14, 2014

QA tests have started for PR 1917. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18559/consoleFull

@SparkQA
Copy link

SparkQA commented Aug 14, 2014

QA results for PR 1917:
- This patch PASSES unit tests.
- This patch merges cleanly
- This patch adds no public classes

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18559/consoleFull

@mengxr
Copy link
Contributor Author

mengxr commented Aug 16, 2014

Merged into master and branch-1.1.

asfgit pushed a commit that referenced this pull request Aug 16, 2014
@mengxr
[SPARK-3001][MLLIB] Improve Spearman's correlation
ce06d7f 
The current implementation requires sorting individual columns, which could be done with a global sort.

result on a 32-node cluster:

m | n | prev | this
---|---|-------|-----
1000000 | 50 | 55s | 9s
10000000 | 50 | 97s | 76s
1000000 | 100  | 119s | 15s

Author: Xiangrui Meng <meng@databricks.com>

Closes #1917 from mengxr/spearman and squashes the following commits:

4d5d262 [Xiangrui Meng] remove unused import
85c48de [Xiangrui Meng] minor updates
a048d0c [Xiangrui Meng] remove cache and set a limit to cachedIds
b98bb18 [Xiangrui Meng] add comments
0846e07 [Xiangrui Meng] first version

(cherry picked from commit 2e069ca)
Signed-off-by: Xiangrui Meng <meng@databricks.com>
@asfgit asfgit closed this in 2e069ca Aug 16, 2014
xiliu82 pushed a commit to xiliu82/spark that referenced this pull request Sep 4, 2014
@mengxr @conviva-zz
[SPARK-3001][MLLIB] Improve Spearman's correlation
4df7252 
The current implementation requires sorting individual columns, which could be done with a global sort.

result on a 32-node cluster:

m | n | prev | this
---|---|-------|-----
1000000 | 50 | 55s | 9s
10000000 | 50 | 97s | 76s
1000000 | 100  | 119s | 15s

Author: Xiangrui Meng <meng@databricks.com>

Closes apache#1917 from mengxr/spearman and squashes the following commits:

4d5d262 [Xiangrui Meng] remove unused import
85c48de [Xiangrui Meng] minor updates
a048d0c [Xiangrui Meng] remove cache and set a limit to cachedIds
b98bb18 [Xiangrui Meng] add comments
0846e07 [Xiangrui Meng] first version
snmvaughan pushed a commit to snmvaughan/spark that referenced this pull request Mar 26, 2024
@kazuyukitanimura
rdar://123796322 Update UC-Spark-Authz Plugin to 0.1.5 (apache#1917)
e4fa93d 
This PR updates UC-Spark-Authz plugin to 0.1.5

Change list: apple-cloud-services/uc-spark-authz@6e9000b...5998e71
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@mengxr @SparkQA