Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[ML-133][Correlation] Add Correlation algorithm #127

Merged
merged 21 commits into from
Oct 21, 2021
Merged

[ML-133][Correlation] Add Correlation algorithm #127

merged 21 commits into from
Oct 21, 2021

Conversation

minmingzhu
Copy link
Collaborator

  1. enable correlation in OAP MLlib
  2. add Correlation example
  3. enable spark 3.1.1 and spark 3.0.0

Signed-off-by: minmingzhu minmingzhu@intel.com

What changes were proposed in this pull request?

(Please fill in changes proposed in this fix)

1. enable correlation on OAP MLlib
d52fd51 
2. add Correlation example
3. enable spark 3.1.1 and spark 3.0.0

Signed-off-by: minmingzhu <minmingzhu@intel.com>
@github-actions
Copy link

Thanks for opening a pull request!

Could you open an issue for this pull request on Github Issues?
https://github.com/oap-project/oap-mllib/issues

Then could you also rename pull request title and commit log in the following format?

[ML-${ISSUES_ID}] ${detailed message}

See also:

xwu99
xwu99 reviewed Aug 31, 2021
View reviewed changes
examples/correlation/pom.xml Outdated Show resolved Hide resolved
xwu99
xwu99 reviewed Aug 31, 2021
View reviewed changes
examples/correlation/run.sh Outdated
DATA_FILE=$HDFS_ROOT/data/sample_linear_regression_data.txt

APP_JAR=target/oap-mllib-examples-$OAP_MLLIB_VERSION.jar
APP_CLASS=org.apache.spark.examples.ml.LinearRegressionExample
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
APP_CLASS=org.apache.spark.examples.ml.LinearRegressionExample
APP_CLASS=org.apache.spark.examples.ml.CorrelationExample

xwu99
xwu99 reviewed Aug 31, 2021
View reviewed changes
examples/correlation/run.sh Outdated Show resolved Hide resolved
xwu99
xwu99 reviewed Aug 31, 2021
View reviewed changes
examples/correlation/run.sh Outdated Show resolved Hide resolved
minmingzhu added 6 commits September 2, 2021 18:12
@minmingzhu
1. resolve comments
757561a 
Signed-off-by: minmingzhu <minming.zhu@intel.com>
@minmingzhu
1. fix oap matrix result error.
c933bdd 
Signed-off-by: minmingzhu <minming.zhu@intel.com>
@minmingzhu
1. add jni file
cebea3f 
Signed-off-by: minmingzhu <minming.zhu@intel.com>
@minmingzhu
1. add test code
d5c1e24 
Signed-off-by: minmingzhu <minming.zhu@intel.com>
@minmingzhu
1. vanilla rdd add cache
7ff6dd4 
Signed-off-by: minmingzhu <minming.zhu@intel.com>
@minmingzhu
remove rdd cache
01e1e8c
@xwu99 xwu99 changed the title enable Correlation algorithms in OAP MLlib [ML-133] Add Correlation algorithm Sep 10, 2021
@github-actions
Copy link

#133

xwu99
xwu99 requested changes Sep 10, 2021
View reviewed changes
Copy link
Collaborator

@xwu99 xwu99 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pls use IDEA and format-cpp to format the code.

examples/correlation/src/main/scala/org/apache/spark/examples/ml/CorrelationExample.scala Outdated Show resolved Hide resolved
mllib-dal/src/main/native/CorrelationDALImpl.cpp Outdated Show resolved Hide resolved
mllib-dal/src/main/native/CorrelationDALImpl.cpp Outdated Show resolved Hide resolved
mllib-dal/src/main/native/CorrelationDALImpl.cpp Outdated Show resolved Hide resolved
mllib-dal/src/main/native/CorrelationDALImpl.cpp Outdated Show resolved Hide resolved
mllib-dal/src/main/scala/org/apache/spark/ml/util/OneDAL.scala Show resolved Hide resolved
minmingzhu added 2 commits September 10, 2021 13:30
@minmingzhu
1. change algorithmFPType float to double
9a5f2d7 
2. modify test
3. enable spark 3.0.1 and spark 3.0.2
4. remove print

Signed-off-by: minmingzhu <minming.zhu@intel.com>
@minmingzhu
1.remove mean vector
4755d02 
Signed-off-by: minmingzhu <minming.zhu@intel.com>
@xwu99 xwu99 changed the title [ML-133] Add Correlation algorithm [ML-133][Correlation] Add Correlation algorithm Sep 18, 2021
minmingzhu added 4 commits September 18, 2021 10:25
@minmingzhu
1. enable Correlation GPU
dafda4a 
Signed-off-by: minmingzhu <minming.zhu@intel.com>
@minmingzhu
merge
ed51ad8 
Signed-off-by: minmingzhu <minming.zhu@intel.com>
@minmingzhu
1. add GPU test script.
cf4bd58 
Signed-off-by: minmingzhu <minming.zhu@intel.com>
@minmingzhu
1. update correlation
04fbe03 
2. add GPU test

Signed-off-by: minmingzhu <minming.zhu@intel.com>
xwu99
xwu99 requested changes Sep 24, 2021
View reviewed changes
examples/correlation/src/main/scala/org/apache/spark/examples/ml/CorrelationExample.scala Outdated Show resolved Hide resolved
mllib-dal/src/spark-3.0.0/main/scala/org/apache/spark/ml/stat/Correlation.scala Show resolved Hide resolved
mllib-dal/src/main/scala/org/apache/spark/ml/stat/CorrelationDALImpl.scala Show resolved Hide resolved
mllib-dal/src/main/scala/org/apache/spark/ml/stat/CorrelationDALImpl.scala Outdated Show resolved Hide resolved
mllib-dal/src/main/scala/org/apache/spark/ml/stat/CorrelationDALImpl.scala Outdated Show resolved Hide resolved
mllib-dal/src/main/native/CorrelationDALImpl.cpp Show resolved Hide resolved
mllib-dal/src/main/native/CorrelationDALImpl.cpp Outdated Show resolved Hide resolved
mllib-dal/src/main/native/CorrelationDALImpl.cpp Outdated Show resolved Hide resolved
mllib-dal/src/main/native/CorrelationDALImpl.cpp Outdated Show resolved Hide resolved
minmingzhu added 4 commits October 8, 2021 11:05
@minmingzhu
1. resolve comments
268e9df 
Signed-off-by: minmingzhu <minming.zhu@intel.com>
@minmingzhu
1. used ccl::gatherv in OneCCL.h instead of ccl::allgatherv
ad7ac6b 
Signed-off-by: minmingzhu <minming.zhu@intel.com>
@minmingzhu
1.used ml.Matrix for Correlation, reduce a step to mllib.Matrix conve…
bf6c00c 
…rt to ml.Matrix.

Signed-off-by: minmingzhu <minming.zhu@intel.com>
@minmingzhu
1. reformat code style
e1d8b31 
2. make-up code

Signed-off-by: minmingzhu <minming.zhu@intel.com>
@minmingzhu minmingzhu closed this Oct 15, 2021
@minmingzhu minmingzhu deleted the add_correleation branch October 15, 2021 06:40
@minmingzhu minmingzhu restored the add_correleation branch October 15, 2021 08:24
@minmingzhu minmingzhu deleted the add_correleation branch October 15, 2021 08:31
@minmingzhu
check whether data was cached, if no cached, set data to be cached
89846f7 
Signed-off-by: minmingzhu <minming.zhu@intel.com>
@minmingzhu minmingzhu restored the add_correleation branch October 19, 2021 02:37
@minmingzhu minmingzhu reopened this Oct 19, 2021
minmingzhu and others added 3 commits October 19, 2021 13:42
@minmingzhu
update Correlation that check whether data was cached, if no cached, …
c7972e0 
…set data to be cached on spark 3.0.1, 3.0.2 and 3.1.1

Signed-off-by: minmingzhu <minming.zhu@intel.com>
@minmingzhu
1. add Correlation to CI
ee8a105 
Signed-off-by: minmingzhu <minming.zhu@intel.com>
@minmingzhu
Update build-all.sh
9ed9807 
build correlation
@xwu99 xwu99 merged commit 9871ab2 into oap-project:master Oct 21, 2021
@xwu99
Copy link
Collaborator

xwu99 commented Oct 21, 2021

Merged, Thanks a lot for your contribution! @minmingzhu

This was linked to issues Nov 2, 2021
[Correlation] Add Correlation algorithm #133
Closed
[Core] Fix ccl::gather and Add ccl::gatherv #135
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@xwu99 xwu99 xwu99 requested changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[Core] Fix ccl::gather and Add ccl::gatherv [Correlation] Add Correlation algorithm
3 participants
@minmingzhu @xwu99 @zhixingheyi-tian