Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[SPARK-1212] Adding sparse data support and update KMeans #117

Closed
wants to merge 26 commits into from
Closed

[SPARK-1212] Adding sparse data support and update KMeans #117

wants to merge 26 commits into from

Commits on Mar 10, 2014

  1. add dense/sparse vector data models and conversions to/from breeze ve… 

    …ctors

use breeze to implement KMeans in order to support both dense and sparse data
    @mengxr
    mengxr committed Mar 10, 2014
    Configuration menu
    Copy the full SHA
    07ffaf2 View commit details
    Browse the repository at this point in the history

  2. minor style update

    @mengxr
    mengxr committed Mar 10, 2014
    Configuration menu
    Copy the full SHA
    4e7d5ca View commit details
    Browse the repository at this point in the history

Commits on Mar 11, 2014

  1. add fastSquaredDistance for KMeans

    @mengxr
    mengxr committed Mar 11, 2014
    Configuration menu
    Copy the full SHA
    ab74f67 View commit details
    Browse the repository at this point in the history

  2. add BreezeVectorWithSquaredNorm case class

    @mengxr
    mengxr committed Mar 11, 2014
    Configuration menu
    Copy the full SHA
    f355411 View commit details
    Browse the repository at this point in the history

  3. update KMeans to use fastSquaredDistance

    @mengxr
    mengxr committed Mar 11, 2014
    Configuration menu
    Copy the full SHA
    0ff8046 View commit details
    Browse the repository at this point in the history

Commits on Mar 12, 2014

  1. tuned the KMeans code: changed some for loops to while, use view to a… 

    …void copying arrays

added some log messages
    @mengxr
    mengxr committed Mar 12, 2014
    Configuration menu
    Copy the full SHA
    87bc755 View commit details
    Browse the repository at this point in the history

  2. update NOTICE

    @mengxr
    mengxr committed Mar 12, 2014
    Configuration menu
    Copy the full SHA
    0107e19 View commit details
    Browse the repository at this point in the history

  3. add doc to BreezeVectorWithSquaredNorm

    @mengxr
    mengxr committed Mar 12, 2014
    Configuration menu
    Copy the full SHA
    3ed1a24 View commit details
    Browse the repository at this point in the history

  4. Merge branch 'fast-dist' into sparse-kmeans

    @mengxr
    mengxr committed Mar 12, 2014
    Configuration menu
    Copy the full SHA
    a4ace73 View commit details
    Browse the repository at this point in the history

  5. line feed at the end

    @mengxr
    mengxr committed Mar 12, 2014
    Configuration menu
    Copy the full SHA
    42b4e50 View commit details
    Browse the repository at this point in the history

  6. add predict(RDD[Vector]) to KMeansModel 

    add a test for two clusteres
    @mengxr
    mengxr committed Mar 12, 2014
    Configuration menu
    Copy the full SHA
    d6e6c07 View commit details
    Browse the repository at this point in the history

  7. Merge branch 'master' into sparse-kmeans

    @mengxr
    mengxr committed Mar 12, 2014
    Configuration menu
    Copy the full SHA
    42512f2 View commit details
    Browse the repository at this point in the history

  8. fix a bug in filtering finished runs

    @mengxr
    mengxr committed Mar 12, 2014
    Configuration menu
    Copy the full SHA
    6f5cdde View commit details
    Browse the repository at this point in the history

Commits on Mar 18, 2014

  1. change Mahout to breeze in doc 

    use a simple lower bound to avoid unnecessary distance computation
    @mengxr
    mengxr committed Mar 18, 2014
    Configuration menu
    Copy the full SHA
    07c3cf2 View commit details
    Browse the repository at this point in the history

Commits on Mar 19, 2014

  1. update breeze version to 0.7

    @mengxr
    mengxr committed Mar 19, 2014
    Configuration menu
    Copy the full SHA
    27858e4 View commit details
    Browse the repository at this point in the history

Commits on Mar 20, 2014

  1. make Vectors.sparse Java friendly 

    rename VectorSuite to VectorsSuite
    @mengxr
    mengxr committed Mar 20, 2014
    Configuration menu
    Copy the full SHA
    712cb88 View commit details
    Browse the repository at this point in the history

  2. clean up code for distance computation

    @mengxr
    mengxr committed Mar 20, 2014
    Configuration menu
    Copy the full SHA
    72bde33 View commit details
    Browse the repository at this point in the history

  3. remove examples/JavaKMeans.java, which is replaced by mllib/examples/… 

    …JavaKMeans.java
    @mengxr
    mengxr committed Mar 20, 2014
    Configuration menu
    Copy the full SHA
    e69b10c View commit details
    Browse the repository at this point in the history

Commits on Mar 21, 2014

  1. add toArray to Vector 

    add Java friendly methods to Vectors

remove RDD[Array[Double]] support from KMeans

update Java KMeans API
    @mengxr
    mengxr committed Mar 21, 2014
    Configuration menu
    Copy the full SHA
    b28ba2f View commit details
    Browse the repository at this point in the history

  2. add VectorRDDs with a converter from RDD[Array[Double]]

    @mengxr
    mengxr committed Mar 21, 2014
    Configuration menu
    Copy the full SHA
    238ba34 View commit details
    Browse the repository at this point in the history

Commits on Mar 22, 2014

  1. update Java friendly methods in Vectors

    @mengxr
    mengxr committed Mar 22, 2014
    Configuration menu
    Copy the full SHA
    226d2cd View commit details
    Browse the repository at this point in the history

  2. optimize SparseVector.toArray

    @mengxr
    mengxr committed Mar 22, 2014
    Configuration menu
    Copy the full SHA
    9bb1b31 View commit details
    Browse the repository at this point in the history

  3. remove dependency on commons-math3 and compute EPSILON directly

    @mengxr
    mengxr committed Mar 22, 2014
    Configuration menu
    Copy the full SHA
    1da1033 View commit details
    Browse the repository at this point in the history

  4. move ArrayRDDs to mllib.rdd

    @mengxr
    mengxr committed Mar 22, 2014
    Configuration menu
    Copy the full SHA
    67abe31 View commit details
    Browse the repository at this point in the history

Commits on Mar 23, 2014

  1. update NOTICE

    @mengxr
    mengxr committed Mar 23, 2014
    Configuration menu
    Copy the full SHA
    5eda0de View commit details
    Browse the repository at this point in the history

  2. fix SparseVector.toArray

    @mengxr
    mengxr committed Mar 23, 2014
    Configuration menu
    Copy the full SHA
    67b368d View commit details
    Browse the repository at this point in the history