Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[SPARK-3491] [MLlib] [PySpark] use pickle to serialize data in MLlib #2378

Closed
wants to merge 41 commits into from
Closed

[SPARK-3491] [MLlib] [PySpark] use pickle to serialize data in MLlib #2378

wants to merge 41 commits into from

Commits on Sep 11, 2014

  1. support unpickle array.array for Python 2.6

    @davies
    davies committed Sep 11, 2014
    Configuration menu
    Copy the full SHA
    60e4e2f View commit details
    Browse the repository at this point in the history

Commits on Sep 13, 2014

  1. cleanup debugging code

    @davies
    davies committed Sep 13, 2014
    Configuration menu
    Copy the full SHA
    c77c87b View commit details
    Browse the repository at this point in the history

  2. Merge branch 'master' into pickle

    @davies
    davies committed Sep 13, 2014
    Configuration menu
    Copy the full SHA
    3908f5c View commit details
    Browse the repository at this point in the history

  3. enable tests about array

    @davies
    davies committed Sep 13, 2014
    Configuration menu
    Copy the full SHA
    f44f771 View commit details
    Browse the repository at this point in the history

  4. use pickle to serialize data for mllib/recommendation

    @davies
    davies committed Sep 13, 2014
    Configuration menu
    Copy the full SHA
    b30ef35 View commit details
    Browse the repository at this point in the history

  5. use new protocol in mllib/stat

    @davies
    davies committed Sep 13, 2014
    Configuration menu
    Copy the full SHA
    52d1350 View commit details
    Browse the repository at this point in the history

  6. refactor clustering

    @davies
    davies committed Sep 13, 2014
    Configuration menu
    Copy the full SHA
    f1544c4 View commit details
    Browse the repository at this point in the history

  7. random

    @davies
    davies committed Sep 13, 2014
    Configuration menu
    Copy the full SHA
    aa2287e View commit details
    Browse the repository at this point in the history

  8. Merge branch 'pickle' into pickle_mllib 

    Conflicts:
	python/pyspark/context.py
    @davies
    davies committed Sep 13, 2014
    Configuration menu
    Copy the full SHA
    8fe166a View commit details
    Browse the repository at this point in the history

Commits on Sep 14, 2014

  1. mllib/tree

    @davies
    davies committed Sep 14, 2014
    Configuration menu
    Copy the full SHA
    cccb8b1 View commit details
    Browse the repository at this point in the history

  2. mllib/util

    @davies
    davies committed Sep 14, 2014
    Configuration menu
    Copy the full SHA
    d9f691f View commit details
    Browse the repository at this point in the history

  3. mllib/regression

    @davies
    davies committed Sep 14, 2014
    Configuration menu
    Copy the full SHA
    f2a0856 View commit details
    Browse the repository at this point in the history

  4. classification

    @davies
    davies committed Sep 14, 2014
    Configuration menu
    Copy the full SHA
    c383544 View commit details
    Browse the repository at this point in the history

  5. fix tests

    @davies
    davies committed Sep 14, 2014
    Configuration menu
    Copy the full SHA
    6d26b03 View commit details
    Browse the repository at this point in the history

  6. remove muanlly serialization

    @davies
    davies committed Sep 14, 2014
    Configuration menu
    Copy the full SHA
    4d7963e View commit details
    Browse the repository at this point in the history

  7. Merge branch 'master' into pickle_mllib 

    Conflicts:
	python/pyspark/mllib/_common.py
    @davies
    davies committed Sep 14, 2014
    Configuration menu
    Copy the full SHA
    84c721d View commit details
    Browse the repository at this point in the history

  8. remove _common.py

    @davies
    davies committed Sep 14, 2014
    Configuration menu
    Copy the full SHA
    b02e34f View commit details
    Browse the repository at this point in the history

  9. remove outdated tests

    @davies
    davies committed Sep 14, 2014
    Configuration menu
    Copy the full SHA
    0ee1525 View commit details
    Browse the repository at this point in the history

Commits on Sep 15, 2014

  1. cleanup _common.py

    @davies
    davies committed Sep 15, 2014
    Configuration menu
    Copy the full SHA
    722dd96 View commit details
    Browse the repository at this point in the history

Commits on Sep 16, 2014

  1. Merge branch 'master' into pickle_mllib 

    Conflicts:
	python/pyspark/context.py
	python/pyspark/mllib/tree.py
    @davies
    davies committed Sep 16, 2014
    Configuration menu
    Copy the full SHA
    f3506c5 View commit details
    Browse the repository at this point in the history

  2. memorize the module and class name during pickleing

    @davies
    davies committed Sep 16, 2014
    Configuration menu
    Copy the full SHA
    df19464 View commit details
    Browse the repository at this point in the history

  3. choose batch size automatically

    @davies
    davies committed Sep 16, 2014
    Configuration menu
    Copy the full SHA
    46a501e View commit details
    Browse the repository at this point in the history

  4. rafactor, address comments

    @davies
    davies committed Sep 16, 2014
    Configuration menu
    Copy the full SHA
    88034f0 View commit details
    Browse the repository at this point in the history

  5. fix style

    @davies
    davies committed Sep 16, 2014
    Configuration menu
    Copy the full SHA
    9dcfb63 View commit details
    Browse the repository at this point in the history

  6. fix tests

    @davies
    davies committed Sep 16, 2014
    Configuration menu
    Copy the full SHA
    708dc02 View commit details
    Browse the repository at this point in the history

  7. refactor

    @davies
    davies committed Sep 16, 2014
    Configuration menu
    Copy the full SHA
    e1d1bfc View commit details
    Browse the repository at this point in the history

  8. speed up pickling array in Python 2.7

    @davies
    davies committed Sep 16, 2014
    Configuration menu
    Copy the full SHA
    44736d7 View commit details
    Browse the repository at this point in the history

  9. fix autobatchedpickler

    @davies
    davies committed Sep 16, 2014
    Configuration menu
    Copy the full SHA
    154d141 View commit details
    Browse the repository at this point in the history

  10. Merge commit '154d141' into pickle_mllib

    @davies
    davies committed Sep 16, 2014
    Configuration menu
    Copy the full SHA
    df625c7 View commit details
    Browse the repository at this point in the history

  11. fix pickle array in python2.7

    @davies
    davies committed Sep 16, 2014
    Configuration menu
    Copy the full SHA
    a379a81 View commit details
    Browse the repository at this point in the history

  12. fix cache

    @davies
    davies committed Sep 16, 2014
    Configuration menu
    Copy the full SHA
    44e0551 View commit details
    Browse the repository at this point in the history

  13. test size of serialized Rating

    @davies
    davies committed Sep 16, 2014
    Configuration menu
    Copy the full SHA
    9ceff73 View commit details
    Browse the repository at this point in the history

  14. fix tests

    @davies
    davies committed Sep 16, 2014
    Configuration menu
    Copy the full SHA
    a2cc855 View commit details
    Browse the repository at this point in the history

  15. address comments

    @davies
    davies committed Sep 16, 2014
    Configuration menu
    Copy the full SHA
    1fccf1a View commit details
    Browse the repository at this point in the history

  16. cleanup

    @davies
    davies committed Sep 16, 2014
    Configuration menu
    Copy the full SHA
    2511e76 View commit details
    Browse the repository at this point in the history

Commits on Sep 17, 2014

  1. refactor Picklers

    @davies
    davies committed Sep 17, 2014
    Configuration menu
    Copy the full SHA
    19d0967 View commit details
    Browse the repository at this point in the history

  2. fix cache of rdd, refactor

    @davies
    davies committed Sep 17, 2014
    Configuration menu
    Copy the full SHA
    e431377 View commit details
    Browse the repository at this point in the history

Commits on Sep 18, 2014

  1. address comments

    @davies
    davies committed Sep 18, 2014
    Configuration menu
    Copy the full SHA
    bd738ab View commit details
    Browse the repository at this point in the history

  2. add more type check and conversion for user_product

    @davies
    davies committed Sep 18, 2014
    Configuration menu
    Copy the full SHA
    032cd62 View commit details
    Browse the repository at this point in the history

  3. fix equal of matrix

    @davies
    davies committed Sep 18, 2014
    Configuration menu
    Copy the full SHA
    810f97f View commit details
    Browse the repository at this point in the history

Commits on Sep 19, 2014

  1. Merge branch 'master' of github.com:apache/spark into pickle_mllib 

    Conflicts:
	mllib/src/main/scala/org/apache/spark/mllib/linalg/Matrices.scala
    @davies
    davies committed Sep 19, 2014
    Configuration menu
    Copy the full SHA
    dffbba2 View commit details
    Browse the repository at this point in the history