-
Notifications
You must be signed in to change notification settings - Fork 28.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[SPARK-3491] [MLlib] [PySpark] use pickle to serialize data in MLlib #2378
Closed
Closed
Changes from all commits
Commits
Show all changes
41 commits
Select commit
Hold shift + click to select a range
60e4e2f
support unpickle array.array for Python 2.6
davies c77c87b
cleanup debugging code
davies 3908f5c
Merge branch 'master' into pickle
davies f44f771
enable tests about array
davies b30ef35
use pickle to serialize data for mllib/recommendation
davies 52d1350
use new protocol in mllib/stat
davies f1544c4
refactor clustering
davies aa2287e
random
davies 8fe166a
Merge branch 'pickle' into pickle_mllib
davies cccb8b1
mllib/tree
davies d9f691f
mllib/util
davies f2a0856
mllib/regression
davies c383544
classification
davies 6d26b03
fix tests
davies 4d7963e
remove muanlly serialization
davies 84c721d
Merge branch 'master' into pickle_mllib
davies b02e34f
remove _common.py
davies 0ee1525
remove outdated tests
davies 722dd96
cleanup _common.py
davies f3506c5
Merge branch 'master' into pickle_mllib
davies df19464
memorize the module and class name during pickleing
davies 46a501e
choose batch size automatically
davies 88034f0
rafactor, address comments
davies 9dcfb63
fix style
davies 708dc02
fix tests
davies e1d1bfc
refactor
davies 44736d7
speed up pickling array in Python 2.7
davies 154d141
fix autobatchedpickler
davies df625c7
Merge commit '154d141' into pickle_mllib
davies a379a81
fix pickle array in python2.7
davies 44e0551
fix cache
davies 9ceff73
test size of serialized Rating
davies a2cc855
fix tests
davies 1fccf1a
address comments
davies 2511e76
cleanup
davies 19d0967
refactor Picklers
davies e431377
fix cache of rdd, refactor
davies bd738ab
address comments
davies 032cd62
add more type check and conversion for user_product
davies 810f97f
fix equal of matrix
davies dffbba2
Merge branch 'master' of github.com:apache/spark into pickle_mllib
davies File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If the first record is very large,
batch
will be 0.