Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[SPARK-25268][GraphX]run Parallel Personalized PageRank throws serialization Exception #22271

Closed
wants to merge 1 commit into from
Closed

[SPARK-25268][GraphX]run Parallel Personalized PageRank throws serialization Exception #22271

wants to merge 1 commit into from

Conversation

shahidki31
Copy link
Contributor

What changes were proposed in this pull request?

mapValues in scala is currently not serializable. To avoid the serialization issue while running pageRank, we need to use map instead of mapValues.

Please review http://spark.apache.org/contributing.html before opening a pull request.

@shahidki31
[SPARK-25268][GraphX]runParallelPersonalizedPageRank throws serializa…
7651652 
…tion Exception

mapValues in scala is currently not serializable. To avoid the serialization issue while running pageRank, we need to use map instead of mapValues.
@shahidki31 shahidki31 changed the title [SPARK-25268][GraphX]runParallelPersonalizedPageRank throws serialization Exception [SPARK-25268][GraphX]run ParallelPersonalizedPageRank throws serialization Exception Aug 29, 2018
@shahidki31 shahidki31 changed the title [SPARK-25268][GraphX]run ParallelPersonalizedPageRank throws serialization Exception [SPARK-25268][GraphX]run Parallel PersonalizedPageRank throws serialization Exception Aug 29, 2018
@shahidki31 shahidki31 changed the title [SPARK-25268][GraphX]run Parallel PersonalizedPageRank throws serialization Exception [SPARK-25268][GraphX]run Parallel Personalized PageRank throws serialization Exception Aug 29, 2018
@mengxr
Copy link
Contributor

mengxr commented Aug 31, 2018

test this please

@SparkQA
Copy link

SparkQA commented Aug 31, 2018

Test build #95561 has finished for PR 22271 at commit 7651652.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@jkbradley
Copy link
Member

LGTM
I tested this locally and confirmed it fixes the serialization issue.
Thank you @shahidki31 !
Merging with master after fresh tests finish

@SparkQA
Copy link

SparkQA commented Sep 6, 2018

Test build #4332 has finished for PR 22271 at commit 7651652.

  • This patch passes all tests.
  • This patch does not merge cleanly.
  • This patch adds no public classes.

@shahidki31
Copy link
Contributor Author

@SparkQA Test this please

@jkbradley
Copy link
Member

test this please

@SparkQA
Copy link

SparkQA commented Sep 6, 2018

Test build #95747 has finished for PR 22271 at commit 7651652.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@asfgit asfgit closed this in 3b6591b Sep 6, 2018
asfgit pushed a commit that referenced this pull request Sep 6, 2018
@shahidki31 @jkbradley
[SPARK-25268][GRAPHX] run Parallel Personalized PageRank throws seria…
085f731 
…lization Exception

## What changes were proposed in this pull request?
mapValues in scala is currently not serializable. To avoid the serialization issue while running pageRank, we need to use map instead of mapValues.

Please review http://spark.apache.org/contributing.html before opening a pull request.

Closes #22271 from shahidki31/master_latest.

Authored-by: Shahid <shahidki31@gmail.com>
Signed-off-by: Joseph K. Bradley <joseph@databricks.com>
(cherry picked from commit 3b6591b)
Signed-off-by: Joseph K. Bradley <joseph@databricks.com>
@shahidki31
Copy link
Contributor Author

Thank you @jkbradley for merging.

@shahidki31 shahidki31 deleted the master_latest branch September 6, 2018 17:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@shahidki31 @mengxr @SparkQA @jkbradley