Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix #SPARK-1149 Bad partitioners can cause Spark to hang #44

Closed
wants to merge 18 commits into from
Closed

Fix #SPARK-1149 Bad partitioners can cause Spark to hang #44

Changes from 1 commit
Commits
Show all changes
18 commits
Select commit Hold shift + click to select a range
6bb725e
fix #SPARK-1149 Bad partitioners can cause Spark to hang
Feb 28, 2014
3a65903
make the code more readable
Feb 28, 2014
e68210a
add partition index check to submitJob
Mar 2, 2014
61e5a87
Merge branch 'master' into SPARK-1149
Mar 2, 2014
3348619
Optimize performance for partitions check
Mar 3, 2014
1e3331e
Merge branch 'master' into SPARK-1149
Mar 3, 2014
db6ecc5
Merge branch 'master' into SPARK-1149
Mar 4, 2014
928e1e3
Added a unit test for PairRDDFunctions.lookup with bad partitioner
Mar 4, 2014
adc443e
partitions check bugfix
Mar 4, 2014
3feb3a8
Merge branch 'master' into SPARK-1149
Mar 4, 2014
ac006a3
code Formatting
Mar 4, 2014
3395ee7
Merge remote-tracking branch 'upstream/master' into SPARK-1149
Mar 5, 2014
d0a6005
review comment
Mar 5, 2014
b0d5c07
review comment
Mar 5, 2014
e3e56aa
Merge branch 'master' into SPARK-1149
Mar 7, 2014
3dad595
review comment
Mar 7, 2014
8425395
Merge remote-tracking branch 'upstream/master' into SPARK-1149
Mar 8, 2014
3dcdcaf
Merge branch 'master' into SPARK-1149
Mar 11, 2014
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions core/src/main/scala/org/apache/spark/SparkContext.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -847,6 +847,8 @@ class SparkContext(
partitions: Seq[Int],
allowLocal: Boolean,
resultHandler: (Int, U) => Unit) {
val rddPartitions = rdd.partitions.map(_.index)
partitions.foreach(p =>require(rddPartitions.contains(p), "partition index out of range: " +p))
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

require(partitions.forall(rddPartitions.contains, "partition index out of range")), more intuitive?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

require(partitions.forall(rddPartitions.contains(_)), "partition index out of range") is more readable?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oops, I found that the parentheses are mismatching in my comments

val callSite = getCallSite
val cleanedFunc = clean(func)
logInfo("Starting job: " + callSite)
Expand Down