Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[SPARK-3571] Spark standalone cluster mode doesn't work. #2436

Closed
wants to merge 4 commits into from
Closed

[SPARK-3571] Spark standalone cluster mode doesn't work. #2436

wants to merge 4 commits into from

Conversation

sarutak
Copy link
Member

@sarutak sarutak commented Sep 17, 2014

I think, this issue is caused by #1106

@SparkQA
Copy link

SparkQA commented Sep 17, 2014

QA tests have started for PR 2436 at commit 4817ecd.

  • This patch merges cleanly.

@andrewor14
Copy link
Contributor

Oops, good catch...

andrewor14
andrewor14 reviewed Sep 17, 2014
View reviewed changes
core/src/main/scala/org/apache/spark/deploy/master/Master.scala
@@ -491,14 +491,13 @@ private[spark] class Master(
val shuffledAliveWorkers = Random.shuffle(workers.toSeq.filter(_.state == WorkerState.ALIVE))
val aliveWorkerNum = shuffledAliveWorkers.size
var curPos = 0
var stopPos = aliveWorkerNum
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Won't this result in an infinite loop if no workers are available? curPos is never equal to aliveWorkerNum

@SparkQA
Copy link

SparkQA commented Sep 17, 2014

QA tests have started for PR 2436 at commit 4e51e35.

  • This patch merges cleanly.

andrewor14
andrewor14 reviewed Sep 17, 2014
View reviewed changes
core/src/main/scala/org/apache/spark/deploy/master/Master.scala

if (aliveWorkerNum > 0) {
var curPos = 0
var stopPos = aliveWorkerNum
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As I mentioned before, this is not correct. curPos != stopPos will always be true since curPos is always mod aliveWorkerNum.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, yes. I'll modify.

@SparkQA
Copy link

SparkQA commented Sep 17, 2014

QA tests have finished for PR 2436 at commit 4817ecd.

  • This patch passes unit tests.
  • This patch merges cleanly.
  • This patch adds the following public classes (experimental):
    • class JavaSparkContext(val sc: SparkContext)
    • class ArrayConstructor extends net.razorvine.pickle.objects.ArrayConstructor
    • throw new IllegalStateException("The main method in the given main class must be static")
    • class TaskCompletionListenerException(errorMessages: Seq[String]) extends Exception
    • class NonASCIICharacterChecker extends ScalariformChecker
    • class Dummy(object):
    • class RatingDeserializer(FramedSerializer):
    • class SCCallSiteSync(object):
    • case class CreateTableAsSelect(
    • class Encoder[T <: NativeType](columnType: NativeColumnType[T]) extends compression.Encoder[T]
    • class Encoder[T <: NativeType](columnType: NativeColumnType[T]) extends compression.Encoder[T]
    • class Encoder[T <: NativeType](columnType: NativeColumnType[T]) extends compression.Encoder[T]
    • class Encoder extends compression.Encoder[IntegerType.type]
    • class Decoder(buffer: ByteBuffer, columnType: NativeColumnType[IntegerType.type])
    • class Encoder extends compression.Encoder[LongType.type]
    • class Decoder(buffer: ByteBuffer, columnType: NativeColumnType[LongType.type])
    • case class CreateTableAsSelect(
    • class JavaStreamingContext(val ssc: StreamingContext) extends Closeable

@SparkQA
Copy link

SparkQA commented Sep 17, 2014

QA tests have started for PR 2436 at commit 118c031.

  • This patch merges cleanly.

@SparkQA
Copy link

SparkQA commented Sep 17, 2014

QA tests have started for PR 2436 at commit 7a4deea.

  • This patch merges cleanly.

@SparkQA
Copy link

SparkQA commented Sep 17, 2014

QA tests have finished for PR 2436 at commit 4e51e35.

  • This patch passes unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Sep 17, 2014

QA tests have finished for PR 2436 at commit 118c031.

  • This patch fails unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@sarutak
Copy link
Member Author

sarutak commented Sep 17, 2014

retest this please.

@SparkQA
Copy link

SparkQA commented Sep 17, 2014

QA tests have finished for PR 2436 at commit 7a4deea.

  • This patch passes unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@andrewor14
Copy link
Contributor

Ok, LGTM... I tested this locally. Merging into master.

@asfgit asfgit closed this in cbc0650 Sep 17, 2014
@sarutak sarutak deleted the SPARK-3571 branch April 11, 2015 05:22
@bwongchaowart bwongchaowart mentioned this pull request Mar 14, 2016
Closed
CodingCat added a commit to CodingCat/spark that referenced this pull request Mar 14, 2016
@CodingCat
recover apache#2436
0bb54f6
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@sarutak @SparkQA @andrewor14